Skip to content
Publicly Available Published by Oldenbourg Wissenschaftsverlag July 19, 2022

Evaluating Virtual Reality Simulations for Construction Safety Training

A User Study Exploring Learning Effects, Usability and User Experience

  • Markus Jelonek

    Markus Jelonek is a research assistant at the Institute of Work Science (IAW) at the Ruhr University Bochum (RUB). He studied Psychology and Applied Computer Science at the University of Twente and RUB and has worked as a software engineer in the industry before joining the IAW. His research focuses on Human-Computer Interaction, particularly Usability and User Experience studies and Virtual Reality as evaluation and prototyping tool for use cases of Human Augmentation.

    ORCID logo EMAIL logo
    , Eileen Fiala

    Eileen Fiala studied Social Science at the Ruhr University Bochum. As research assistant at the Institute of Work Science she has been involved in conducting usability studies.

    , Thomas Herrmann

    Thomas Herrmann is professor of Information and Technology-Management at the Ruhr University Bochum, Germany. Current research interests include design methods for socio-technical systems in various areas such as healthcare, computer supported collaboration, knowledge management, process management, smart factories and the interaction between humans and AI. He holds a PhD in Software Engineering from the Technical University of Berlin.

    , Jochen Teizer

    Jochen Teizer is Professor in the Department of Civil and Mechanical Engineering at Technical University of Denmark (DTU) where his research seeks lean and injury-free construction work environments. He earned a Ph. D. from The University of Texas at Austin in 2006 and a Dipl.-Ing. from the Karlsruhe Institute of Technology in 2002. He is the Director of the Construction Automation and Information Technologies Laboratory and a Vice-President for the International Association for Automation and Robotics in Construction (IAARC). With over 250 peer-reviewed publications in books, journals, and conference proceedings and numerous academic and construction industry teaching and research awards, he also serves as a visionary and consultant in an industry that seeks active learning tools for safety education and training.

    , Stephan Embers

    Stephan Embers studied Applied Computer Science at the Ruhr University Bochum and graduated with a Master of Science degree in 2018. He currently works as a research assistant at the Chair of Computer Science in Engineering at Ruhr University Bochum. His research there includes augmented and virtual reality in construction as well as human-machine interaction and safety in construction.

    , Markus König

    Prof. Dr.-Ing. Markus König holds the Chair of Computing in Engineering at the Ruhr University Bochum since 2009. He has been teaching and researching digitization in the construction industry for many years. He was in charge of the scientific evaluation of the first BIM projects in Germany. In 2015, together with other experts, he developed the German road map for digital design and construction. Currently, he is deputy head of the German competence center for the digitalization of construction (BIM Germany). Prof. König published more than 250 technical papers and led several large research projects. For his scientific and practical achievements regarding Building Information Modeling, he received the Konrad Zuse Medal of the German Construction Industry in 2020.

    and Arno Mathis

    Arno Mathis is responsible for digital service and training development at the Hilti headquarters in Liechtenstein. He obtained his Ph. D. in 2008 at the University of Twente in the Netherlands. It is the aim of Hilti to support players in the construction industry to make effective and efficient use of new digital solutions to enhance worker health & safety.

From the journal i-com

Abstract

The construction industry is one of the sectors with the highest accident rates. To prevent accidents, construction workers receive occupational safety training and safety instructions. However, experience-based learning of dangerous situations is hardly possible or justifiable in reality. Virtual reality (VR) simulations can be a potential solution in this regard by allowing workers to experience dangerous situations in a very vivid but safe way without being exposed to real hazards. In this study, a VR simulation for construction safety training was developed and tested with trainees that learn the safe operation of hand-operated power tools. In this particular case study, the objective for the participants in the VR simulation was to successfully consider all safety aspects in the operation of an angle grinder. The usability, user experience and implicit learning were investigated during the study. Additionally, we conducted post-play interviews with participants. Results found learning effects of participants as well as a satisfying user experience and usability. The results also show that participants might learn content as presented, risking the learning of false information if the simulation does not cover relevant safety aspects.

1 Introduction

One of the most challenging tasks for the construction and industry sector to date is to increase occupational health and safety and reduce accidents on construction sites. For 2020, 97 deadly accidents were reported for the German construction industry [2]. Factors that lead to occupational accidents are diverse, including: inattentiveness or distraction [14], [32], fatigue [11], mismatch of competence [10] or inexperience [11], to name a few. In addition, construction workers are not prepared for dangerous situations in the typically very complex and highly dynamic work environment. Consequentially, it is hardly possible to experience hazards beforehand in a safe environment and train adequate and safe reactions. In order to avoid accidents and hazardous situations, construction companies use various approaches to solve this problem, for instance, by applying safety inspections, safety rules or even penalties for non-compliance with safety rules [27].

Figure 1 
            Starting scene: viewpoint of the trainee’s participant of the VR simulation in the foreman’s office.
Figure 1

Starting scene: viewpoint of the trainee’s participant of the VR simulation in the foreman’s office.

Another method that is widely used in the industry and focusses on workers’ safety trainings for accident prevention. These training methods include hands-on practice, not just theoretical safety workshops and are a promising approach for workers as knowledge has to be applied directly. We suppose that the more direct an experience – e. g. a near-accident – is related to the lesson to be learnt, the more it results in a learning effect that directs an individual’s behavior [15]. However, in the case of training that helps to avoid safety risks, learning must not be based on real accidents. With the use of Virtual Reality (VR), such training can take place in immersive virtual environments that also allow for experiencing hazards and safety risks. Therefore, this paper addresses the research question to what extent VR can contribute to virtually experiencing accidents or near-accidents that have a similarly instructive effect as real events.

Several studies have already shown benefits of VR trainings, like, for example, an increased attention and concentration increasing knowledge acquisition of study participants compared to conventional methods [26], [37]. In comparison to theoretical lessons, it was observed that participants were able to maintain focus during a VR safety training for a longer period of time and thus create a stronger learning effect [37].

However, to our knowledge, there is only minor research available that examines the question whether VR can be used to provide an experience of accidents that has a sufficient training effect of increasing awareness for safety risks. For example, in [22] a collaborative construction safety VR system for experiential learning is proposed that allows to understand accident causes and their prevention. To evaluate the system, the researchers focused on usability of the system and task performance but did not assess the knowledge acquisition of each student via the VR experience. In our study, we seek to answer our research question with an exemplary study design of an available VR-solution.

In the present research, a VR based safety training simulation to work with angle grinders was evaluated with 14 machine operator trainees and two instructors (Figure 1). We analyzed the perceived usability of the system, the participants’ user experience and learning of safety knowledge. For a deeper understanding of participants’ decision behavior during the simulation, semi-structured interviews were conducted afterwards with each participant.

2 Virtual Reality Trainings for Construction Workplace Safety

In the construction industry, considerable research interest exists in developing Virtual Reality (VR) simulations and applications for safety education, safety and health training and hazard identification (e. g. [25], [34]). One of the major advantages of VR is the potentially high immersion to experience hazards and the handling of dangerous work equipment without being exposed to any real danger [13]. Thus, with virtual environments, on-the-job-training or experiential learning can be done in risk-free environments while still feeling a high amount of presence in the scene.

With the ability to experience hazards in a safe environment, there are many studies on how VR simulations can reduce accidents and increase safety in the construction industry (e. g. [13], [16], [25], [43], [44]) or other areas like pedestrian safety or evacuation tasks (e. g. [19], [20], [30]). For example, it was examined how VR can contribute to safety promotion in construction tasks, by designing effective training modules that allow trainees to become familiar with all kinds of hazards in the work environment [46]. VR is also used to investigate very specific use cases of construction site work, for example, hazard detection on construction sites [34], working in heights [35] or examining workers‘ inattentiveness during construction tasks [17], [18]. A prototype of a VR interactive training environment that enables players to experience and reflect on consequences in a VR construction site is described in [12]. Experiencing hazards in VR simulations can also bring out memories of previously experienced hazards and trigger reflection processes [8].

The use of Building Information Modeling (BIM) in the construction industry facilitates to use daily accurate 3D-representations of construction sites for VR scenarios [7], [45]. Thus, future use of BIM data could enable to analyze accident scenarios virtually, or to walk through potentially hazardous environments beforehand [40]. For an efficient development of VR environments for engineering and construction, solutions incorporating BIM into the unreal game engine have been proposed in [13].

Evaluating a VR simulation for medical emergency training, Lerner et al. [23] have compared pre- and post-knowledge of participants after experiencing the VR simulation. They also assessed the Igroup Presence Questionnaire [39], the System Usability Scale [4], and the Training Evaluation Inventory (TEI) [36], in addition to other measurements. The results of their study have shown a significant correlation between the experience of presence (in the scene) and the perceived training effectiveness. The system usability of the VR training was rated as sufficiently with a score of 65.56 points. However, the results did not show a significant difference between the pretest and posttest of the knowledge test.

Although a considerable body of research has investigated the use of VR in various construction settings for work safety, less attention has been paid to acquiring safety knowledge about construction tools through VR simulations. For example, Le, Pedro & Park [22] focused on providing a collaborative VR system for students and teachers using a construction environment to understand and explain general safety risks in construction work. In their evaluation they assessed the subjective views on the usability of the system, the participants’ task load and compared the system between collaborative use and a single user, but did not compare the individual knowledge acquisition of each user. The results showed that the ease of use was rated as good, the collaboration function was rated as very useful, but that educators criticized the creation of scenarios as time consuming. Some research contributions in VR safety training highlight the development process of VR simulations or focus on technical implementations without evaluating the prototypes for teaching safety knowledge. For example, Filigenzi & Orr [9] have shown a VR system for mine safety training, but did not conduct an evaluation of the system. Similarly, Pérez et al. [33] describe the development of a VR safety training for offshore oil platforms but have not conducted an evaluation of their prototype. The same applies for [21], where a VR construction safety training is proposed and the potentials of such system are highlighted, and for [31], where a VR experience-based safety training for chemical plants is proposed.

Sacks et al. [37] have proposed a CAVE (Cave Automatic Virtual Environment) construction safety training and compared it to conventional methods in a ‘between-subjects’ experimental design. To evaluate the system, participants did a safety knowledge test as baseline measurement and were then divided into two groups: one group received the VR training, the other group received conventional classroom instructions. In the VR training, the group participants simultaneously experienced several accident scenarios of an avatar in a ‘third person’ view. After the training, both groups repeated the safety knowledge test and filled out an experience questionnaire. The results have shown significant advantages of the VR training for stone cladding work and cast-in-situ concrete work, but not for general construction site safety. Additionally, the learning effectiveness was higher in the VR training and participants in the VR group did maintain a higher level of alertness and concentration compared to the group with the conventional training.

In our research, we present the evaluation of an interactive VR safety training in which participants individually experienced a construction site scenario and received safety aspects about working safely with angle grinders, without providing additional feedback or learning materials any other than given by the simulations’ content. To finish the simulation successfully, participants had to consider multiple safety aspects while they were pursuing the task to cut an air shaft. Whenever a participant made a mistake or forgot a safety measure, an accident happened and the simulation restarted, giving participants visual feedback about their mistake. In addition to the visual feedback, the participants received an explanation of the foreman as non-player character (NPC) (Figure 1).

The aim of the study was to see how participants perceive such VR simulation and whether knowledge about occupational safety while working with angle grinders is acquired incidentally by experiencing the simulation.

3 Methodological Approach

Figure 2 
            Inside the VR simulation, participants had to choose between a safe angle grinder (on the left) and an unsafe one (right).
Figure 2

Inside the VR simulation, participants had to choose between a safe angle grinder (on the left) and an unsafe one (right).

In this study, we used a VR simulation in which participants had to pay attention to their safety and the safety of co-workers in a construction building. Inside the VR simulation, participants were given a task by a foreman (an NPC) to make a cut with an angle grinder into an air shaft (see Figure 5). In order to accomplish the given task, participants first had to choose the correct personal protection equipment (PPE) and a safe angle grinder, then had to walk across the construction site via a specific path to reach the air shaft. During the simulation, participants could experience several hazards or accidents and had to make the right decision for their own safety:

  1. Choosing the correct personal protection equipment (PPE), by deciding between a helmet with or without a full-face shield. Since not only sparks can fly when working with angle grinders, but also cut-off parts, safety goggles or face shields should be used in any case.

  2. Selecting a functioning and safe working tool, by deciding between an angle grinder with or without guard and handle. Depending on the material to be cut, angle grinders can become jammed in the material. This can result in strong recoil, so angle grinders should always be used with both hands. In addition, there should always be a wheel guard on an angle grinder, which should be positioned with the back to the body, as this ensures that in the event of a shattering cut-off wheel, the parts do not hit the body.

  3. Choose a safe path between the starting point of the simulation (supervisors’ office) and the area in which the task had to be fulfilled. Due to parked material, traffic and other workers, construction sites can be confusing. To avoid putting yourself in danger, a safe path to the worksite should be used.

  4. Keeping the personal workspace clear of other workers and avoid to walk into or through the workspace of other co-workers (see Figure 4). It should always be ensured that one’s worksite is free of other workers who could otherwise be struck by flying or falling parts. Likewise, one should not enter the work area of other workers unannounced, as there is also the risk of being hit by flying parts.

  5. Pay attention to the traffic on the construction site and do not get run over by a forklift.

A schematic overview of the virtual construction site and the placement of hazards is shown in Figure 3. In addition to these safety hazards, a trip, slip, fall accident hazard was integrated in the scene. On the path to the air shaft was an oil puddle placed and the participants had to deliberately avoid the puddle. When participants stepped on the puddle virtually, one would fall down and the screen would turn black before restarting the simulation. However, in several pre-runs of the study (see Section 3.3), we found that participants never noticed the oil puddle visually, as it was at the lower field of view, and therefore never avoided it. Additionally, the animation of falling down caused symptoms of motion sickness among participants in the pre-runs. Therefore, we have refrained from testing this hazard and have removed it from the simulation prior to the main study.

Figure 3 
            Schematic overview of the virtual construction site, the safe path participants had to take and the placement of hazards.
Figure 3

Schematic overview of the virtual construction site, the safe path participants had to take and the placement of hazards.

Figure 4 
            Example of an unsafe path, as the participant (wearing the safety helmet) would have to walk through the workspace of a co-worker that is using an angle grinder.
Figure 4

Example of an unsafe path, as the participant (wearing the safety helmet) would have to walk through the workspace of a co-worker that is using an angle grinder.

To play the simulation, participants were equipped with an HTC Vive Pro Headset and two controllers. In the game, their player character was only equipped with safety gloves. After a brief introduction by the supervisor/foreman in the game, participants could begin the simulation. The playtime varied between 3–15 minutes, depending on each users’ decisions and necessary trials to complete the task. If a participant made a mistake, e. g. was run over by a forklift truck, the game started from the beginning and the supervisor game character explained what the participant did wrong.

3.1 Participants

In this study, 14 machine operator trainees participated (2 female, 12 male), aged 18 to 32 years (M=22.79, SD=3.98). Ten of the participants stated that they had either some experiences with VR or tried it at least once. All participants were tested in the facility of the participating training center’s facility, had prior knowledge of construction site safety aspects, and two years of experience as trainees. They got no other introduction to their task other than what the foreman in the VR simulation gave them and had to make their choices based on their judgement. Additionally, two training instructors, aged 32 and 61, were also interviewed after going through the same procedure as the 14 trainees, to get the view and opinions of instructors towards such VR simulations. However, their results were not included into the quantitative analysis.

As two participants showed signs of motion sickness during or after playing the virtual reality simulation and their ability to concentrate was influenced negatively, their results were removed for the quantitative analysis, resulting in a final sample of n=12 (M=23.17, SD=4.15). However, we still considered the answers both participants made during the interviews for the qualitative analysis, as their opinions could still give insights about their experience with the virtual reality simulation.

3.2 Measures

To evaluate the perceived usability, the System Usability Scale (SUS) was used. The SUS consists of 10 questions about the perception of a systems’ usability [4]. It measures the agreement or disagreement with a given statement on a 5-point scale, to evaluate effectiveness, efficiency and satisfaction. The SUS is an often used quantitative tool for empirical analyses to “quickly and easily collect a user’s subjective rating of a product’s usability” [1].

The User Experience Questionnaire (UEQ) was used to evaluate the perceived user experience. The UEQ uses 26 items for user experience aspects with a 7-point Likert scale to measure five different categories: the attractiveness, perspicuity, efficiency, dependability, stimulation and novelty of a system [38], [41].

Additionally, we designed a questionnaire with multiple questions about safety on construction sites, which specifically included safety questions regarding angle grinders. The reason for creating a separate questionnaire was to investigate whether participants learn safety measures when working with angle grinders purely on experiencing the simulation and there was no questionnaire about the safe use of angle grinders publicly available. The designed questionnaire is based on the official recommendations of the German Employer’s Liability Insurance Association for the Construction Industry (BG BAU). Thirty-six items of the questionnaire had simple yes or no answers given, so that participants had to decide whether they believed the statements were true or false. The two remaining items were multiple-choice questions. Of the 36 dichotomic items, 22 items addressed general safety aspects (without relation to the content of the simulation) and 14 items had a direct relation to the content of the VR simulation and the work-tool (angle grinder). For instance, if angle grinders should always be operated with work gloves and with both hands. In addition, one of the two multiple-choice questions asked specifically about the needed personal protective equipment (PPE) for the work on construction sites, whereas the other multiple-choice question asked specifics about angle grinders which were not part of the VR simulation.

Finally, the qualitative data collected through semi-structured interviews, was used to gain insights about the decisions that participants took during the simulation, their preferences in training methods, their opinion about VR-trainings, also concerning their view regarding workplace safety before and after playing the simulation. The interviews were organized into four categories: (1) the background of the person, his/her experience with VR, and contend pre-experience of construction work and general safety aspects, (2) content of the simulation, (3) visualization in the simulation and (4) built up comprehension about safety training/aspects. The expert interview was constructed in the same manner, except for an assessment of the simulation that focused on other training methods used in the education, like theoretical training manuals, in order to establish a teacher’s perspective for this study. All interviews were analyzed by applying content analysis [28], [29].

3.3 Study Procedure

Prior to the actual testing of the simulation, several pre-runs of the study were iteratively conducted with professionals from the construction industry and research. Based on the observations and comments during these tests, adjustments were made to the simulation and to the study design. These five pre-runs were mainly intended for quality assurance purposes and to resolve inconsistencies and inaccuracies in the content of the simulation and its incremental development. For example, the hazard of an oil puddle was removed as it was overlooked in every pre-run and the resulting fall over (within the simulation) led to motion sickness symptoms (see Section 3). Adjustments were also made to other hazard areas to ensure that they were presented correctly in terms of content and, most importantly, that they were clearly understandable to the participants.

The main study was conducted on two consecutive days. On the first day, the participants were informed about the general course of the study without going into the content of the simulation. They received the privacy statement and were informed that they could withdraw their participation in the study at any time. Additionally, they also got the opportunity to ask questions in case they had any. Each participant then completed the questionnaire with 38 items about the general occupational safety on construction sites. At the end of this first session, the participants received an individual time slot for their participation on the next day.

Figure 5 
              The simulation was successfully completed when all safety measures were followed and participants started cutting a ventilation shaft.
Figure 5

The simulation was successfully completed when all safety measures were followed and participants started cutting a ventilation shaft.

On the second day, each participant was given a brief explanation of the technology and the controllers. There were no further instructions than that they will have to fulfill a task during the simulation. Before they started the simulation, each participant could try out the controllers in a short VR tutorial. During the tutorial they learned how to move around in the VR environment by using the controllers and how to pick up and use a power tool. To this point, participants had the possibility to ask questions if anything was unclear. When they were ready the simulation started in the office of a supervisor/foreman on a construction site. The avatar of the supervisor told participants their task in the simulation. Participants had to choose between two different helmets which differed only in having a visual protection or not, and between two different angle grinders, where one of the angle grinders was missing the side arm. Next to the exit door of the office was an overview map on which the participants could see their place of work for the task.

The simulation could only be completed successfully when all necessary safety aspects had been considered. The participants were then relieved of the two controllers and the head-mounted device and were asked briefly about their general state of health. They then went with the investigator to a room next door and completed the questionnaire on general occupational safety again. Following this, participants filled out the SUS and the UEQ. After completing the questionnaires, participants were surveyed in a semi-structured qualitative interview.

4 Results

The data analysis focused on three different aspects. First, the general usability and user experience of the simulation was determined to understand whether the usability of the simulation is sufficient and how the participants evaluate the VR simulation in terms of user experience. Second, the questionnaire on general safety was used to check to which extent short-term learning effects could be identified. By analyzing the results of this questionnaire, we can examine whether the VR training has led to any knowledge acquisition at all. Finally, the qualitative interviews were evaluated in order to obtain additional information about the acceptance, user experience, usability and the content of the simulation and the system.

The data collected by the four questionnaires (pre and post safety questionnaire, SUS, UEQ) and the semi-structured interviews of all 12 participants and the 2 experts, were analyzed in regard to learning effects, usability and perception and preferences for a specific learning method for safety training.

Methodically, the two general safety questionnaires were evaluated comparing the pre and post results, calculating mean and using a paired t-test to estimate the statistical significance, while Cohens’ d was used to predict the correlation and the effect strength. Here the analyses were realized separately for the overall 14 items and one multiple-choice question concerning the content of the scenario and the whole questionnaires, counting 38 items.

4.1 Usability and User Experience

For the analysis of the systems’ usability and the user experience, the SUS and UEQ were used and combined with participants’ statements made during the interviews. The results of the SUS scores were calculated as proposed in [4] and show a score of 78.7 (SD=11.6), which can be described as highly acceptable on the acceptability range and “good” on the adjective rating [3]. A SUS score over 68 is considered above average and a score of 78.7, considering percentile rankings indicate a very positive perception and usability. During the interviews, participants supported the usability rating with further comments about the simulation. For example, participants noted that controlling the player in the simulation was easy after trying it during the tutorial. No other problems were mentioned than to memorize which button resulted in which action, but even that was considered as acceptable and would vanish with frequent use.

The results for the UEQ show good to excellent rating of the simulation for each category. The overall attractivity reaches a score of 2.06 (out of −3/+3), the pragmatic quality (perspicuity, efficiency and dependability) is scored at 1.67 and the hedonic quality (stimulation, novelty) reaches a score of 1.95 (an overview of the results for each individual scale is shown in Figure 6).

Figure 6 
              Results of the User Experience Questionnaire (UEQ), showing that participants gave the VR simulation positive ratings for attractiveness (2.06), perspicuity (1.88), efficiency (1.6), dependability (1.52), stimulation (2.21), and novelty (1.7).
Figure 6

Results of the User Experience Questionnaire (UEQ), showing that participants gave the VR simulation positive ratings for attractiveness (2.06), perspicuity (1.88), efficiency (1.6), dependability (1.52), stimulation (2.21), and novelty (1.7).

The qualitative data supports the findings of the UEQ. The interviews showed that most trainees would prefer VR training over theoretical training and gave a very positive assessment. During the semi-structured interviews, all 14 participants rated the visualization of simulation as very positive, while 11 of 14 participants experienced the simulation as very positive and realistic overall. Considering the training of work safety aspects, 9 out of the 15 participants acknowledged the current version simulation as helpful. Especially picking the right PPE and the possibility to virtually experience work-hazards was seen as highly effective. Additionally, 10 participants mentioned they felt safer after exploring the simulation, especially concerning general safety aspects.

Finally, all participants managed to solve the task on their own, without any interfering comments from the experiment supervisor, making the simulation itself usable without further external supervision.

4.2 Results of Angle Grinder Safety Questionnaire

After testing for normal distribution of the population, the safety questionnaire results were analyzed by using a paired sample t-test to compare participants scores in the safety questionnaire (prior and after the simulation). The scores were compared on the complete safety questionnaire (36 items), the subset about general safety aspects without relation to the simulations’ content (22 items) and the 14 questions related to the content of the simulation (Table 1).

Results show that differences in the scores did not reach statistical significance for the complete safety questionnaire and its 22-item subset with no relation to the content of the simulation. As the complete safety questionnaire and the 22-item subset contain mostly general safety questions that were not part of the simulation and participants had to answer those items to their best knowledge, the results suggest that participants did not gain any further knowledge in these areas through the VR experience. However, there was a significant difference in the scores for content-related items before (M=11.08, SD=1.31) and after (M=12.17, SD=1.19) playing the simulation; t(11)=2.32, p=.041, d=0.67. These results indicate that playing the simulation led to short-term learning effects concerning safety aspects when working with angle grinders. Participants therefore had higher scores in the questionnaire subset about safety aspects when working with angle grinders after experiencing the VR simulation.

Table 1

Paired sample t-test to examine pre- and post- results of the safety questionnaire (n = 12).

Questionnaire Test results pre VR Test results post VR Paired t-test


Mean Std. D. Mean Std. D.
Complete Safety questionnaire (36 items) 28.17 2.69 29.00 2.49 NS
General safety questions (22 item subset) 17.17 1.99 17.33 2.06 NS
Simulation specific questions (14 item subset) 11.08 1.31 12.17 1.19 *
  1. Notes: The safety questionnaire was applied one day prior and immediately after the training. T-test results: * p < 0.05, ** p < 0.01, NS = no significance.

The multiple-choice (mc) questions were analyzed separately. The first mc-question focused on technical knowledge about angle grinders and asked participants which of the aspects have to be considered when working with angle grinders (Table 2). All three aspects should be considered, although only the first two should be common knowledge among construction workers. Correct answers were the same before and after playing the simulation, suggesting that participants did not gain any knowledge about these aspects by playing the simulation and matches the simulations’ content, as the VR simulation did not teach participants about these three technical aspects.

Table 2

Number of correct answers to the technical safety questions about working with angle grinder discs (n = 12).

Which of the following aspects should be considered when working with angle grinders? Pre VR Post VR
Disc must be certified for the material. 12 12
Disc must fit on the protection. 12 12
Expiration date of the disc may not be exceeded. 3 3

The mc-question about necessary PPE, however, showed differences in the scores. The question asked which kind of PPE one should use, when working with an angle grinder on construction sites (Table 3). Except for barrier tape, using all of the listed PPE is recommended when working with angle grinders [6]. The results show higher scores for helmet, safety shoes and gloves after the simulation was played through. However, scores for breathing mask and hearing protection dropped, compared to the scores before playing the simulation. This result suggests that some participants mapped the content of the simulation pretty accurately to the answers of this mc-question, as no other information was given before or after playing the simulation. In the simulation, the participant wore gloves (virtually) and had to choose a helmet with or without face protection, whereas hearing protection and a breathing mask were not offered or shown for selection in the simulation. In addition, it should be noted that the scores on the PPE that was not presented in the simulation (hearing protection and breathing mask) dropped after playing the simulation. The results suggest, that if the content of the simulation is incorrect or incomplete, participants might learn the content as presented, risking the learning of false information.

Table 3

Correct answers to the MC-questions about Personal Protection Equipment (PPE) (n = 12).

What kind of PPE should you wear, when working with angle grinders? Before After
Safety glasses/Face shield 12 12
Helmet 3 11
Breathing mask 5 2
Safety shoes 11 12
Gloves 4 8
Barrier tape 0 0
Hearing protection 12 9
No PPE is required 0 0

4.3 Interview Analysis

During the interviews, 7 of 11 participants stated that the taught lessons (to make possible mistakes in the simulation) were clear and understandable. Most participants considered the motion control as easy and felt comfortable wearing the 3D Headset, stating that the use of VR as a training method is user-friendly. Participants also had various ideas to further improve the simulation to further increase the positive perception.

With regard to user experience and usability, the interviews largely confirmed that both are fulfilled to a satisfactory degree. In addition, many suggestions came from the participants on how to extend or modify the control system and which extensions would be interesting for the simulation. For example, one participant requested that the forklift driver (NPC) should not drive the forklift towards him while the participant tried to hold eye contact with the driver or even tried to wave at the driver. Such limitations in interaction with NPCs was also considered to reduce reality of the whole simulation. The interviews also highlighted the desire for more complexity, e. g. that much more PPE would have been helpful, even some that would not have been necessary for the use case.

While analyzing the interviews, we became aware that, during the interviews, participants started reflecting prior experiences. They compared and related past experiences to the situations they had experienced in the VR simulation and were able to reconsider their own behavior. Such stimulated reflections can enhance learning effects by VR simulations. A detailed analysis of this can be found in [8].

The two expert interviews have shown a slightly diverse perception of the VR simulation. One instructor rated the simulation as an effective training method and would readily apply it. He also stated, that especially for young people, VR could be a very efficient way to experience hazardous situations and consequences (in a safe environment) and sees advantages especially for memory retention, thus increase overall awareness. The second instructor, age over 60, rated VR trainings as useful (to create awareness and experience different situations) but would apply it only in addition to theoretical and on-the-job-trainings. This instructor was also critically about the instructions given in the VR simulation. As he had no prior experience with VR, he expected not only instructions about the task but also additional instructions about the controls inside the simulation.

Finally, it should be considered to what extent the novelty effect of VR might have influenced the participants’ statements. As described in Section 3.1, four participants stated they had no prior experience with VR, while 10 participants had tried VR at least once before the study. In addition, all participants had experience with digital building simulations, as they had access to stationary construction machine simulators, which were used in the training center. These construction machine simulators consisted of a seat, steering wheel, gearshift and buttons as usually found in construction machines, as well as a display, and allowed to simulate cranes, excavators or wheel loaders. All in all, using VR was not a completely new experience for the participants.

5 Conclusion and Discussion

The results have shown, that the VR scenario for training received great acceptance and was seen as very motivational by the participating trainees. However, great efforts were required to avoid inaccuracies and inconsistencies in the technical implementation of the simulation, as the five pre-runs had shown. The main study showed that the simulation was very positively received by the participants. It was also considered to be an enjoyable experience and we found slightly better results in the safety questionnaire after playing the simulation. Raising working safety awareness is important, especially in high risk areas like construction work. Thus, adding VR simulations to safety training not only supports existing training but also increases consciousness for work related accidents.

In this regard, the hypotheses that construction workers benefit overall in construction safety training through VR simulations can be confirmed through the interviews and quantitative data. Findings show significant learning improvements for working with angle grinders when comparing pre- and post-results of the safety knowledge questionnaire but no significant increase in the results for general work safety on construction sites. Due to the chosen study design, we cannot exclude the possibility that priming effects on safety knowledge for working with angle grinders may have occurred through the pretest questionnaire. However, if such a priming effect took place, it does hardly explain the difference regarding the effects on work with angle grinders vs. general work safety.

The evaluation of the VR simulation could be extended to make a comparison to conventional learning materials. For example, it is conceivable to compare experiential learning via VR simulation with other learning materials, for example, with respect to experience, motivation, cognitive load, and knowledge acquisition. However, it was demonstrated that the attractivity and usability in users’ perception increases the awareness of participants, hence the efficiency, backing former findings of [37]. Conclusively, many participants chose VR as a preferred method over theoretical trainings, backing the findings about dissatisfaction with existing models of training [24], [42]. The overall positive assessment and desire for further development of the scenario shows the importance for further investigation and implementation of VR as a training method.

Since our final sample included only 12 participants, the results are hardly statistically representative. However, as the training should primary be used in professional training, our sample consists of participants of the considered target group. Additionally, the sample size is comparable to other studies in the same area of research [5]. In addition, the sample size is quite appropriate for usability tests. Only the learning effect should be treated with caution and further research in the field of VR based experience learning should be carried out.

In the VR simulation, the choice of the selectable PPE was very limited (only a helmet with or without visual cover) and the VR character was already wearing safety gloves, resulting in very short play durations for some participants. To extend the simulation, it would be useful to offer much more PPE, including some that is not necessary for the particular scenario. To minimize the memorization of the scenario on repetitions (after each accident), the dynamics could be adjusted automatically on each repetition, for example, by varying the number of other workers in the environment, varying the arrangement of the PPE, or placing hazards in different spots. The adaptations would allow the simulation to include multiple decision-making tasks, encouraging participants to think and reflect on their decision even more. Additionally, multiplayer scenarios would be conceivable. Either by several participants working simultaneously on their own tasks in the simulation and having to pay attention to their own safety and that of the other workers. Or also for collaborative working on construction tasks.

Furthermore, the training contents need to match the real-life experiences completely, as inconsistencies during the play was seen as confusing for participants. As a result of these inconsistencies, there is a risk that participants will acquire incorrect knowledge if the simulation content does not fully follow the country’s current safety regulations. Therefore, to prevent the simulation from teaching inaccurate knowledge, we recommend that safety regulations and recommendations should be included in the design of the scenario from the beginning. Additionally, during the development process, the implementation of these regulations into the simulation should be evaluated iteratively.

Finally, we want to address the practicality of such VR simulations for experiential learning purposes. While an at least short-term learning effect could be shown, the development of this VR simulation needed approximately 6 months development time in multiple iterations with a small team of three developers. In addition to the technical implementation, the development also included the selection of a use case (angle grinder), the development of typical accident hazards within the use case and the creation of a scenario for the simulation. Furthermore, it should be considered that the required skills for VR developers are quite high. In addition to programming expertise, they also have to be proficient in the use of game engines to ensure that the VR simulation scenarios are well implemented both audio-visually. Despite the positive perception of the VR simulation by the participants, the development effort clearly minimizes the practicability of such simulations. However, apart from the VR hardware, VR simulations are easily scalable to be distributed on a larger scope, thus having the potential to be used as experiential trainings in the industry.

Award Identifier / Grant number: 02L15A170

Funding source: European Social Fund

Funding statement: This research is funded by the German Federal Ministry of Education and Research (BMBF) and the European Social Fund (ESF) within the Framework Concept “Innovations for Tomorrow’s Production, Services, and Work” (02L15A170) and managed by the Project Management Agency Forschungszentrum Karlsruhe (PTKA-PFT).

About the authors

Markus Jelonek

Markus Jelonek is a research assistant at the Institute of Work Science (IAW) at the Ruhr University Bochum (RUB). He studied Psychology and Applied Computer Science at the University of Twente and RUB and has worked as a software engineer in the industry before joining the IAW. His research focuses on Human-Computer Interaction, particularly Usability and User Experience studies and Virtual Reality as evaluation and prototyping tool for use cases of Human Augmentation.

Eileen Fiala

Eileen Fiala studied Social Science at the Ruhr University Bochum. As research assistant at the Institute of Work Science she has been involved in conducting usability studies.

Thomas Herrmann

Thomas Herrmann is professor of Information and Technology-Management at the Ruhr University Bochum, Germany. Current research interests include design methods for socio-technical systems in various areas such as healthcare, computer supported collaboration, knowledge management, process management, smart factories and the interaction between humans and AI. He holds a PhD in Software Engineering from the Technical University of Berlin.

Jochen Teizer

Jochen Teizer is Professor in the Department of Civil and Mechanical Engineering at Technical University of Denmark (DTU) where his research seeks lean and injury-free construction work environments. He earned a Ph. D. from The University of Texas at Austin in 2006 and a Dipl.-Ing. from the Karlsruhe Institute of Technology in 2002. He is the Director of the Construction Automation and Information Technologies Laboratory and a Vice-President for the International Association for Automation and Robotics in Construction (IAARC). With over 250 peer-reviewed publications in books, journals, and conference proceedings and numerous academic and construction industry teaching and research awards, he also serves as a visionary and consultant in an industry that seeks active learning tools for safety education and training.

Stephan Embers

Stephan Embers studied Applied Computer Science at the Ruhr University Bochum and graduated with a Master of Science degree in 2018. He currently works as a research assistant at the Chair of Computer Science in Engineering at Ruhr University Bochum. His research there includes augmented and virtual reality in construction as well as human-machine interaction and safety in construction.

Markus König

Prof. Dr.-Ing. Markus König holds the Chair of Computing in Engineering at the Ruhr University Bochum since 2009. He has been teaching and researching digitization in the construction industry for many years. He was in charge of the scientific evaluation of the first BIM projects in Germany. In 2015, together with other experts, he developed the German road map for digital design and construction. Currently, he is deputy head of the German competence center for the digitalization of construction (BIM Germany). Prof. König published more than 250 technical papers and led several large research projects. For his scientific and practical achievements regarding Building Information Modeling, he received the Konrad Zuse Medal of the German Construction Industry in 2020.

Arno Mathis

Arno Mathis is responsible for digital service and training development at the Hilti headquarters in Liechtenstein. He obtained his Ph. D. in 2008 at the University of Twente in the Netherlands. It is the aim of Hilti to support players in the construction industry to make effective and efficient use of new digital solutions to enhance worker health & safety.

Acknowledgment

The authors are responsible for the content of this publication.

References

[1] Bangor, A. et al. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies. 4, 3 (2009), 114–123.Search in Google Scholar

[2] BG BAU 2021. Pressemappe zur Online-Pressekonferenz am 1. Juli 2021. (Jul. 2021), 12.Search in Google Scholar

[3] Brooke, J. 2013. SUS: a retrospective. Journal of usability studies. 8, 2 (2013), 29–40.Search in Google Scholar

[4] Brooke, J. 1996. SUS-A quick and dirty usability scale. in: Usability evaluation in industry. P.W. Jordan et al., eds. Taylor & Francis. 189–194.Search in Google Scholar

[5] Caine, K. 2016. Local Standards for Sample Size at CHI. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems – CHI ’16 (Santa Clara, California, USA, 2016), 981–992.10.1145/2858036.2858498Search in Google Scholar

[6] Deutsche Gesetzliche Unfallversicherung e. V. 2017. DGUV Information 209-002: Schleifen.Search in Google Scholar

[7] Du, J. et al. 2018. Zero latency: Real-time synchronization of BIM data in virtual reality for collaborative decision-making. Automation in Construction. 85, (Jan. 2018), 51–64. DOI:https://doi.org/10.1016/j.autcon.2017.10.009.10.1016/j.autcon.2017.10.009Search in Google Scholar

[8] Fiala, E. et al. 2020. Using Virtual Reality Simulations to Encourage Reflective Learning in Construction Workers. Learning and Collaboration Technologies. Human and Technology Ecosystems (Cham, 2020), 422–434.10.1007/978-3-030-50506-6_29Search in Google Scholar

[9] Filigenzi, M.T. et al. 2000. Virtual Reality for Mine Safety Training. Applied Occupational and Environmental Hygiene. 15, 6 (Jan. 2000), 465–469. DOI:https://doi.org/10.1080/104732200301232.10.1080/104732200301232Search in Google Scholar PubMed

[10] Gibb, A. et al. 2014. Construction accident causality: learning from different countries and differing consequences. Construction Management and Economics. 32, 5 (May 2014), 446–459. DOI:https://doi.org/10.1080/01446193.2014.907498.10.1080/01446193.2014.907498Search in Google Scholar

[11] Gibb, A.G.F. et al. 2006. What causes accidents? (2006). DOI:https://doi.org/10.1680/cien.2006.159.6.46.10.1680/cien.2006.159.6.46Search in Google Scholar

[12] Goulding, J. et al. 2012. Construction industry offsite production: A virtual reality interactive training environment prototype. Advanced Engineering Informatics. 26, 1 (Jan. 2012), 103–116. DOI:https://doi.org/10.1016/j.aei.2011.09.004.10.1016/j.aei.2011.09.004Search in Google Scholar

[13] Hilfert, T. and König, M. 2016. Low-cost virtual reality environment for engineering and construction. Visualization in Engineering. 4, 1 (Jan. 2016), 2. DOI:https://doi.org/10.1186/s40327-015-0031-5.10.1186/s40327-015-0031-5Search in Google Scholar

[14] Hinze, J.W. 1996. The distractions theory of accident causation. International conference (Rotterdam, 1996).Search in Google Scholar

[15] Hoover, J.D. 1974. Experiential Learning: Conceptualization and Definition. Developments in Business Simulation and Experiential Learning: Proceedings of the Annual ABSEL conference. 1, (Mar. 1974).Search in Google Scholar

[16] Isleyen, E. and Duzgun, H.S. 2019. Use of virtual reality in underground roof fall hazard assessment and risk mitigation. International Journal of Mining Science and Technology. 29, 4 (Jul. 2019), 603–607. DOI:https://doi.org/10.1016/j.ijmst.2019.06.003.10.1016/j.ijmst.2019.06.003Search in Google Scholar

[17] Jelonek, M. and Herrmann, T. 2019. Attentiveness for Potential Accidents at the Construction Site: Virtual Reality Test Environment with Tactile Warnings for Behavior Tests in Hazardous Situations. Proceedings of Mensch und Computer 2019 (New York, NY, USA, Sep. 2019), 649–653.10.1145/3340764.3344885Search in Google Scholar

[18] Kim, N. et al. 2021. Predicting workers’ inattentiveness to struck-by hazards by monitoring biosignals during a construction task: A virtual reality experiment. Advanced Engineering Informatics. 49, (Aug. 2021), 101359. DOI:https://doi.org/10.1016/j.aei.2021.101359.10.1016/j.aei.2021.101359Search in Google Scholar

[19] Kinateder, M. et al. 2014. Social influence on route choice in a virtual reality tunnel fire. Transportation Research Part F: Traffic Psychology and Behaviour. 26, (Sep. 2014), 116–125. DOI:https://doi.org/10.1016/j.trf.2014.06.003.10.1016/j.trf.2014.06.003Search in Google Scholar

[20] Kinateder, M. et al. 2014. Virtual reality for fire evacuation research. 2014 Federated Conference on Computer Science and Information Systems (Sep. 2014), 313–321.10.15439/2014F94Search in Google Scholar

[21] Kiral, I.A. et al. 2015. Enhancing the construction safety training by using virtual environment: V-SAFE. (Vancouver, 2015), 10.Search in Google Scholar

[22] Le, Q.T. et al. 2015. A Social Virtual Reality Based Construction Safety Education System for Experiential Learning. Journal of Intelligent & Robotic Systems. 79, 3–4 (Aug. 2015), 487–506. DOI:https://doi.org/10.1007/s10846-014-0112-z.10.1007/s10846-014-0112-zSearch in Google Scholar

[23] Lerner, D. et al. 2020. An Immersive Multi-User Virtual Reality for Emergency Simulation Training: Usability Study. JMIR Serious Games. 8, 3 (Jul. 2020), e18822. DOI:https://doi.org/10.2196/18822.10.2196/18822Search in Google Scholar PubMed PubMed Central

[24] Li, H. et al. 2012. Visualizing safety assessment by integrating the use of game technology. Automation in Construction. 22, (Mar. 2012), 498–505. DOI:https://doi.org/10.1016/j.autcon.2011.11.009.10.1016/j.autcon.2011.11.009Search in Google Scholar

[25] Li, X. et al. 2018. A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Automation in Construction. 86, (Feb. 2018), 150–162. DOI:https://doi.org/10.1016/j.autcon.2017.11.003.10.1016/j.autcon.2017.11.003Search in Google Scholar

[26] Lucas, J. and Thabet, W. 2008. Implementation and Evaluation of a VR Task-Based Training Tool for Conveyor Belt Safety Training. (2008), 23.Search in Google Scholar

[27] Mahalingam, A. and Levitt, R.E. 2007. Safety Issues on Global Projects. Journal of Construction Engineering and Management. 133, 7 (Jul. 2007), 506–516. DOI:https://doi.org/10.1061/(ASCE)0733-9364(2007)133:7(506).10.1061/(ASCE)0733-9364(2007)133:7(506)Search in Google Scholar

[28] Mayring, P. 2001. Combination and Integration of Qualitative and Quantitative Analysis. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research. 2, 1 (Feb. 2001).Search in Google Scholar

[29] Mayring, P. 2010. Qualitative Inhaltsanalyse. Beltz.10.1007/978-3-531-92052-8_42Search in Google Scholar

[30] McComas, J. et al. 2002. Effectiveness of Virtual Reality for Teaching Pedestrian Safety. CyberPsychology & Behavior. 5, 3 (Jun. 2002), 185–190. DOI:https://doi.org/10.1089/109493102760147150.10.1089/109493102760147150Search in Google Scholar PubMed

[31] Nakai, A. et al. 2014. The Experience-Based Safety Training System Using Vr Technology for Chemical Plant. International Journal of Advanced Computer Science and Applications. 5, 11 (2014). DOI:https://doi.org/10.14569/IJACSA.2014.051111.10.14569/IJACSA.2014.051111Search in Google Scholar

[32] Namian Mostafa et al. 2018. Effect of Distraction on Hazard Recognition and Safety Risk Perception. Journal of Construction Engineering and Management. 144, 4 (Apr. 2018), 04018008. DOI:https://doi.org/10.1061/(ASCE)CO.1943-7862.0001459.10.1061/(ASCE)CO.1943-7862.0001459Search in Google Scholar

[33] Perez, B.Z. et al. 2007. Developing a Virtual Environment for Safety Training. Electronics, Robotics and Automotive Mechanics Conference (CERMA 2007) (Sep. 2007), 545–550.10.1109/CERMA.2007.4367743Search in Google Scholar

[34] Perlman, A. et al. 2014. Hazard recognition and risk perception in construction. Safety Science. 64, (Apr. 2014), 22–31. DOI:https://doi.org/10.1016/j.ssci.2013.11.019.10.1016/j.ssci.2013.11.019Search in Google Scholar

[35] Rey-Becerra, E. et al. 2021. The effectiveness of virtual safety training in work at heights: A literature review. Applied Ergonomics. 94, (Jul. 2021), 103419. DOI:https://doi.org/10.1016/j.apergo.2021.103419.10.1016/j.apergo.2021.103419Search in Google Scholar PubMed

[36] Ritzmann, S. et al. 2014. The Training Evaluation Inventory (TEI) – Evaluation of Training Design and Measurement of Training Outcomes for Predicting Training Success. Vocations and Learning. 7, 1 (Apr. 2014), 41–73. DOI:https://doi.org/10.1007/s12186-013-9106-4.10.1007/s12186-013-9106-4Search in Google Scholar

[37] Sacks, R. et al. 2013. Construction safety training using immersive virtual reality. Construction Management and Economics. 31, 9 (Sep. 2013), 1005–1017. DOI:https://doi.org/10.1080/01446193.2013.828844.10.1080/01446193.2013.828844Search in Google Scholar

[38] Schrepp, M. et al. 2017. Construction of a Benchmark for the User Experience Questionnaire (UEQ). International Journal of Interactive Multimedia and Artificial Intelligence. 4, 4 (2017), 40. DOI:https://doi.org/10.9781/ijimai.2017.445.10.9781/ijimai.2017.445Search in Google Scholar

[39] Schubert, T. et al. 2001. The Experience of Presence: Factor Analytic Insights. Presence: Teleoperators and Virtual Environments. 10, 3 (Jun. 2001), 266–281. DOI:https://doi.org/10.1162/105474601300343603.10.1162/105474601300343603Search in Google Scholar

[40] Teizer, J. et al. 2021. Digitalisierung der Arbeitssicherheit auf Baustellen. Arbeit in der digitalisierten Welt: Praxisbeispiele und Gestaltungslösungen aus dem BMBF-Förderschwerpunkt. W. Bauer et al., eds. Springer. 399–414.10.1007/978-3-662-62215-5_26Search in Google Scholar

[41] User Experience Questionnaire (UEQ): 2018. https://www.ueq-online.org/. Accessed: 2019-09-26.Search in Google Scholar

[42] Wilkins, J.R. 2011. Construction workers’ perceptions of health and safety training programmes. Construction Management and Economics. 29, 10 (Oct. 2011), 1017–1026. DOI:https://doi.org/10.1080/01446193.2011.633538.10.1080/01446193.2011.633538Search in Google Scholar

[43] Wolf, M. et al. 2022. Investigating hazard recognition in augmented virtuality for personalized feedback in construction safety education and training. Advanced Engineering Informatics. 51, (Jan. 2022), 101469. DOI:https://doi.org/10.1016/j.aei.2021.101469.10.1016/j.aei.2021.101469Search in Google Scholar

[44] Xie, H. et al. Development of a Virtual Reality Safety-Training System for Construction Workers. 9.Search in Google Scholar

[45] Zaker, R. and Coloma, E. 2018. Virtual reality-integrated workflow in BIM-enabled projects collaboration and design review: a case study. Visualization in Engineering. 6, 1 (Sep. 2018), 4. DOI:https://doi.org/10.1186/s40327-018-0065-6.10.1186/s40327-018-0065-6Search in Google Scholar

[46] Zhao, D. and Lucas, J. 2015. Virtual reality simulation for construction safety promotion. International Journal of Injury Control and Safety Promotion. 22, 1 (Jan. 2015), 57–67. DOI:https://doi.org/10.1080/17457300.2013.861853.10.1080/17457300.2013.861853Search in Google Scholar PubMed

Published Online: 2022-07-19
Published in Print: 2022-08-26

© 2022 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 11.12.2023 from https://www.degruyter.com/document/doi/10.1515/icom-2022-0006/html
Scroll to top button