Get ready for automated driving using Virtual Reality

Get ready for automated driving using Virtual Reality Daniele Sportillo, Alexis Paljic, Luciano Ojeda To cite this version: Daniele Sportillo, Alexis Paljic, Luciano Ojeda. Get ready for automated driving using Virtual Reality. Accident Analysis and Prevention, Elsevier, 2018, 118, pp.102-113. ฀10.1016/j.aap.2018.06.003฀. ฀hal01858450฀ HAL Id: hal-01858450 https://hal.archives-ouvertes.fr/hal-01858450 Submitted on 21 Aug 2018 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Get Ready for Automated Driving using Virtual Reality Daniele Sportilloa,b,∗, Alexis Paljica , Luciano Ojedab a MINES ParisTech, PSL Research University, Centre for robotics, 60 Bd St Michel 75006 Paris, France b PSA Group, Technical Center of Velizy Abstract In conditionally automated vehicles, drivers can engage in secondary activities while traveling to their destination. However, drivers are required to appropriately respond, in a limited amount of time, to a take-over request when the system reaches its functional boundaries. Interacting with the car in the proper way from the first ride is crucial for car and road safety in general. For this reason, it is necessary to train drivers in a risk-free environment by providing them the best practice to use these complex systems. In this context, Virtual Reality (VR) systems represent a promising training and learning tool to properly familiarize drivers with the automated vehicle and allow them to interact with the novel equipment involved. In addition, Head-Mounted Display (HMD)-based VR (light VR) would allow for the easy deployment of such training systems in driving schools or car dealerships. In this study, the effectiveness of a light Virtual Reality training program for acquiring interaction skills in automated cars was investigated. The effectiveness of this training was compared to a user manual and a fixed-base simulator with respect to both objective and self-reported measures. Sixty subjects were randomly assigned to one of the systems in which they went through a training phase followed by a test drive in a high-end driving simulator. Results show that the training system affects the take-over performances. Moreover, self-reported measures indicate that the light VR training is preferred with respect to the other systems. Finally, another important outcome of this research is the evidence that VR plays a strategic role in the definition of the set of metrics for profiling proper driver interaction with the automated vehicle. Keywords: conditionally automated vehicles, virtual reality, head-mounted display, take-over request, training 1. Introduction 5 10 15 20 Imagine you are reading this article in your car as you 25 drive on the highway. Suddenly, your car asks you to “take-over”. What would you do? At the time of writing, this scenario breaks numerous laws and is potentially very dangerous. In the future, it would not only be legal and safe, but you would likely know how to react to 30 your car’s demands to hand over control, keeping yourself, passengers, and other vehicles out of harm’s way. In future automated vehicles the above situation would be fairly common. In particular, conditionally automated vehicles (SAE Level-3 [1]) do not require drivers to con- 35 stantly monitor their driving environment; they can, therefore, engage in secondary activities such as reading, writing emails and watching videos. However, when the automated system encounters unexpected situations, it will assume that drivers who are sufficiently warned will ade- 40 quately respond to a take-over request. The reestablishment of the driving context (i.e. rapid onboarding) is one challenge of conditionally automated vehicles [2] for the car industry. The revolution of the driving activity, the complexity of these new systems and 45 ∗ Corresponding author Email address: daniele.sportillo@mpsa.com, mines-paristech.fr (Daniele Sportillo) Preprint submitted to Accident Analysis & Prevention the variety of situations that the driver can face requires that drivers must have already acquired the core skills necessary to securely interact with the automated car before their first ride. Establishing drivers’ role and avoiding confusion [3] is crucial for the safety of both the drivers themselves and other road users. At present, a vehicle’s functionalities are demonstrated to customers via an informal presentation by the car dealer during the hand-over process; for further information, customers are required to read the car owner’s manual. For an automated vehicle, these traditional procedures would not be feasible to familiarize the new car owner with the automated system, primarily because the acquisition of skills by the customer is not ensured. In addition, car dealers themselves must be trained and kept up to date of each new version of the system. In this context, Virtual Reality (VR) constitutes a potentially valuable learning and skill assessment tool which would allow drivers to familiarize themselves with the automated vehicle and interact with the novel equipment involved in a free-risk environment. VR allows for the possibility of encountering dangerous driving conditions without putting the driver at physical risk and enable the controllability and reproducibility of the scenario conditions [4]. VR has usually been associated with high costs and June 5, 2018 50 55 60 65 70 huge computational power. For these reasons immersive training based on CAVEs or Head-Mounted Displays has105 until now been prohibitive in mainstream settings. However, in recent years, technological progress and the involvement of dominant technology companies has allowed the development of affordable VR devices. The objective of this research is to explore the poten-110 tial of the role of light Virtual Reality systems, in particular, for the acquisition of skills for the Transfer of Control (ToC) in highly automated cars. By using the adjective light, we want to mark the difference between VR systems that are portable and/or easy to set up (HMDs, mobile115 VR) and systems that are cumbersome and require dedicated space to operate (CAVE systems). The idea is that thanks to the portability and the cost-effectiveness, light VR systems could be easily deployed in car dealerships to train a large amount of people in an immersive environ-120 ment in a safe and reliable way. The light VR system proposed in this paper consists of a consumer HMD and a racing wheel. This paper aims to compare the effectiveness of a training program based on this system with a user manual and with a fixed-base driv-125 ing simulator. To validate the light VR system, user performances are evaluated during a test drive in a high-end driving simulator and self-reported measures are collected via questionnaires. 130 75 80 85 90 95 100 1.1. Related work Virtual Reality has been extensively used to train professionals and non-professionals in various domains. The unique characteristics of learning in the 3D environment provided by immersive VR systems such as CAVEs or135 HMDs, can enable learning tasks that are not possible or not as effective in 2D environments provided by traditional desktop monitors. Dalgarno et al. [5] highlighted the benefits of this kind of 3D Virtual Learning Environments (3D VLEs) by proposing a model based on their distinctive fea-140 tures such as the representational fidelity and the learner interaction. More in detail, HMD-based VR turns out to be more effective when compared to other training systems, for a wide range of applications such as surgery [6] (HMD com-145 pared to video trainer), aircraft visual inspection [7] (HMD compared to PC-based training tool), power production [8] (HMD compared to traditional training), mining industry [9] (HMD compared to screen-based and projector-base 150 training). When it comes to Driving Simulation (DS), VR is used to study several aspects of the driving task. In this context, moving-base simulators [10] are preferable to fixedbase simulators [11, 12] for their closer approach to real155 world driving [13]. By investigating the physical, behavioral and cognitive validity of these kind of simulators with respect to the real driving task [11], it has been also shown that DS can be a useful tool for the initial resumption of driving, because 160 2 it helps to avoid stress that may lead to task failure or deterioration in performance. Although most of the studies in DS uses static screens as the display system, recent studies prove that HMDbased DS leads to similar physiological response and driving performance when compared to stereoscopic 3D or 2D screens [14]. Taheri et al. [15] presented a VR DS system composed of HMD, steering wheel and pedals to analyze drivers’ characteristics; Goedicke et al. [16] instead proposed an implementation of an HMD in a real car to simulate automated driving as the vehicle travels on a road. Even if the steering wheel is the most used driving interface, novel HMD systems usually come with wireless 6-DoF controllers which can be used to control a virtual car. In a pilot study, Sportillo et al. [17] compare steering wheel and controller-based interaction in HMD-based driving simulators. The authors conclude that even though objective measures do not provide decisive parameters for determining the most adequate interaction modality, selfreport indicators show a significant difference in favor of the steering wheel. Among other things, DS provides the opportunity to implement, in a forgiving environment, critical scenarios and hazardous situations which are ethically not possible to evaluate on real roads [18]. For this reason and to overcome the limited availability of physical prototypes for research purposes, DS is extensively used for studies on automated vehicles to design future automotive HMI [19] for Take-Over Requests (TORs) and to investigate the behavioral responses during the transition from automated to manual control [20]. A research area that is gaining interest in the automated driving community concerns the impact of nondriving activities on take-over performance. To study driver’s distraction during automated driving, researchers generally use standardized and naturalistic tasks. Standardized tasks (such as the cognitive n-back task [21], the SuRT task [21, 22], the Twenty Questions Task (TQT) [23]) provide experimental control, but they do not usually correspond to what the driver will do in the vehicle. Naturalistic tasks, instead, provide ecological validity, but they could introduce experimental bias. Important findings were found by Zeeb et al. [24] who studied how visual-cognitive load impacts take-over performance by examining the engagement in three different naturalistic secondary tasks (writing an email, reading a news text, and watching a video clip). The authors found that the drivers’ engagement in secondary tasks only slightly affected the time required to regain the control of the vehicle, but non-distracted drivers performed better in the lane-keeping task. Most of the studies in this domain implement safetycritical take-over scenarios caused by an obstacle (usually a broken down vehicle) on the current lane [24, 17, 21, 25, 23] and non-critical scenarios caused by the absence of lane markings [24, 26]. To ensure security and to succeed in the take-over process, it is important to understand how much time before a system boundary a driver who is out 165 170 175 180 185 190 195 200 205 210 215 of the loop should be warned. Gold et al. [22] indicate that shorter TOR-time leads to a faster but worse reaction. However, assessing the quality of the take-over per-220 formance remains an open problem. Reaction times (such as gaze reaction time, hands on wheel time, and intervention time) are analyzed [21]. Time To Collision, lateral accelerations and minimum clearance towards are objective metrics used in obstacle avoidance scenarios [21]. Con-225 cerning subjective measures, drivers are usually asked to reply to questionnaires: the Driver Skill Inventory (DSI) [27] and Driver Behaviour Questionnaire (DBQ) [28] have been largely used to evaluate the self-assessment of driving skills [29] in the last decades. In recent studies, questionnaires have been used to investigate the importance of initial skilling and to predict the deskilling in automated230 vehicles [30]. In the same field, surveys have also been used to evaluate usefulness and satisfaction of take-over requests [31]. In the above studies it is not always clear how participants were taught to use the automated system. Zeeb et235 al. [24] used a traditional approach that provided the participants with a description of the system, the functional boundaries and the alert notifications. In the vehicle, participants were also instructed to activate and deactivate the automated driving system. This approach could not240 be adapted to the real case because it does not ensure the correct acquisition of knowledge; thus, the drivers would not be sufficiently skilled to safely respond to a take-over request. In other studies participants could freely practice in the high-end driving simulator before the actual test drive [22]. This solution would not be feasible in terms245 of costs, space and maintenance because it would require every car dealership to be equipped with a simulator. A lighter VR system, such as the one proposed in this paper, could instead be more easily deployed and used for 250 training purposes at a much lower cost. Payre et. al [26] addressed the problem of drivers’ training in an automated car by comparing two types of training: a simple training based only on practice in a driving simulator and an elaborated training which included a text, a tutorial video and a more elaborated practice in255 the simulator. They found that participants in the elaborated training group trusted more the automated driving and were able to take-over faster than those in the simple training group. Automated car research also has relevance in the field260 of aviation [32], and in particular in studies concerning flight simulation for pilot training [33]. Although this kind of training is targeted towards professionals, important findings from this research include the occurrence of positive transfer and the fact that abstracted rendering265 simulators allow people to learn better than with the real thing [34]. Pilots trained on a simulator are thus able to co-pilot a craft immediately after their simulation training [33]. However, it is crucial that the training practices allow for the generalization of the skills acquired in the270 virtual environment and not only for an application of the 3 rote-memorized skills specific to the training situation [35]. The considerable findings from aviation and the intense scientific production in recent years suggest that the transition of control in automated cars is a valuable research topic worth investigating from the design stage to the final implementation of the new systems. Moreover, the compelling need and interest of the car industry to train a large amount of people in a reliable and cost-effective way, without compromising security, make light virtual reality system tools a promising solution for this purpose. 2. Methods This study contained two parts: training and test drive. The aim of the training was to introduce the principles of the Level 3 Automated Driving System (ADS)-equipped vehicle, present the novel Human-Machine Interface (HMI), help the drivers to localize the HMI in the vehicle, and describe the actions to perform in order to appropriately respond to unplanned requests to intervene. The betweensubject study with 60 participants was designed in order to compare a light Virtual Reality system to a user manual and a fixed-base driving simulator in terms of training effectiveness evaluated through a test drive. The test drive required the application of knowledge and skills acquired during the training. 2.1. The target vehicle This study takes into account Level 3 (Conditional Driving Automation) automated vehicles. In this level of automation the ADS performs the Dynamic Driving Task (DDT) with the expectation that the human driver is receptive to a Take-Over Request (TOR), also known as request to intervene, and will respond appropriately. The DDT includes [1] lateral vehicle motion control via steering; longitudinal vehicle motion control via acceleration and deceleration; monitoring the driving environment via object and event detection, recognition, classification, and response preparation; object and event response execution; maneuver planning; enhancing conspicuity via lighting, signaling and gesturing, etc. For a more detailed taxonomy and description please refer to the Recommended Practice by SAE [1]. A TOR is a notification by the ADS to a human driver that s/he should promptly begin or resume performance of the DDT. Unplanned TORs are prompted by the ADS when it reaches system boundaries because of unpredictable and potentially hazardous situations that it cannot handle. These situations could be represented by an obstacle on the road, missing road markings or system failure. The target vehicle provided two driving modes on highways: Manual Driving and Conditionally Automated Driving. The vehicle was not expected to execute automatic lane changes. In the implementation the vehicle had 5 possible states: (a) Manual driving: the human driver is in charge of all the aspects of the dynamic driving task (execution of steering and acceleration/deceleration) (b) ADS available: the human driver can transfer control to the ADS, by operating the HMI. 275 280 285 290 295 300 305 310 315 (c) ADS enabled: the ADS performs all the aspects of the dynamic driving task, namely the control of the longitudinal and the lateral guidance. (d) Take-over request: the ADS reaches a system boundary and thus is no longer able to perform the dynamic driving task. The human driver is notified with a visual-auditory alert indicating the time budget s/he has to take-over. Figure 1: The HUD of the target vehicle and the symbols that represent the states of the vehicle: (a) manual driving, (b) autonomous driving system available (c) autonomous driving system activated, (d) take over request with countdown, (e) emergency brake. The arrows represent the possible transition between the states. (e) Emergency brake: the human driver does not take over in the allotted amount of time and the vehicle325 performs an emergency brake on the lane. The alert continues until the control is transferred back to the human driver. during the automated driving and the best practice to respond to a take-over request. For all the participants, the training program started with an introduction video that briefly presented the main functionalities of a Level 3 ADS-equipped car. The video was displayed onto a different support according to the display system used during the training. In the study three different training systems were compared (Figure 2) When the ADS was activated, the car kept a constant 330 longitudinal speed of 90 km/h, accelerating or decelerating if the speed at the activation was respectively lower or higher. 2.1.1. Human-Machine Interface The Human-Machine Interface in the target vehicle consisted of a head-up display (HUD) and a button on335 the steering wheel. The HUD (see Figure 1) showed information about current speed, speed limit, distance traveled and current state of the vehicle. In Figure 1, the different symbols representing the states of the system are illustrated; the arrows indicate the possible transition between states. The symbols are taken from previous studies [36].340 The background color of the HUD also changed according to the current state of the vehicle. Take-over requests were notified to the human driver with a visual-auditory alert. The visual alert consisted of the symbol in Figure 1d with a countdown indicating the345 budget of time available to take over. The auditory alert was a 0.7 second beep looped every second. In the implementation of the automated driving system, the human driver could activate the ADS (if available) by pushing a button on the steering wheel. When the ADS was enabled, at any time the human driver could de-350 activate it and immediately take back control. This could be done in three ways: (i) pushing the same button on the steering wheel, (ii) using the brake pedal, or (iii) using the accelerator pedal and the steering wheel. Since all the participants were French speakers, all the355 text in the HMI was displayed in French to avoid language comprehension problems. 2.2. The training 320 The aim of the training was to teach drivers how to in-360 teract with automated cars in three situations: the manual mode, automated mode and the take-over request. To do so, the training introduced the participants to the HMI for each situation, the actions they were free to perform 4 • a User Manual (UM) displayed on a laptop; • a Fixed-Base driving simulator (FB) with real cockpit and controls (pedals and steering wheel); • a Light Virtual Reality (LVR) system consisting of a Head-mounted display (HMD) and a game racing wheel. These systems differed in terms of level of immersion and interaction they provided. “Immersion” refers to the technological capabilities a system is able to deliver from an objective point of view [37]. “Interaction” refers to the modality through which the user can perform actions in the virtual environment. Immersion and interaction do not apply to the User Manual group. The fixed-base driving simulator and the LVR system shared the same interaction modalities, but the level of immersion was different. In what follows, the three systems are described. 2.2.1. User manual training The user manual (UM) consisted of a slide presentation displayed on a 13.3” screen of a laptop computer (Figure 2a). First, the introduction video was played. Then, the participants were asked to carefully read each of the 8 slides and to go to the next one when they felt ready. They did not have any time limit. The slides used text and images to present the actions to be performed during the manual driving, the automated driving and the take-over requests. For each situation the correspondent icons were also presented. An animated slide was included to show how to activate the automated driving. This system represented the non-immersive and noninteracting training environment. The participants could only browse forward and backward the slides, with no time a b c Figure 2: The three training systems: (a) the user manual displayed on the laptop computer, (b) the fixed-base driving simulator, (c) the light VR system 365 370 375 380 385 390 395 limit; however, they were not involved in a driving situa-400 tion and they could not practice the action required with the real equipment. 2.2.2. Fixed-base simulator The fixed-base simulator (FB) consisted of an actual405 car cockpit including a driving seat, a dashboard, a forcefeedback steering wheel and a set of pedals (Figure 2b). All of these components were real components of a Citroen C3; this allowed participants to have a more natural interaction with the driving controls. A 9.7” tablet used410 by the driver to perform the secondary activity was placed in the center console. To display the virtual environment a 65” plasma screen was positioned behind the cockpit at 1.5m from the driver. This simulator represented the low-immersion training415 environment. The limited size of the screen did not allow the implementation a 1:1 scale between the virtual and the real world. Also, another implication of the reduced field of view was the lack of isolation for the participant who was surrounded by the experimental room during the420 training. 2.2.3. Light Virtual Reality system The light VR system (LVR) included an HMD as a display system, and a Logitech G25 Racing Wheel as driving425 system (Figure 2c). The HMD was an HTC Vive which provides stereoscopic vision at 90 FPS, 2160 x 1200 (1080 x 1200 per eye) resolution, a field of view of 110 degrees and low-latency positional tracking. Spatial sound was presented via headphones. Thanks to these features, the430 LVR system represented the high-immersion training system. The trainee was totally surrounded by the virtual environment, but once wearing the headset s/he lost the possibility to see any part of his/her own body. Although the field of view of the HTC Vive is not comparable with the human vision, the design choices for the training sce-435 nario (no traffic, straight lane) helped to reduce the stimuli 5 in the peripheral vision, which is one of the causes of simulator sickness [38]. At the beginning, the participants were immersed in a virtual room with white walls. This room represented a transitional environment from the real world to the virtual learning activity. A transparent effect was applied (Figure 3a) to the car to ease the transition to the virtual world. The introduction video was displayed on the front wall. We hypothesized that, at the beginning of the experiment, a simpler environment with a few visual elements could help participants better accept the system [39]. The purpose of this environment was twofold. First, novices of Virtual Reality and participants who were using an HMD for the first time could become familiar with the new system by experiencing the effects of their actions (head rotation, head movement) on the system. Second, since the participants could not see their hands, they could become aware of the car controls, identifying the position of the steering wheel, the button on the steering wheel, and the pedals. The participants were located inside a virtual model of a Citröen DS3 car [40]. To have a spatial correspondence between the real steering wheel and the virtual one, the steering wheel inside the virtual car was a 3D model of the real racing wheel with which the participants were interacting. Moreover, the position and the movements of the virtual model corresponded to the real one, allowing for co-located manipulation. After this phase of acclimatization, the virtual environment evolved into the training environment. The car was displayed with as much realism as possible. Thus, the participant performed the training described in Table 1. 2.2.4. The Virtual Learning Environment For the training using the LVR system and the fixedbase driving simulator, a step-by-step tutorial was developed in the form of a Virtual Learning Environment (VLE). The VLE provided the same information and stim- 440 445 450 455 460 465 470 475 480 485 490 Table 1: The tasks in the Virtual Learning Environment. Each action was notified to the driver with visual and auditory messages. uli to the two groups of participants, except for the differences due to the nature and the limits of the two systems involved. The characteristics of the target vehicle described in Section 2.1 were implemented in the VLE. The task of the participants consisted of interactions with the car following the instruction of a virtual vocal assistant. The messages announced by the assistant were also displayed on a yellow panel in front of the trainee (Figure 3c). The panel appeared when the user intervention was required, and disappeared as soon as the trainee performed the required actions. No other actions were possible other than the required one. The driving scenario was a straight 2-lane road delimited by guardrails. No traffic was implemented. Only trees were placed on the roadside. A simple environment was specifically chosen to focus participants on the training task without any distractions, and to reduce the peripheral optical flow which can contribute to simulation sickness [41]. The training steps are described in Table 1. Before the driving scenario, an acclimatization virtual environment was proposed to the participants to help them locate and identify the controls of the car. Secondary activity. This training also included a secondary activity that required the use of a tablet (a real one in the case of the fixed-base simulator, a virtual one in the case of LVR system). The tablet was used to distract the human driver from the driving task during the automated driving. The distraction task was the same for all the participants and consisted of a video of a TEDx Talk in French. The participants were asked, but not forced, to look at the tablet. The video was automatically played when the automated system was enabled and paused during the manual driving and the take-over requests. 2.3. The test drive After the training, the participant performed a test drive designed to evaluate their performance in a more realistic driving scenario. The system used for this purpose was a high-end driving simulator consisting of the front part of a real car surrounded by a panoramic display (Figure 4). The display was placed 2.5m from the driver and covered a field of view of 170 degrees. Three threechip DLP projectors displayed the scene. The rear part of the car was substituted with a monitor that displayed the virtual environment from the rear window. The lateral495 mirrors consisted of two LCD displays as well. The cockpit was also equipped with a microphone to communicate with the experimenter and 4 cameras to record the scene inside the car. Data including position, speed and acceleration of the car, and current driving mode were recorded. Inside the car, a 10.8 inch tablet was placed in the500 center console. It provided 9 different secondary activities: 3 games (a solitary, 2048, Simon) and 6 videos (3 talks, 2 movies and 1 movie trailer). The tablet was only available 6 0 km Manual driving The trainee drove the car in the manual mode to familiarize with the simulator 1 km Delegation of driving The trainee was required to activate the automated driving system by pushing the button on the steering wheel 2 km Control take-over The trainee was required to switch back to manual mode by pushing the button on the steering wheel 2.75 km Delegation of driving 4.25 km Control take-over The trainee was required to switch back to manual mode by using the accelerator pedal and the steering wheel 5 km Delegation of driving 5.75 km Accelerator override The trainee was required to use the accelerator pedal in order to increase the speed of the vehicle without deactivate the automated driving system. 6.5 km Steering override The trainee was asked to use the steering wheel to perform a lane change task without deactivate the automated driving system. 7.5 km Take-over Request A 30-second TOR was issued. The trainee was assisted during the take-over phase 8.25 km Delegation of driving 9.25 km Take-over Request A 10-second TOR was issued and the trainee had to take-back without any assistance. An obstacle was placed 300 meter after. An emergency brake was performed if the trainee did not take-back in time. 10 km Free driving The trainee was free to practice the delegation of driving and to take-back. 11 km End of the training during autonomous driving and it displayed the message “Take back control” during the requests to intervene. Before starting the test, participants were instructed about the use of the equipment inside the car and were shown the button to activate/deactivate the automated driving system. The choice of a fixed rather than a moving-base simulator was justified by the driving scenario which did not provide important lateral or longitudinal acceleration. 2.3.1. Driving scenario The driving scenario of the test drive represented a dual carriageway with two lanes in each direction. Dense traffic was added to both directions. The aim of the test drive was to investigate the skills acquired by the participants a b c Figure 3: The Virtual Learning Environment. In (a) the familiarization environment. The car is displayed with a transparent effect and the panel on the front wall shows the indicators for the accelerator and brake pedal and for the steering wheel. In (b) a post-production illustration of a participant immersed in the virtual environment. In (c) a view of the interior of the car with the training message (on the yellow panel) and the virtual tablet used for the secondary activity a b Figure 4: The test drive simulator: (a) the real cabin with the 170 degree panoramic display and (b) a view of the cabin interior 525 Figure 5: The test drive scenario with the three situations that provoked the TORs: (A) stationary car on the lane, (B) loss of ground marking, (C) sensor failure 530 505 510 515 520 during the training and their reaction when a non-planned take-over request was issued. For this purpose, 3 requests to intervene Figure 5 were issued during the test drive: 535 (A) a 10-second TOR caused by a road narrowing provoked by a stationary car on the right lane; this situation (Obstacle) required the driver to brake and to change lane in order to avoid the obstacle. the simulator, the three TORs were issued after an autonomous driving phase, at 11.5 km, 19km and 23km. During the autonomous driving, participants were asked to engage in one of the secondary activities proposed by the tablet. 2.4. Measures Defining the quality of take-over is not an easy exercise, because assessing the ability to drive or to operate an automated vehicle requires the evaluation of various aspects related or not to the actual driving task. In literature there exists a set of well-known parameters which can be used to evaluate performance in driving scenarios like the once used in the test-drive. To evaluate the training systems and the learning environment, objective and self-reported measures were collected and treated anonymously. 2.4.1. Self-reported measures (B) a 10-second TOR caused by a loss of road marking540 In total 6 different questionnaires were proposed to (Road Marking); this situation required the driver to the participants. A demographic questionnaire (containhold the vehicle inside the lane. ing also questions about driving habits, familiarity with Virtual Reality and previous experiences with driving sim(C) a 5-second TOR caused by a sensor failure (Failulators) was administered at the begininning of the study ure); this situation did not require any specific ac545 along with a survey about opinion concerning automated tions from the driver. cars. This last survey was also administered at the end of the study. To evaluate the appreciation of the training, To control order effects, the arrival order of TOR A and participants were asked to answer to 10 questions survey TOR B was randomized. TOR C was always issued as the and to evaluate graphical and physical realism of the Virlast one in order. The test drive lasted for about 20 minutes time dur-550 tual Environment (only for FB and LVR groups). After the training, the Simulator Sickness Questionnaire [42] was ing which the participants drove for 24 km. After a first phase of manual driving (4km) to familiarize drivers with 7 555 560 administered to the LVR group. After the test-drive, all600 Training (15 minutes). The training contained two parts: the groups answered to a final questionnaire. the introductory video (2 minutes) and the actual training (slides for the user manual group, and the Virtual Learning 2.4.2. Objective measures Environment for the fixed-base and light VR system). The training for the user manual group generally lasted for less To evaluate the take over quality and the state of the driver during the autonomous phase, objective measures605 time with respect to the LVE one. were used as performance factors in the test drive with the Second questionnaire (5 minutes). The participants filled high-end simulator. According to the take over situation, out questionnaire C. Participants of the VLE group filled both raw data from the simulator (such as position and out questionnaire D. Participant of the LVR group filled speed of the car, current driving mode, etc.) and video out also the questionnaire E. feeds were used to assess the following variables: • Reaction time (measured in seconds), the elapsed610 Test Drive (20 minutes). The participants drove in the high-end simulator. time from TOR until the driver takes back control. 565 570 • Maximum deviation from the lane center (measured in meters), within an interval of 30s after the takeover request. Third questionnaire (5 minutes). The participants completed questionnaires F and B. • Time To Collision (measured in seconds), “the time required for two vehicles to collide if they continue at their present speed and on the same path”. This615 measure was used to evaluate the evasive maneuver to avoid the stationary car [21]. 3. Results • Stress and confidence in the vehicle, during the automated driving phases. 620 2.5. Participants 575 580 585 Sixty subjects participated in the experiment. The participants included 30 females (50%) and 30 males (50%) aged between 22 and 71 (M = 43, SD = 14). Three groups of age were identified: the first group contained625 participants aged between 22 and 34 years old (7 males, 11 females); the second group participants aged between 35 and 54 (14 males, 9 females); the third group participants older than 55 (9 males, 10 females). They were randomly assigned to one of the system in which they would630 be trained. The three groups contained 20 subjects each. All the subjects were volunteers recruited by a company specialized in hiring consumer tests participants and had a valid driving license. At the end of the experiment, each participant was rewarded with a 40 euros voucher. 635 590 595 2.6. Procedures The duration of the full experiment was about 60 minutes for each participant. Participants were divided into three groups of 20. Each group underwent training with one of the systems described above. The study consisted640 of the following phases: Introduction (10 minutes). The participants were welcomed and informed in detail about the purpose of the study. They signed the consent form. 645 First questionnaire (5 minutes). The participants completed questionnaires A and B. 8 All the participants completed the experiment. Selfreport and performance variables were tested for group differences using ANOVAs (and Tukey’s HSD test for pairwise comparison) for continuous normally distributed data and Kruskal-Wallis (and Fisher’s LSD for pairwise comparison) test for ordinal, categorical and non-normally distributed data. Paired t-test was used for PrePost questionnaires. The significance level of 5% was chosen for all the tests. 3.1. Self-report measures Self-reported measures were collected through a set of questions at the beginning of the test, after the training and after the test drive. The measures of user appreciation and simulator sickness were tested for group differences using Kruskal-Wallis test. In case of significant differences among the three groups (p < 0.05), the Fisher’s LSD test was used to identify which pairs of means were significantly different, and which were not. The measure of confidence on automated vehicles was tested using a paired t-test. 3.1.1. Appreciation of the training To evaluate the appreciation of the training the participants filled out a 10-question survey containing questions about perceived usefulness, easiness, pleasantness, realism and so on. The Likert results are reported in Figure 6. The LVR scored better in all the questions, and in 4 of them the difference was significant. Moreover, to have a general score, all the questions were summed up. Up to a total of 50 points (the highest the better), results showed that the LVR scored significantly better (M = 43) than both the fixed-base simulator (M = 40, p < 0.05) and the user manual (M = 39.5, p < 0.05). The results of the survey about realism and comfort are reported in Figure 7. Table 2: Demographic features distributed across the different systems System UM FB LVR Total Gender (F/M) 11/9 10/10 9/11 30/30 Age y (SD) 45 (12.9) 46.9 (15.5) 43.5 (13.9) 45.1 (14) Age Group 5/9/6 7/6/7 6/8/6 18/23/19 Car with Cruise Control? Yes (no use) / No 11(2) / 8 16(4) / 4 11(1) / 9 38(7) / 21 First time in a driving simulator (Y/N) 16/3 16/4 14/6 46/13 Figure 6: Likert responses to the questionnaire of training appreciation. Figure 7: Likert responses to the realism survey for FB and LVR groups 650 655 660 3.1.2. Simulator Sickness Questionnaire (SSQ) The SSQ was filled out only by the participants who performed the training with the LVR. The Total Score (TS) and the subscales relative to Nausea, Oculomotor and Discomfort symptoms were calculated according to665 the formulas described by Kennedy et. al [42]. Results are reported in Table 3 and Figure 8. According to the categorization of SSQ proposed by Kennedy et. al. [43], 50% of the subjects reported no symptoms (TS = 0) or minimal symptoms (TS < 10). The highest scores were reported by a participant affected by monocular vision impairment670 (TS = 71) and a participant affected by kinetosis also in traditional vehicles (TS = 97.24). However, they as well as all the other participants were able to complete the training (no dropouts occurred). There were no significant differences with respect to participants’ age or gen-675 der. Analyzing the subscales, the Disorientation subscale 9 Figure 8: Results of SSQ scores (Nausea, Oculomotor, Disorientation subscales and Total) for the LVR group. In orange, the percentile graph. The vertical blue lines represent the value of SSQ if all the symptoms were reported as “slight” on that subscale. (with symptoms related to vestibular disturbances such as dizziness and vertigo) registered the highest scores. This result was expected and is mainly due to the nature of the HMD, which causes conflicts between the vestibular and the visual signal. 3.1.3. Pre-post Questionnaire on automated vehicles At the beginning of the test, participants were asked to give a score from 1 to 5 to a set of 8 sentences to express their opinions on automated vehicles. After the test drive, they reply to the same questionnaire for the second time. The questionnaire contained sentences about confidence in the actions performed by the automated system, perceived security, usefulness in the society and so on. The Wilcoxon Rank Sum Test was performed to compare the answers to Table 3: Results of the Simulator Sickness Questionnaire. [N]ausea [O]culomotor [D]isorientation Total Score [TS] Mean 16.7 18.95 20.18 21.32 Median 14.31 7.58 6.96 13.09 SD 21.19 21.52 32.09 26.7 Min 0 0 0 0 Max 85.7 75.8 125.28 97.24 710 715 to the ones trained with the user manual. Moreover, the reaction time of the second TOR of the User Manual group was significantly higher than the reaction time of the first TOR of the two other groups. No difference was observed for the third TOR between the three groups. The order of arrival of the three TORs did not impact the reaction time (p = 0.51). Figure 9: Mean of the answer to the pre-post questionnaires. A light-green bar indicates an increase in the post questionnaire for the given question. The values indicate the percentage of change in the questions. 680 685 690 695 700 705 (a) Mean and 95% Confidence Interval of reaction times for each Take-Over Request. the pre and the post questionnaires. Results are reported in Figure 9 and they show a general increase of the post-questionnaire score. It is very interesting to point out that the sentence n. 5 is the only one for which the post-score was lower than the pre-score in all the system (“I think that the semi-autonomous car can reduce the risk of accident.” ). Nevertheless, the difference for this question was not statistically significant. Important increases in the post-questionnaire score can be observer in particular for the sentences 7 (“I see myself doing other tasks than driving in a semi-autonomous car”) and 8 (“I have confidence in the decisions that the semiautonomous car would take at my place”). The positive difference for n. 7 was significant (p < 0.01) only for the LVR group; for n. 8 was significant (p < 0.05) for all the three groups. TOR 1 2 3 1, 2 UM 7.36 (3.55)a 5.97 (3.06)a 3.20 (0.67) 6.66 (3.34)a FB 3.80 (1.61)b 3.01 (0.92)b 2.87 (0.71) 3.41 (1.37)b LVR 3.34 (0.95)b 3.275 (0.89)b 2.84 (0.83) 3.32 (0.91)b (superscripts indicate significance groups) (b) Mean and Standard Deviation Figure 10: Take-over reaction times for each systems and for each TOR. The first two were 10-second TORs; the last one was a 5-second TOR. 3.2. Performance measures The performance measures evaluated the quality of the take-over in terms of reaction time, maximum deviation 720 from the lane center, and the trajectory during an evasive maneuver. These variables were tested for group difference using ANOVA (for normally distributed data) or Kruskal-Wallis (for non-normal distributed data); in case of significant differences (p < 0.05) each pair was tested 725 with the Tukey’s HSD test (after ANOVA) or Fisher’s LSD test (after Kruskal-Wallis test). 3.2.1. Reaction time Three TORs were issued during the test drive after an 730 automated driving phase. In Figure 10 the mean of the reaction time per TOR for each system is reported. For the first and the second TOR, the participants trained with the LVR and the FB simulator reacted faster with respect 10 3.2.2. Deviation from lane center Considering the TOR caused by loss of road marking, the stability of the trajectory was evaluated. Performing lane change in situation in which is not required is usually considered a low-quality take over [44]. In the driving scenario, given that the width of the lane was lane w = 3m and the width of the car was car w = 2m, the maximum possible distance from the lane center beyond which the car does not cross the separation line is d = 0.75m. For each participant, the maximum deviation from the lane center in the 10 seconds after the TOR was calculated. The value was kept with the sign (from −1.75m to +1.75m). The difference between this value and the maximum deviation was evaluated. Results are reported in Figure 11. It can be observed that even though the medians are not significantly different, the LVR group has a lower standard deviation (σ LVR = 0.48) than the UM (σ UM = 1.05) and the FB (σ FB = 1.03) group. Figure 13: Means and 95% Confidence Interval of the stress and confidence score attributed to the participants during the 3 autonomous driving phases. Figure 11: Box plot of deviation from lane center for each system. The two lanes are plotted. The value 0 in the y-axis represents the center of the right lane in the driving scenario. Table 5: Number of gaze switch between the secondary activity and the road environment for each autonomous driving phase; Eyes-onRoad is the ratio between the total amount of time spent looking at the road and the duration of the autonomous driving phase N. of gaze Eyes-on-Road Figure 12: Time To Collision [21]: the red block represents the stationary obstacle; the red line is the trajectory of the vehicle; in green, the position of the car when the TTC h is calculated; in blue, the position of the car when the TTC l is calculated. 735 740 745 750 755 3.2.3. Time To Collision One of the TOR was issued because of a stationary760 car on the right lane. The task of the trainee was to take over and avoid the car. The quality of the maneuver was evaluated with the Time To Collision (TTC) Figure 12 in the same way described by Happee et al. [21]. The TTC 765 was computed using the following formula T T C = dx v , where dx was the distance of the car from the obstacle and v was the speed of the car at that moment. Two TTC were evaluated: TTC h was computed when the heading of the car no longer intersected the obstacle; TTC l was computed when the full vehicle front was in the new lane.770 The results are shown in table Table 4 and no significant differences were observed among the three groups. 3.3. Stress and confidence during autonomous driving 775 To evaluate drivers stress and confidence in the vehicle during automated driving, the video feed recorded during the test drive was analyzed and annotated with a videolabeling tool developed by the authors. During the automated driving phase drivers were free 780 to engage in non-driving related activities by using the Table 4: Means (and standard deviation) of Time To Collision (TTC) and Distance To Collision (DTC) used to evaluate the evasive maneuver DTC h [m] DTC l [m] TTC h [s] TTC l [s] UM 54.21 (35.61) 18.64 (19.12) 4.01 (2.89) 1.27 (1.13) FB 58.45 (49.63) 33.35 (78.17) 3.87 (2.59) 1.09 (1.69) LVR 67.5 (62.72) 23.8 (41.61) 3.43 (2.10) 1.25 (1.86) p = .70 p = .67 p = .77 p = .93 785 UM 17.9 | 16 | 5 0.25 | 0.32 | 0.16 FB 26.6 | 22.9 | 5.6 0.21 | 0.33 | 0.15 LVR 18 | 18.9 | 6.5 0.22 | 0.25 | 0.16 tablet in the car. Before the beginning of the test-drive, participants were instructed in using the tablet to switch between active (games) or passive activities (videos, movies). A score was attributed to each participant during the three autonomous phases. The score ranged from 1 to 5 (the higher the better), where 1 corresponded to “Complete monitoring of the driving environment” and 5 corresponded to “Complete focus on the non-driving activity”. The score of 3 was assigned to drivers who occasionally monitored the driving environment. To attribute this score several aspects were taken into consideration, such as the rate of gaze switch between the driving environment and the tablet and the length of the gazes, the insistence to talk to the experimenter, the position of the driver on the seat, the position of the hands. In Table 5 are reported the number of gaze switch and the ratio between the total amount of time spent looking at the road and the duration of each automated driving phase. Results are reported in Figure 13 and they show that the group trained with the User Manual scored less in the first autonomous phase with respect to the others. Considering that those participants were experiencing autonomous driving for the first time, this behaviour is expected. It can be observed that the score for the UM group increases in the second autonomous phase, while for the FB and HMD groups the behaviour is almost the same. In the third and last phase the score for all the groups increases. 4. Discussion A first outcome of the study is that the training allowed all the participants to respond to the Take-Over Requests. In summary, according to the objective metrics 11 790 795 800 805 810 815 820 825 830 835 840 measured during the test drive, it is possible to identify845 two groups of participants that significantly differed for the reaction time. The group of participants trained with the Virtual Learning Environment (FB and LVR) were able to respond to the take-over request faster than the group of participants trained with the user manual. After850 the take-over, the training system did not significantly influence the driving performance in the lane keeping task and in the evasive maneuver. Furthermore, self-reported measures showed responses in favor of the LVR training system. There are no variables (self-reported nor objec-855 tive) for which the LVR system scored significantly worse than the other training systems. 4.1. Self-report measures Self-report measures showed statistically significant re-860 sults. In particular, significant differences were observed in the answers to the training appreciation questionnaire in which participants evaluated, among other characteristics, its usefulness, ease of understanding and pleasantness. In this questionnaire the LVR system scored sig-865 nificantly better than the fixed-base simulator and user manual training. Analyzing each questions of the survey, we found that the participants considered the FB training more confusing than the LVR even though the training program was exactly the same. Another interesting outcome is that the LVR-based training was considered easier870 to understand than the other systems. Although it is possible that these results can be attributed to a VR “wow factor”, participants’ previous experiences with driving simulators (p = 0.41) and their knowledge of the concept of Virtual Reality (p = 0.25) did not significantly impact the875 answers. A hypothesis to explain these results is that the technical characteristics exclusive to the LVR system, such as the large field of view, the head tracking, the 1:1 scale between the real and the virtual world, affected the perception of the learning. A second hypothesis is that since880 the participants of the LVR group were isolated from the real world, both visually and acoustically, they could better focus on the training. There were no significant differences (p = .66) in the answers of the questionnaire concerning the graphic real-885 ism and the physical realism of the simulator. This result suggests that even if with the LVR the participants interacted with a racing wheel instead of the real steering wheel and they were not able to see their hands, these factors did not play a significant role. Also, the transitional890 environment (Figure 3a) from the real to the virtual world familiarized the participant with the lack of visual representation of the hands. Some of the participants were surprised to not see their hands, but then they realized in this environment that they could rely on proprioception895 to interact in a natural way with the commands. Analyzing the single questions of the pre-post questionnaire about automated cars, it is important to point out that for all the questions but one, the post-questionnaire score was higher than the pre-questionnaire among all the 12 three groups. The questionnaire aimed at evaluating the confidence on automated cars in terms of usefulness, perceived security, willingness to perform secondary tasks and so on. While at the end of the study participants trusted more the actions of the automated system than the beginning, they did not confirm their expectations that the conditionally automated car could reduce the risk of accidents. The hypothesis is that people tended to idealize the autonomous car as a perfect entity, but then their perception was influenced by the driving scenario. In fact, during the test drive the automated driving system prompted three non-planned take-over requests in a short time. This could let the participants think that this kind of TORs were more frequent than they actually are. Furthermore one of the TOR was caused by a critical situation (stationary car). This result suggests that the driving scenario should also present planned take-over requests with longer time buffers and no critical situation. The participants were also asked to self-evaluate on a 1-5 likert scale their readiness to drive an automated vehicle after the training and after the test drive. The answer after the test drive was higher among all the systems, but the difference (+10%) was significant only for the user manual group. 4.2. Simulator Sickness The aim of the evaluation of the SSQ was not to compare the LVR and the FB groups since previous studies in literature already proved that HMDs usually produce more simulator sickness than fixed base simulators [45, 14]. Instead, the experimental protocol included the SSQ [42] with the objective of investigating if the use of an HMD would prevent participants from being trained in an effective way and analyzing the causes of sickness in case of dropouts. A first promising outcome is that no dropouts occurred; this result is very important in particular because 70% of participants in the LVR group used a Virtual Reality headset for the first time. Furthermore, the HMD produced no or minimal symptoms of simulator sickness in 50% of the cases. These results agree with Kennedy et. al [42] who showed that in their survey “the 0-value (the zero point) contained at least 40%, and as much as 75%, of the observations”. Although this study was not focused on the reduction of simulator sickness, we adopted some wellknown strategies [38] on both the Virtual Environment (such as straight road, simple environment and low peripheral optical flow) and the physical system (positional coherence of the virtual and the real steering wheel) to limit the manifestation of oculo-vestibular conflicts. Thanks to these choices, the SSQ results (mean score T S = 21.32 and the absence of dropouts), are comparable, or even lower, than the score found in recent studies about simulation sickness related to virtual reality driving simulation [46, 14]. However, further studies focused on this issue are needed to validate this result and to improve the training experience. 4.3. Objective and performance measures 900 905 910 915 920 925 930 935 940 945 950 955 In the test drive, data in the simulator were recorded to assess the take-over quality and the driver’s behaviors during the automated driving. The take-over quality was evaluated according to the reaction time, the maximum lateral position on the lane, and the time to collision during960 the evasive maneuver to avoid the stationary car on the lane. Concerning the reaction time in the two 10-second TORs, the participants in the UM group reacted slower than the other two groups. No differences were instead observed965 comparing the FB and the LVR groups. For the 5-second TOR the difference was not significant among any of the groups. This result suggests that participants who actually performed a take-over during the training were able to respond better to the first request to intervene in a970 realistic situation. Furthermore, we hypothesize that the decrease of reaction time for the 5-second TOR is due to (i) a learning effect and (ii) the results of Gold et al. [22] who showed that the reaction time depends on the time budget given for the take-over request. With respect to the975 TOR caused by the stationary car on the right lane, not all participants were able to perform a safe evasive maneuver to avoid the obstacle. However, no significant differences were observed between the three groups as far as the time to collision is concerned. Furthermore, according to the980 age group, no significant differences were observed regarding the reaction time nor the maximum lateral position; this result is in agreement with Korber et al. [23] who found that older drivers handle critical traffic events and adapt to the experience of take-over situations as well as younger drivers. Finally, considering the TOR caused by loss of road marking, the stability of the trajectory in the985 30 seconds after the TOR was evaluated and no significant differences between the groups were observed comparing the maximum lateral distance from the center of the lane. The absence of significant differences in these measures suggests that, although these metrics could be useful to as-990 sess driving performance and driver behavior after a takeover request, they are not suitable to compare training effectiveness. In fact, evaluating driving and interaction skills in automated cars is a hard task because it is still not clear what it means to be able to drive or operate995 this kind of vehicles. According to Walker et al. [47] lack of practice arising from sustained automatic control could erode driving skills. The same concern about driving skills emerges from the paper of Stanton et al. [32] who asked “if drivers are not performing a function, how can they be1000 expected to take it over adequately when the automated systems fail to cope?”. Therefore, in the near future, traditional metrics could no longer be relevant to assess driving performance in automated vehicles. The results of this work suggest that1005 Virtual Reality represents a promising tool to evaluate as well all those metrics that are not strictly related to the driving activity. 13 4.4. About learning-by-driving An important observation which challenged the study was the feeling that the participants trained with the VLE (FB and LVR groups) did not experience the training as an actual training program, but more like a session of automated driving simulator. Let us recall that the aim of the VLE was (i) to inform drivers about the characteristics of the automated driving system, (ii) help them in identifying and localize the HMI in the car and (iii) teach the appropriate response (activation and deactivation of the automated driving system) to a given stimulus. For the last two objectives, a virtual vocal assistant provided instructions to the participants; while instructions relative to the takeover were provided to the subjects during the secondary activity, those relative to the activation of the automated driving systems were given to them while they were performing the driving task; in other words, participants were asked to aim attention at the training instructions while they were already focusing on a high-demand cognitive activity. However, the driving scenario during the training was kept as simple as possible (no traffic, straight lane) in order to limit driver interventions. Although this hypothesis, all the trainees were able to assimilate the procedural skills. One of the research questions we will investigate in the future study will be if learning-by-driving actually improves the skill acquisition process, or if the driving task requires a cognitive load which deteriorates the training performance. 5. Conclusion The aim of this study was to investigate the effectiveness of a light Virtual Reality system based on HMD for the training of conditionally automated vehicle drivers. To evaluate the usefulness of the system and to assess the effectiveness of the training, the LVR was compared to a fixed-base simulator and a user manual; a test drive in a high-end simulator was performed by the participants after the training. To the best of our knowledge, this study represents the first attempt of use of HMD-based Virtual Reality for training purposes in automated vehicles. The results of this research persuade us that light Virtual Reality systems represent a valuable tool for the acquisition of operational skills in conditionally automated vehicles. The proposed training system, composed of an HMD and a game racing wheel, is a portable and costeffective device that provides an adequate level of immersion for teaching drivers how to respond to a take-over request in a safe environment. Therefore, this system could be employed for the training of future customers of automated cars before their first ride. The step-by-step tutorial implemented in the Virtual Learning Environment impacted on the performance and provided faster reaction time in the test drive. Moreover, among all the training systems, participants preferred the light VR system in terms of usefulness, ease of use and realism. 1010 1015 1020 A direct outcome of these results is the acknowledgment of VR as key player in the definition of the set of1070 metrics for profiling drivers’ behavior in automated vehicles in order to ensure a safer interaction between humans and automated systems. Starting from these results, in future work we will im-1075 plement the training program in the form of a serious game in which the level of instruction adapts to the drivers’ needs in order to assess the acquisition of skill during the training itself. Furthermore, the need of the simulated1080 driving task during the training will be investigated. Finally, longer test-drives with real vehicles are considered of primary importance to validate current results. [13] [14] [15] [16] 1085 [17] Acknowledgement 1025 This research was supported by the French Foundation1090 of Technological Research under grant CIFRE 2015/1392 for the doctoral work of D. Sportillo at PSA Group. [18] References 1030 1035 1040 1045 1050 1055 1060 1065 1095 [1] S. International, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. [2] S. M. Casner, E. L. Hutchins, D. Norman, The challenges of partially automated driving, Communications of the ACM 59 (5)1100 (2016) 70–77. [3] I. Y. Noy, D. Shinar, W. J. Horrey, Automated driving: Safety blind spots, Safety Science 102 (2018) 68 – 78. doi:https: //doi.org/10.1016/j.ssci.2017.07.018. URL http://www.sciencedirect.com/science/article/pii/1105 S0925753517304198 [4] J. De Winter, P. Van Leeuwen, R. Happee, Advantages and disadvantages of driving simulators: a discussion, in: Proceedings of Measuring Behavior, 2012, pp. 47–50. [5] B. Dalgarno, M. J. Lee, What are the learning affordances of1110 3-d virtual environments?, British Journal of Educational Technology 41 (1) (2010) 10–32. [6] E. Hamilton, D. Scott, J. Fleming, R. Rege, R. Laycock, P. Bergen, S. Tesfay, D. Jones, Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic1115 skills, Surgical Endoscopy and Other Interventional Techniques 16 (3) (2002) 406–411. [7] J. Vora, S. Nair, A. K. Gramopadhye, A. T. Duchowski, B. J. Melloy, B. Kanki, Using virtual reality technology for aircraft visual inspection training: presence and comparison studies, Ap-1120 plied ergonomics 33 (6) (2002) 559–570. [8] G. Avveduto, C. Tanca, C. Lorenzini, F. Tecchia, M. Carrozzino, M. Bergamasco, Safety training using virtual reality: A comparative approach, in: International Conference on Augmented Reality, Virtual Reality and Computer Graphics,1125 Springer, 2017, pp. 148–163. [9] H. Zhang, Head-mounted display-based intuitive virtual reality training system for the mining industry, International Journal of Mining Science and Technology. [10] W.-S. Lee, J.-H. Kim, J.-H. Cho, A driving simulator as a vir-1130 tual reality tool, in: Robotics and Automation, 1998. Proceedings. 1998 IEEE International Conference on, Vol. 1, IEEE, 1998, pp. 71–76. [11] I. Milleville-Pennel, C. Charron, Driving for real or on a fixedbase simulator: is it so different? An explorative study, Presence1135 24 (1) (2015) 74–91. [12] D. L. Fisher, N. E. Laurie, R. Glaser, K. Connerney, A. Pollatsek, S. A. Duffy, J. Brock, Use of a fixed-base driving simulator 14 [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] to evaluate the effects of experience and pc-based risk awareness training on drivers’ decisions, Human factors 44 (2) (2002) 287–302. M. Klüver, C. Herrigel, C. Heinrich, H.-P. Schöner, H. Hecht, The behavioral validity of dual-task driving performance in fixed and moving base driving simulators, Transportation research part F: traffic psychology and behaviour 37 (2016) 78–96. F. Weidner, A. Hoesch, S. Poeschl, W. Broll, Comparing VR and non-VR driving simulations: An experimental user study, in: Virtual Reality (VR), 2017 IEEE, IEEE, 2017, pp. 281–282. S. M. Taheri, K. Matsushita, M. Sasaki, Development of a Driving Simulator with Analyzing Driver’s Characteristics Based on a Virtual Reality Head Mounted Display, Journal of Transportation Technologies 7 (03) (2017) 351. D. Goedicke, J. Li, V. Evers, W. Ju, Vr-oom: Virtual reality on-road driving simulation, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM, 2018, p. 165. D. Sportillo, A. Paljic, M. Boukhris, P. Fuchs, L. Ojeda, V. Roussarie, An immersive virtual reality system for semiautonomous driving simulation: a comparison between realistic and 6-dof controller-based interaction, in: Proceedings of the 9th International Conference on Computer and Automation Engineering, ACM, 2017, pp. 6–10. Q. C. Ihemedu-Steinke, R. Erbach, P. Halady, G. Meixner, M. Weber, Virtual reality driving simulator based on headmounted displays, in: Automotive User Interfaces, Springer, 2017, pp. 401–428. V. Melcher, S. Rauh, F. Diederichs, H. Widlroither, W. Bauer, Take-over requests for automated driving, Procedia Manufacturing 3 (2015) 2867–2873. N. Merat, A. H. Jamson, F. C. Lai, M. Daly, O. M. Carsten, Transition to manual: Driver behaviour when resuming control from a highly automated vehicle, Transportation research part F: traffic psychology and behaviour 27 (2014) 274–282. R. Happee, C. Gold, J. Radlmayr, S. Hergeth, K. Bengler, Takeover performance in evasive manoeuvres, Accident Analysis & Prevention 106 (2017) 211–222. C. Gold, D. Damböck, L. Lorenz, K. Bengler, “Take over!” How long does it take to get the driver back into the loop?, in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 57, SAGE Publications Sage CA: Los Angeles, CA, 2013, pp. 1938–1942. M. Körber, C. Gold, D. Lechner, K. Bengler, The influence of age on the take-over of vehicle control in highly automated driving, Transportation research part F: traffic psychology and behaviour 39 (2016) 19–32. K. Zeeb, A. Buchner, M. Schrauf, Is take-over time all that matters? The impact of visual-cognitive load on driver take-over quality after conditionally automated driving, Accident Analysis & Prevention 92 (2016) 230–239. J. Navarro, M. François, F. Mars, Obstacle avoidance under automated steering: impact on driving and gaze behaviours, Transportation research part F: traffic psychology and behaviour 43 (2016) 315–324. W. Payre, J. Cestac, N.-T. Dang, F. Vienne, P. Delhomme, Impact of training and in-vehicle task performance on manual control recovery in an automated car, Transportation Research Part F: Traffic Psychology and Behaviour 46 (2017) 216 – 227. doi:https://doi.org/10.1016/j.trf.2017.02.001. URL http://www.sciencedirect.com/science/article/pii/ S1369847816300651 K. Spolander, Drivers’assessment of their own driving ability, Tech. rep. (1983). J. Reason, A. Manstead, S. Stradling, J. Baxter, K. Campbell, Errors and violations on the roads: a real distinction?, Ergonomics 33 (10-11) (1990) 1315–1332. M. M. Roy, M. J. Liersch, I am a better driver than you think: examining self-enhancement for driving ability, Journal of applied social psychology 43 (8) (2013) 1648–1659. S. Trösterer, M. Gärtner, A. Mirnig, A. Meschtscherjakov, 1140 1145 [31] [32] 1150 [33] [34] 1155 [35] 1160 [36] 1165 [37] [38] [39] 1170 [40] 1175 [41] [42] 1180 [43] 1185 [44] 1190 [45] 1195 [46] 1200 [47] R. McCall, N. Louveton, M. Tscheligi, T. Engel, You Never Forget How to Drive: Driver Skilling and Deskilling in the Advent of Autonomous Vehicles, in: Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ACM, 2016, pp. 209–216. P. Bazilinskyy, A. Eriksson, B. Petermeijer, J. de Winter, Usefulness and satisfaction of take-over requests for highly automated driving. N. A. Stanton, P. Marsden, From fly-by-wire to drive-by-wire: safety implications of automation in vehicles, Safety Science 24 (1) (1996) 35–49. J. Vince, 10 virtual reality techniques in flight simulation, Virtual Reality Systems (1993) 135. P. J. Stappers, W. Gaver, K. Overbeeke, Beyond the limits of real-time realism: Moving from stimulation correspondence to information correspondence, Virtual and adaptive environments (2003) 91–110. S. M. Casner, R. W. Geven, K. T. Williams, The effectiveness of airline pilot training for abnormal events, Human factors 55 (3) (2013) 477–485. M. Bueno, E. Dogan, F. H. Selem, E. Monacelli, S. Boverie, A. Guillaume, How different mental workload levels affect the take-over control after automated driving, in: Intelligent Transportation Systems (ITSC), 2016 IEEE 19th International Conference on, IEEE, 2016, pp. 2040–2045. M. Slater, A note on presence terminology, Presence connect 3 (3) (2003) 1–5. H. A. Stoner, D. L. Fisher, M. Mollenhauer, Simulator and scenario factors influencing simulator sickness. M. Sisto, N. Wenk, N. Ouerhani, S. Gobron, A study of transitional virtual environments, in: International Conference on Augmented Reality, Virtual Reality and Computer Graphics, Springer, 2017, pp. 35–49. Citroen DS3, https://www.turbosquid.com/3d-models/ citroen-ds3-city-car-max/577206. L. J. Hettinger, G. E. Riccio, Visually induced motion sickness in virtual environments, Presence: Teleoperators & Virtual Environments 1 (3) (1992) 306–310. R. S. Kennedy, N. E. Lane, K. S. Berbaum, M. G. Lilienthal, Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness, The international journal of aviation psychology 3 (3) (1993) 203–220. R. S. Kennedy, J. M. Drexler, D. E. Compton, K. M. Stanney, D. S. Lanham, D. L. Harm, Configural scoring of simulator sickness, cybersickness and space adaptation syndrome: Similarities and differences, Virtual and adaptive environments: Applications, implications, and human performance issues (2003) 247. C. Braunagel, W. Rosenstiel, E. Kasneci, Ready for take-over? a new driver assistance system for an automated classification of driver take-over readiness, IEEE Intelligent Transportation Systems Magazine 9 (4) (2017) 10–22. B. Aykent, Z. Yang, F. Merienne, A. Kemeny, Simulation sickness comparison between a limited field of view virtual reality head mounted display (oculus) and a medium range field of view static ecological driving simulator (eco2), in: Driving Simulation Conference Europe 2014 Proceedings, Society for Modeling & Simulation International, 2014, pp. 65–71. Q. C. Ihemedu-Steinke, S. Rangelova, M. Weber, R. Erbach, G. Meixner, N. Marsden, Simulation sickness related to virtual reality driving simulation, in: International Conference on Virtual, Augmented and Mixed Reality, Springer, 2017, pp. 521– 532. G. H. Walker, N. A. Stanton, Human factors in automotive engineering and technology, CRC Press, 2017. 15