Driver intervention performance assessment as a key aspect of L 3 – L 4 automated vehicles deployment

In L3–L4 vehicles, driving task is performed primarily by automated driving system (ADS). Automation mode permits to engage in non-driving-related tasks; however, it necessitates continuous vigilance and attention. Although the driver may be distracted, a request to intervene may suddenly occur, requiring immediate and appropriate response to driving conditions. To increase safety, automated vehicles should be equipped with a Driver Intervention Performance Assessment module (DIPA), ensuring that the driver is able to take the control of the vehicle and maintain it safely. Otherwise, ADS should regain control from the driver and perform a minimal risk manoeuvre. The paper explains the essence of DIPA, indicates possible measures, and describes a concept of DIPA framework being developed in the project.


Introduction
Driving is commonly considered as a multifactorial process that requires significant cognitive resources. It necessitates continuous road monitoring, surroundings analysis, and quick decision-making [1]. The process involves three levels of control [2]: • Operationalrefers to the driver's reaction to traffic conditions with limited decision information. This level takes advantage of sensorimotor abilities and occurs in short intervals of time, e.g. steering and braking due to sudden changes. • Tacticrequires vehicle manoeuvring in response to normal road conditions, e.g. at junctions. This type of control occurs for a few seconds. • Strategicrefers to route planning, including destination selection. Occurs at intervals of minutes to hours.
Technology development in the field of advanced sensors, software, and artificial intelligence encouraged car companies to develop a self-driving vehicle (e.g. refs. [3,4]). Automated driving systems (ADS) [5] in which perception and decision-making are being made by machine/artificial components are said to become a reality in the following decades. Automation requires drivers to relinquish control of the vehicle, while maintaining awareness to enable safe performance in case the system reaches its limits. ADS should have both automated and manual modes. According to the society of automotive engineers (SAE 2018 [6]), there is a six-level scale of driving automationfrom 0 (no automation) to 5 (full automation): • Level 0no automation, the driver performs the entire dynamic driving task (DDT), even when enhanced by active safety systems. • Level 1driver assistance, the driver controls the vehicle, and the system makes adjustments to speed and direction (e.g. adaptive cruise control, lane keep assist). The system executes only one subtasknot both simultaneously. • Level 2partial automation, the driver controls the vehicle, and the system makes simultaneous adjustments to both speed and direction (e.g. park assist, traffic jam assist). • Level 3conditional automation, the system has full control over the vehicle (speed, direction, environment monitoringe.g. Steering Collision Avoidance) but only under specific conditions (e.g. motorway, limited speed, no crossings). The driver has to be constantly ready to intervene if a dangerous situation is detected.
• Level 4high automation, the system has full control over the vehicle, and driver presence is not necessary, but only up to the system's limits. The driver can undertake non-driving-related tasks. If the actual driving conditions exceed the system performance limits, it may ask the driver to intervene or decide to stop the journey. The system works under specific conditions. • Level 5full automation, the human-driver is not necessary and considered as a passenger. The system works unconditionally. An ADS performs the entire DDT and DDT fallback without any expectation that a user will respond to a request to intervene (RtI).
There are two paths to achieve full automation. The first one involves designing the vehicle from scratch, while the second uses incremental development through all the above-mentioned levels of automation. In this article, the second path will be discussed, namely the L3 and L4 levels, where the vehicle can move independently, but only under certain conditions, and there may be a need to retake control. It is assumed that in L3-L4 vehicles, during periods of automated driving, human will be out-of-the-loop and may not have enough information to safely maintain control at the operational and tactic level. Attention shifting may result in reduced situation awareness, causing distraction or fatigue. A Driver Intervention Performance Assessment (DIPA) uses a variety of methods to ascertain that the driver is able to respond safely to dangerous traffic incidents. DIPA comprises time-based intervention performance measures (e.g. for assessing different time budget components), as well as objective (e.g. minimum time headway) and subjective (e.g. perceived controllability) take over quality measures. Owing to the risk of an accident, which varies depending on vehicle type, driving skills, age, gender, personal characteristics [7], and arousal level, the assessment should focus on the knowledge and attitude of the driver. It could identify the risk management capability and ensure that drivers are aware of vulnerable road users, in particular pedestrians, cyclists, and motorcyclists.

DIPA definition
DIPA involves the definition of relevant objective measures to assess the quality of intervention performance, such as driver take over time (TOT) from onset of RtI, driver intervention time, control stabilization time, remaining action time, as well as subjective measures for the quality of intervention performance.
From the ADS's perspective, DIPA is implemented as a decision system, laying above modules intended to monitor driver's state, behaviour, and reactions. DIPA module activates when a RtI appears. It gathers information from driver state monitoring (DSM) and vehicle monitoring. When system identified that driver took the control, DIPA module ascertains if the human is able to safely maintain manual driving, or a decision of performing a Minimal Risk Maneuver should be taken (otherwise). Figure 1 shows a simplified flow diagram of the input information to the DIPA module.

Control transition
The assessment of the driver is crucial primarily in relation to reacting to a sudden, unexpected event on a road. It necessitates the completion of a sequence of processes in a very short time. In level 3 conditional automation, the driver is not required to constantly monitor the environment and the automated system performance. That may result in shifting attentional resources to other tasks. However, the driver has to maintain readiness for taking over vehicle control in "situations that exceed the operational limits of the ADS" [8], detected and announced by the automation system. The transition from automated driving to manual mode may be human-or system-initiated.
DIPA refers to transition forced by ADS, which is started by a RtIa critical event requiring active driver intervention. The driver has to relocate hands and feet to the driving position, regain situation awareness, and execute an adequate response to the system limit [9,10]. ADS can met with trust by end-users if the driver state is continuously monitored, his/her availability properly evaluated and sufficiently triggered (through tailored human-machine interfaces -HMIs) [11]. The human factor is essential for the safety deployment of L3-L4 vehicles, mainly due to the necessary driver-vehicle interaction in cases where the boundaries of the Operational Design Domain are being reached. ADS limits may be caused by sensor malfunction, extreme weather conditions, appearance of evolving accident scenes, unexpected roadblocks, hazardous traffic violations falling of goods, etc. The figure below ( Figure 2) describes the processes during the transition from automated to manual driving initiated by ADS.

DIPA measures
While the capabilities of an automated system may be exceeded in some circumstances, the RtI signal may appear. Once a RtI occurred, a driver has limited time to react, to ensure safety. The time interval between the RtI and the system limit is defined as "total time budget." It represents the maximum time window after a RtI for a successful resumption of manual control. In assessing the quality of driver intervention, selected factors should describe the driver's behaviour and driving technique, meeting the following criteria [12,13]: • suggesting an unambiguous method of measuring, • measurement repeatability, • can be applied both in a simulation environment and in a real vehicle, • have one interpretation, • can be easily measured and presented graphically, • applicable to engineering procedures and calculations, • applicable in various countries, vehicles, and conditions, • used in research and described in the literature, • use SI units.
There is still a lack of unanimity on how to ensure driver's ability to take over the control and safely perform the manoeuvre. The monitoring and assessment of a driver should ensure that he/she is able to control the vehicle himself/herself. It includes monitoring of sensory, motoric, cognitive, arousal, and emotional states, as refers also to DSM, but expanding these factors by measures of driving performance. The take over phases should be measured ensuring the minimal safety level after turning the driver back-in-the-loop. It could be done using objective and subjective measures. Objective measures relate to speed, manoeuvres, or other forms of behavioural analysis (time-to-collision, time headway, time-to-lane-change), as well as driver reaction measures (TOT, driver intervention time, control stabilization time, remaining action time). Subjective ones are associated with perceived controllability [5].
The objective measures are time-based intervention performance and take over quality measures as follows: • TOTan interval between RtI occurring and userinitiated intervention or deactivation of an engaged automation function. The transition can be divided into further subprocesses, e.g.: time to first driver reaction, time to visually fixate RtI message (if visual HMI is involved), time to start to move (at least one) hand to wheel/feet to pedals, and time to start to operate relevant vehicle controls (e.g. blinker) [8]. TOT is a humanbased measure and should not be considered in relation with system deactivation time. • Reaction timea subprocess of take over, it is defined as the time that elapses from the moment of the stimulus (nervous impulse) activation to the moment of movement initiation [14]. • Decision timean interval between detection of a silent system failure and the decision to disengage the automation feature. • Intervention timea period of time required to handle the imminent take over situation by performing an appropriate driving manoeuver. • Driving recovery timea measure combined with TOT and intervention time. Comparison between total time budget and driving recovery time should be used to provide guidelines that take account of all the driver's reactions to ensure a successful manoeuvre. • Remaining action timea comparison between total time budget and a successfully finished driver intervention time. It may be paraphrased as a remaining time to last intervention possibility at the time of intervention [3]. • Control stabilization timea period of time until vehicle control performance is fully re-established. After the immediate driver intervention phase, manual vehicle control performance can be below the average performance of an individual driver [8]. • Headwaythe space between the reference points of two vehicles. It may be defined in both time and distance domains. Headway can be defined as the space between the outer edges of the leading vehicle's rear bumper and the following vehicle's front bumper (Figure 3). The size of this parameter determines the safety margin that the driver of the following vehicle leaves to react to manoeuvres of the leading vehicle. Measuring this parameter necesitates meeting the following conditions [13]: vehicles move in the same lane, vehicles follow each other on the passage path, longitudinal parameter of the distance is measured (assuming movement on a straight road).
In the time domain, the headway is expressed by the length of time that elapses from the moment it is measured until the following vehicle's reference point reaches the point that determined the location of the leading vehicle's reference point, assuming that the speed of the following vehicle does not change.  Figure 4). Low values of minimum TTC indicate either a slow braking reaction time or an inadequately low braking force [15]. TTC is only calculated when the following vehicle has a higher speed than the vehicle in front [16].  • Steering wheel reversal ratean indicator of steering wheel changes from clockwise to counter clockwise, or vice versa. The frequency and amplitude of these movements identify driver's ability to maintain control over the vehicle.   intervention. The best intervention method may differ in various driving scenarios [18]. • Quality assessment of safety effectsa measure using subjective rating of criticality. Each level is set based on measurements, but categorization and intervention time division is done subjectively. The method may be designed with categories, e.g. uncontrollable situation (no time to react), dangerous, unpleasant, harmless, and nothing noticed (no danger). The assessment method may be based on time-metrics, acceleration values, or distance [16]. • Crash probabilitya method similar to the previous one, but in addition to measuring time or motionoriented values; it is also based on the likelihood of an accident. It can result in categorization or present results in percentages [10]. • Self-reported difficulty/dangera survey carried out after performing a take over. The subjective driver's rating of the difficulty/danger of the situation. The outcomes may be presented on the scale or converted to a percentage [19]. • Expert-based assessmenta method of integrating different timing and quality parameters into a global controllability measure [8]. Trained evaluators are required to integrate different aspects of driving performance by using a standardized rating scheme ( Figure 6). The method proposed by Naujoks et al. [20] was developed in reference to automated vehicles, based on an existing assessment procedure that has been successfully applied to assess the criticality of driving situations in manual driving conditions (Kaussner's driving behaviour assessment [21]). The TOC-Rating (Take over controllability rating) may be used in empirical studies on controllability in accordance with the Code of Practice [22].

DIPA framework
Trustonomy (trust + autonomy) is a project connecting stakeholders from different automotive industry, research, and transport areas. It aims to compare in terms of performance, ethics, and acceptability different technologies and approaches in different scenarios for automatic driving and requests to take control. The project investigates several ADS-related domains including DSM, performance assessment, and training. A DIPA framework is being under the development within the project [23]. Other assumptions, goals, and expected results are described in the article Bringing Trust to Autonomous Mobility, written in Figure 6: Overview of TOC-rating process [20].

Expected framework components
DIPA framework is a set of objective measures ensuring the minimal safety level after turning the driver back-inthe-loop. At the current stage of research (eight months since the project started), this set will most likely consist of: • set of measures allowing to assess if driver performs within expected boundaries, • DIPA framework software.
Decisions taken by the DIPA module, which is part of the ADS decision system, are crucial for vehicle safety. It is extremely important that the system remains reliable and provide low latency. Therefore, the DIPA module must operate in the vehicle ecosystem, regardless of any external communication.

Development plan
DIPA Framework development plan requires few steps to be taken. First, applicable methods, measurements, and scenarios will be selected based on literature review. Second, simulator-based research will be conducted to identify which methods and measurements are a good DIPA indicators in particular scenarios and sensors configurations. Third, DIPA Framework will be developed, based on preliminary pilot research results. Fourth, DIPA Framework will be tested in the second series of trials.

DIPA software
The main task of DIPA software will be to analyse input data from different sources (mainly sensors) and perform situation assessment. The result of the analysis is the choice of take over manoeuvre, i.e. making a decision about manual or automatic emergency mode. Planned interfaces are: • vehicle monitoring: information about vehicle controls, steering wheel hands sensor, etc. • vehicle movement monitoring: vehicle trajectory, acceleration, etc.
• Human-Machine interface: warning indication • Automated vehicle decision layer: information about RtI (input) and resultant decision (output).

Conclusion and discussion
In assessing driver's intervention performance, it is essential to present the overall safety effects of a transition in a particular traffic situation. Driving scenario context is crucial, as there is no intervention type suitable for all dangerous situations. Thus, different outcomes and measures of take over quality should always be compared carefully and with extensive attention to details. It is always recommended to use a combination of appropriate measures as single measures may not be sufficient enough to differentiate safe and unsafe events. DIPA can be considered in various sensor configurations, involving different measures of driver state and behaviour. It can be also considered in multiple road situations, with different causes of RtI, which may differ driver behaviour and maximal available time for reaction (total time budget). Since it is not possible to consider all possible cases, in Trustonomy, several typical scenarios will be selected and used as a basis for the DIPA framework to be developed. All details presented in this paper are early estimations and may change in further work.
Although a driver is considered to be an end-user of the final design, the main outcomes of the project directly address the needs of the automotive industry. DIPA framework will provide methodology and measures that can be used to develop decision system for the RtI situation. The outcomes of the project may be useful for developers and integrators of Automated Vehicle systems and treated as a background for standardization activities or for test protocols development.

Acknowledgement:
The publication discusses analysis that is being conducted within Trustonomy. The authors would like to thank all partners involved in the Trustonomy project (www.h2020-trustonomy.eu) for their precious collaboration in the research activities that contributed to achieve the results herein described.
Funding information: This work has been performed in the context of the Trustonomy project, which has received funding from the European Community's Horizon 2020 research and innovation programme under the grant agreement No 812003.