Autonomy in surgical robots and its meaningful human control

Abstract This article focuses on ethical issues raised by increasing levels of autonomy for surgical robots. These ethical issues are explored mainly by reference to state-ofart case studies and imminent advances in Minimally Invasive Surgery (MIS) and Microsurgery. In both area, surgicalworkspace is limited and the required precision is high. For this reason, increasing levels of robotic autonomy can make a significant difference there, and ethically justified control sharing between humans and robots must be introduced. In particular, from a responsibility and accountability perspective suitable policies for theMeaningfulHuman Control (MHC) of increasingly autonomous surgical robots are proposed. It is highlighted how MHC should be modulated in accordance with various levels of autonomy for MIS and Microsurgery robots. Moreover, finer MHC distinctions are introduced to deal with contextual conditions concerning e.g. soft or rigid anatomical environments.


Introduction
Autonomous robots give rise to novel ethical issues concerning responsibility ascriptions for their wrongdoings and moral constraints to impose on their autonomous behaviour. Reflective work on these issues is rapidly advancing in the case of autonomous weapons systems (AWS) [1] and driverless cars [2], in view of the fact that both kinds of robotic systems may already undertake autonomous actions that are ethically critical, such as the targeting and attacking of humans in the case of AWS or the management of unavoidable collisions involving human passengers and pedestrians in the case of driverless vehicles.
Robotic surgery has not reached a similarly advanced stage in the way of robotic system autonomy. In Robot-Assisted Surgery (RAS), one usually operates in a masterslave control mode, so that the behaviours of surgical robots unfold under the surgeon's hands-on supervision and real-time overriding authority. No substantive responsibility and accountability issues arise when these settings are in place, for human surgeons obviously retain meaningful control on each pre-and intraoperative aspect of surgery. It is therefore unsurprising that the ethical discussion of surgical robot autonomy is still in its infancy and mostly embedded into technologically distant scenarios of highly autonomous systems. In addition to alleged responsibility and accountability gaps for surgery outcomes, in these technologically distant scenarios the recurring ethical themes include increasing machine authority and ensuing threats to surgeon-patient trust, surgeon deskilling and human-machine competition for surgical theatre practice¹.
In this contribution, the discussion of emerging ethical issues concerning surgical robot autonomy is framed and anchored more firmly into current and imminent developments of both RAS research and clinical practice. In particular, clinical motivations for developing increas-ingly autonomous robots are identified in the areas of MIS and Microsurgery. In both areas surgical workspace is limited and the required precision is high. For this reason, increasing levels of robotic autonomy can make a significant difference there. However, from an ethical standpoint, one has to make sure that increasing robot autonomy does not jeopardize the preservation of human responsibility and accountability chains. Accordingly, in both RAS areas one has to develop suitable policies for the Meaningful Human Control (MHC) of increasingly autonomous surgical robots. This ethical policy development issue is addressed here by highlighting how MHC should be modulated in accordance with various levels of autonomy one may grant to MIS and Microsurgery robots. Moreover, finer distinctions of MHC are introduced to deal with contextual conditions concerning e.g. soft or rigid anatomical environments. Finally, it is argued that one may have to give up MHC in some emergency situations, on account of medical beneficence considerations, enabling robots to act with unconditional control capabilities in task execution.
The article is organized as follows. In Section 2 (Varieties of robotic autonomy in MIS), some commercially available and research robotic platforms for Minimally Invasive Surgery (MIS) are selectively examined with the sole purpose of identifying varieties of extant robotic autonomy in MIS. Moreover, driving forces towards greater robotic autonomy are highlighted in the case of robot-assisted microsurgery (Section 3). Microsurgery is an exemplary domain for robotic autonomy drivers, insofar as one can envisage significant benefits for patients and public health deriving from the automation of surgical tasks that are at the limits of human capabilities. In addition to what robots are already (or will be soon) able to perform autonomously in these application domains, we examine both the where and the how of surgical robot autonomy. Specifically, growing challenges for autonomous robotic action are identified as one moves from the where of rigid anatomic structures to soft tissue surgical sites. Similarly, the how of surgical robot autonomy and its growing challenges are examined as one moves towards adaptive control, operative strategy planning, and the underlying learned skills.
In Section 4 (Robotic autonomy and MHC), it is shown that the what, where, and how dimensions of robotic autonomy in surgery jointly enable one to revisit and refine some recently proposed hierarchies of increasing autonomy for surgical robots [3,4]. These refinements are ethically motivated, insofar as they enable one to advance the discussion of what it means to exert and modulate properly Meaningful Human Control (MHC) of robotic autonomy within each level of the proposed hierarchies. The notion of MHC was originally introduced in ethical and le-gal discussions of autonomous weapons systems (AWS) [5] and was more recently investigated in connection with driverless autonomous vehicles and other autonomous systems [6]. There, MHC is broadly understood as an ethical policy approach which is supposed to ensure human responsibilities and accountability while admitting limited forms of robotic autonomy in those application domains. The MHC approach to surgical robot autonomy is explored here chiefly from the distinctive ethical perspective of human surgeon supervision duties. Wider ethical implications of MHC in robotic surgery are explored too. This is done in connection with information disclosure in informed consent procedures and with the identification of surgeon retrospective responsibilities after the occurrence of some harmful autonomous robotic action.
The overall aim of this contribution is to raise awareness and stimulate continuing ethical reflection about MHC for increasingly autonomous surgery robots, to explore how the MHC normative requirements may contribute to enrich proposed classifications of increasing autonomy for surgery robots, and how MHC affects our understanding of prospective and retrospective responsibilities of human surgeons in present and forthcoming situations of shared human-robot control within the operating room. As a preliminary step, let us now turn to identify and illustrate various forms of extant robot autonomy in minimally invasive surgery.

Varieties of robotic autonomy in MIS
MIS is being increasingly performed with the assistance of advanced robotic systems. Robotic systems for MIS are mostly operated in a master-slave control mode, which enables surgeons to control the entire procedure, including data analysis, pre-and intraoperative planning, decisions and actual execution. In the master-slave control mode, robotic assistance usually improves surgical procedures by scaling motion, attenuating tremor and enhancing precision. This is the currently standard use of the da Vinci system [7] for laparoscopic surgery, which is controlled in telemanipulation, so that the human has direct control over each procedure. In this case and other similar circumstances, an unqualified use of the expression "robotic surgery" may be potentially misleading: robotic surgical systems do not perform on their own any designated part of surgery; they are limited to act as slave devices teleoperated by surgeons.
In certain special conditions, such as transanal surgery [8], where teleoperated instruments move in narrow spaces with a very small relative angle, or ophthalmologic surgery, where manipulation requirements and positioning precision reach or even go beyond the boundaries of human skills [9], the semi-autonomous control modality can effectively make the difference: mechanical precision and tools' dexterity, untiring and stable motion execution are some notable advantages of robotic systems assistance.

Sensor-based shared control in MIS
Even though teleoperation is the primary control mode [10], sensor-based shared control of robot trajectories has been extensively investigated and successfully used to augment surgeon abilities.
Medical imaging (X-ray fluoroscopy, MRI, CT, and ultrasound) and multimodal sensory integration have a proven usefulness by providing important anatomic information about patients. They can be used both preoperatively to assist surgeon planning or intraoperatively to guide some procedure. The potential of sensors-based automated procedures is in restricting instruments paths to satisfy safety constraints, or in responding quickly to changing environmental conditions based on sensors information and integration.
Besides model-based planning and control procedures, machine learning provides powerful tools to advance robotic process automation, insofar as it enables one to deal with situations where complete modelling of system behaviours is unfeasible. Learning methods are being increasingly used to acquire motor information from surgeons and to adapt future robot behaviours accordingly. Learning-by-demonstration approaches are used in [11] to automate surgical subtasks such as suturing, with an increased performance in terms of speed and smoothness, and in [12] to define a Finite State Machine (FSM) that models discrete components of some surgical subtasks. In [13] a record of surgical tasks from expert surgeons, followed by a decomposition procedure and a Gaussian mixture regression (GMR) is used to extract a smooth trajectory reproducing the task. An extensive survey on the application of machine learning techniques in the context of surgery is found in [14].
Active constraints is a topical framework used in different surgical procedures and at various levels of autonomy. Fruitful research directions and clinical applications concern the dynamical identification of active constraints (ACs, aka Virtual Fixtures, and from now on VFs). These are broadly grouped into Guidance VFs (GVFs), which assist the user in moving the manipulator along desired paths or surfaces in the workspace, and Forbidden-Region VFs (FRVFs), which prevent the manipulator from entering certain workspace regions. In both cases, the robot offers more resistance in selected directions by providing haptic responses. The force/motion relationships is either of admittance or of impedance type. In less automated systems, the robot cannot distinguish between safe versus forbidden regions, and surgeons are required to supply the robot with this information. Advances in VFs automation are attained by designing robot control strategies that locate VFs in the human body, analyse and correct them without any human guidance toward complete VFs automatic generation [15], VFs adaptive generation [16] and VFs generation by learning from expert surgeons [17]. A most significant research challenge concerns the generation of constraint geometries in an effective [18] and dynamic [17] way, considering scene changes arising from tissue motion [19]. A state-of-art review on ACs/VFs is found in [20].

Suturing and the promise of task automation
While autonomous robotic performance of complete surgical procedures is currently farfetched from a technological point of view, one may decompose complex surgical procedures into smaller surgical tasks and identify some such tasks, like suturing or resection, that are more feasible to automate [4]. Suturing is the most significant surgical task to be automated, due to the difficulty of carrying it out using telerobotic systems without force feedback. Accordingly, a considerable amount of research work is currently devoted to autonomous suturing. In [21], a single camera is used in combination with an elliptical pose measurement algorithm to find the needle, while simple markers are used to find the suturing points. A framework to optimize needle trajectory for a multilateral suturing procedure with the da Vinci Research Kit (dVRK) is presented in [22]. An autonomous path planner for suturing that attempts to minimize interaction forces between tissue and needle is presented in [23]. A relevant advancement towards greater robotic autonomy in suturing was achieved with the Smart Tissue Autonomous Robot (STAR) platform [24], developed at the Sheikh Zayed Institute for Pediatric Surgical Innovation of the Children's National Health System. This platform, based on a KUKA medical robot equipped with a 1-DoF suturing tool, carries out intestinal anastomosis on exvivo tissue and on in-vivo tissue in an anaesthetized pig.
In experimental tests on this animal model, STAR was found to outperform expert human surgeons in manual laparoscopic surgery conditions and clinically used RAS approaches with respect to various anastomosis metrics. To approach the challenges raised by feature extraction from soft tissues, STAR integrates a vision system relying on Near-InfraRed Fluorescent (NIRF) tags placed in the intestinal tissue and a specialized NIRF camera to track those markers, while a 3D camera records images of the entire surgical field. A force sensor placed between the robot end-effector and the surgical tool enables one to limit suture tension between thread and tissue. The automation software generates a geometrically optimized suture plan by fitting a polyline through the tracked 3D NIRF markers, and it adjusts that plan as tissues move and undergo intraoperative deformations. The surgeon is additionally able to make positional adjustments in supervisory mode.

More than slave devices
A comprehensive review since the mid-1990s of commercial surgical robots available on the market or as research platforms is provided in [25]. The listed robots are grouped into the four categories of commercially available, discontinued, under development and advanced research prototypes. Various surgical robots are more than slave devices and have been successfully deployed in operating rooms. For example, ACROBOT [26] and RIO [27], used for precise bone cutting, operate with active constraint in a shared control modality. ROBODOC [28], used for orthopaedic surgery, and CyberKnife [29], used for stereotactic radiosurgery, are examples of robotic systems working in supervised autonomy by carrying out image-based preoperative plans without human interruption except in emergency situations. And NeuroMate [30], a robotic system for minimally-invasive neurosurgery, is endowed with supervised planning capabilities, taking advantage of fiducials markers located in the patient's head using ultrasound, while its trajectory planning is based on high-resolution preoperative images.
Autonomous robotic planning and execution of surgical task has been selectively achieved in orthopaedics and other areas where the surgical environment is chiefly constituted by rigid anatomical parts. An early successful example is the already mentioned ROBODOC surgical system developed for hip arthroplasty [31]. This system consists of a preoperative planning computer workstation and a robotic arm equipped with a high-speed milling device to prepare the femoral canal. ROBODOC is an example of how robotic technology can enhance in various ways surgery outcomes. Tools such as high-frequency ultrasound, drills, microscopes and endoscopes have improved patient outcomes, additionally making medical and surgical practices possible that would not exist otherwise. In [32] the effects of conventional hand rasping and robotic milling on the clinical and radiographic results of 78 cementless total hip arthroplasties have been comparatively evaluated under the same pre-operative planning conditions, i.e. the same computed tomography (CT)-based 3dimensional planning obtained by the ROBODOC workstation. Notably, there were no intraoperative femoral fractures in the robotic milling group, and a radiographically superior implant fit was additionally obtained.
ROBODOC operates in rigid anatomical environments, whereas soft-tissue surgery takes place in non-rigid anatomical environments. Less predictable scene changes occur in the latter case, insofar as soft-tissue surgery requires one to constantly adjust to non-rigid bodily deformations and physiological motion due to respiration and blood flow. This contextual factor is the source of major challenges towards an increasing automation of surgical tasks in soft-tissue surgery research. Nevertheless, the STAR system [24], described in the previous subsection, affords a demonstration of autonomous planning and task performance as far as designated sorts of soft-tissue suturing are concerned. While extremely innovative and promising, this approach is limited by (1) the need of injecting a fluorescent dye to extract features under fluorescent imaging, (2) the lack of manipulation capabilities, (3) a very large footprint that is incompatible with microsurgery.
In conclusion, the selection of systems mentioned in this Section² is sufficient to highlight different dimensions of robotic autonomy in surgery: (i) the wide variety of existing robotic autonomies in MIS, spanning VFs automation and generation, supervised task performance, and pre-operative planning; (ii) the special technological challenges towards robotic autonomy arising in anatomical contexts that are comparatively less structured; (iii) the cognitive and sensory-motor skills licensing robotic autonomy, emphasizing in particular the challenges arising from shared control modalities and from predictability issues concerning learning systems in particular. Roughly speaking, dimension (i) has to do with the what of autonomy, (ii) with the where of autonomy and (iii) with its how, in the sense of robotic capabilities that are sufficient to undertake some given autonomous action.
Most forms of autonomy that we have considered so far are well within the capabilities of human surgeons, in the sense that human surgeons can step in and replace the autonomously acting robot if something goes wrong. We turn now to consider powerful drivers towards robotic autonomy in microsurgery, as an exemplary case which enables one to explore robotic autonomy prospects in connection with surgical procedures that one encounters at the limits or even beyond the pale of human surgical capabilities.
devoted microsurgeons, performing at the limit of human hand precision and physiological tremor, are in the position to offer such reconstructive procedures. However, the global incidence of chronic conditions which may benefit from microsurgical interventions -including those arising from diabetes complications and a variety of other cardiovascular and neurological disorders -is rapidly growing, jointly with the numbers of the geriatric population, where the incidence of these chronic conditions is particularly high. These epidemiological trends are expected to boost the demand for microsurgical interventions. Robotics can help to reduce the mismatch between the growing demand for microsurgical interventions and its offer, which is supported by relatively small cohorts of highly-trained microsurgeons. Clearly, microsurgery is a surgical domain where robotics can really make a significant difference. Notably, robotic technologies may support wider groups of surgeons to perform operations that are otherwise limited to very experienced clinicians and may contribute to reduce dramatically the high numbers of untreated cases. Third, various parts of microsurgery interventions -such as anastomosis, dissection and suturing -have the potential to be autonomously performed by robots with significant advantages for patients. Anastomosis, which consists in the suturing of two severed tracts of tubular structures, is a notable case in point. The small scale of these tubular structures -vessels, nerves, ducts -and a very constrained workspace usually pose serious challenges for human surgeons performing anastomosis. Automated anastomosis has the potential to reduce the surgeons' burdens and fatigue, increasing at the same time accessibility to microsurgery for patients. Since the instrumentation required is rather standard, and well-understood is the behavior of the non-rigid anatomical structures that are involved, anastomosis is a promising candidate task for autonomous robotic execution in microsurgery.
From an ethical viewpoint, the automation drivers of wider accessibility to microsurgery, greater quality of microsurgical procedures and their improved outcomes are justified by the bioethical principle of medical beneficence, which demands the implementation of feasible clinical and research actions to promote the wellbeing of patients. Robotics research actions ultimately aimed at promoting the wellbeing of patients may span from the development of robotic systems for tremor elimination and motion scaling -helping microsurgeons in the execution of sub-millimeter surgical gestures -to VFs automation and generation, up to and including the autonomous robotic performance of anastomosis and other microsurgical tasks. The outcomes of these research efforts are eventually expected to improve the efficacy, efficiency and repeatability of many kinds of microsurgical procedures.

Natural Orifice Transluminal Endoscopic Surgery
(NOTES) [50] -This is another potential field of application of robotic microsurgery, which includes any surgical operation performed only through a natural orifice (e.g. anus, vagina, mouth, urethra). For example, in transanal microsurgery the dimension of the surgical space is so constrained to make dissection and even more suturing really difficult to perform, requiring extensive training on microsurgical endoscopic platforms. A robotic micro-surgical platform may improve precision and maneuverability, in addition to overcoming the technical limitations of executing dissection and suturing through a minimally invasive scarless access.
Current research on robotic automation in microsurgery is focused on the development of adaptable instruments, for both existing platform, such as the da Vinci [51], and robotic devices designed from scratch. The da Vinci platform is being extended by developing novel adjunctive tools that provide enhanced optical magnification, micro-Doppler sensing of vessels down to a 1mm size, vein mapping capabilities, hydro-dissection, micro-ablation capabilities and other required functionalities. Additional examples include the robotic research platforms NeuroArm, RAMS, ER2, MicroSure, EurEyeCase and Pico that we now turn to briefly describe.
The NeuroArm robot [52] developed at the University of Calgary is the result of pioneering efforts in the field of robotic neurosurgery towards brain tumours removal. It is composed by two SCARA-like manipulators and by a surgeon console providing a 3-degrees of freedom (DoFs) force feedback, a complete operating environment view showing in parallel the 3D stereoscopic view, the magnetic resonance (MR) image of the patient and the control panel. The high cost of all integrated components, compatible with MR imaging, has prevented its commercialization. RAMS (Robot Assisted Micro Surgery) [53] is a telerobotic platform constituted by two 6-DOF arms control and by two 6-DOF force-reflecting haptic interfaces with programmable controls. The robot has been used especially for eye surgery. Despite its many benefits, this robotic platform has never been commercialized, possibly due to insufficiently miniaturized tools and counterintuitive master replication of slave kinematics. The ER2 (Eye Robot2), is a microsurgical robot developed by Johns Hopkins University for eye surgery characterized by a Remote Centreof-Motion (RCM) mechanism. An integrated custom microforce sensing for micro-force guided cooperative control allows actively guiding the operator in setting up virtual fixtures to help protecting the patient [54]. MicroSure is a robot able to carry out highly precise operations to repair blood vessels or nerve fibres in hand or face reconstructions. The robot has two joysticks operated by the surgeon. The movements of the joystick are scaled, with a large deflection of the joystick translated into a small movement of robot arms. Moreover, the robot achieves a five-time greater precision than by hand. A foot pedal allows the surgeon to select the degree of scaling. The robot also filters out hand tremors and gives the robot arms force feedback [55]. The EurEyeCase [9] is an integrated setup for vitreoretinal eye surgery. The system is based on existing hardware and affords a robot-assisted operation suite used for the treatment of retinal vein/artery occlusion through cannulation and epiretinal membrane treatment. The Pico [56] is a master-slave platform designed for microsurgery, equipped with the smallest worldwide available instruments with wrist articulation at the tip. Differently from other surgical robots, the articulated microinstrument tips for the first time reproduce accurately the shape and size of traditional microsurgical instruments thanks to a proprietary microfabrication method.
On the whole, the bulk of research is currently aimed at creating new robots and tools for tremor compensation and motion scaling that are operated in master-slave control mode. However, in [15] methods to compose within a shared control mode autonomous control actions with teleoperation are presented for a surgical robot. The cooperation between robot and surgeon may help to achieve important surgical goals such as micro-damage thanks to accurate positioning, non-destructive diagnosis and testing, and the extension of robot-aided surgery to interventions of other kinds. In [57] the authors propose a robotic autonomous control method for avoiding possible collisions between the shaft of a surgical robotic instrument and surrounding tissues, while in [42] a novel force sensing cannulation needle based on Fiber Bragg Gratings is developed for instrument insertion precision using a comanipulation strategy instead of a teleoperation strategy. Shared control strategies mainly based on VF are presented in [53,54]. Shared control is a suitable approach to integrate the best of human surgeon capabilities (creativity, adaptability, interaction) and automation (speed, reliability, precision and inexhaustible task execution capability), from which microsurgery could reap great benefits.
An exemplary case in microsurgery where great benefits can be gained from shared control integrating the best of human capabilities and automation is Transanal Endoscopic Microsurgery (TEM).
Rectal tumours surgery can gain great benefit from the minimally invasive technique for neoplasm excision. TEM avoids conventional pelvic resection surgery along with its risks and side effects providing excellent functional and oncologic outcomes in the treatment of large sessile benign rectal lesions and selected early rectal cancers [58]. However, the associated cost and complex learning curve limit TEM. Often, despite training, surgeons performing TEM prefer not to close the rectal defect, as this is surely the most difficult part of the procedure, due to the narrow space constraint. This entails a relatively limited but nonnegligible rate of pelvic abscesses with even severe pain in some cases and risk of systemic sepsis [59]. Robotics in this field can provide great benefit overcoming human performance thanks to submillimetre precision of motions, vision magnification and enhanced tool dexterity. Nowadays, experience with Robotic Transanal Surgery (RTS) is still limited to few centres worldwide. Nevertheless results are encouraging.
The main limits of robot-aided surgery are (i) the absence of a system designed ad hoc and (ii) the narrow space of the surgical site. This means that also with robotic systems like the da Vinci robot there are limitations [8]. Robotic instruments, even though endowed with enhanced dexterity, need to be teleoperated by the surgeon. Only very experienced physicians can perform precise manoeuvres. This also entails surgeon fatigue that influences the results. The previously described real-time sensing and shared control strategies can provide the missing abilities to a teleoperated robotic platform.
To sum up, the state of the art in robot-aided microsurgery is mostly based on the use of teleoperated/telemanipulated robots possessing hardly any autonomy. Very few robot prototypes are provided with semiautonomous control strategies for sharing autonomy with the surgeon in few elementary and selected tasks. However, limits of human surgical capabilities in certain procedures, such as anastomosis of small vessels and nerves or suturing in narrow space, and the advances of technologies and research are driving new robotic developments towards more challenging autonomous task execution.

Robotic autonomy and MHC
Hierarchies of surgical robot autonomy. A hierarchy of six different levels of autonomy for medical robots, including surgical robots, was introduced in [3]: starting from medical robots having no autonomy (level 0), to robotic assistants constraining or correcting human action (level 1 autonomy), robotic systems carrying out tasks designated by humans and under human supervision (level 2), and robotic systems generating task execution strategies under human supervision (level 3), this hierarchy is ideally rounded out by reference to technologically more distant perspectives: at level 4 one envisages robots performing under human supervision an entire medical procedure (e.g., an entire intraoperative surgical intervention), while full autonomy without any human supervision characterizes the culminating level 5.
An ethical argument for introducing technologically distant levels 4 and 5 for surgical robots is presented in [4]. This argument is grounded in the bioethical principles of beneficence and distributive justice, insofar as the idea of a fully autonomous surgical robot reflects "a potential goal in automated healthcare in that it ...could be distributed to remote areas for improving patient access to high-quality surgical care." For all one knows today, however, the cost of deploying fully autonomous surgical robots, if any, may turn out to be so prohibitively high that their use will be confined to circumstances in which more traditional surgery is not available (e.g. astronauts in outer space or wounded soldiers in enemy territory) and to a uent population segments, thereby defeating distributive justice motivations [60] and severely limiting their beneficence impact. Briefly, autonomy levels 4 and 5 are not likely to be instantiated in a foreseeable technological future by commercial or research robots; ethical analyses about these levels are correspondingly bound to stay purely speculative for some time to come.
To avoid the nested "ifs" of speculative ethics [61], we focus here on present and technologically imminent surgical robots which one can already endow with lower autonomy levels. Specifically, we are concerned with levels up to and including autonomy level 3, for the selective review of technologies and systems carried out in the previous section demonstrates that all these levels are already instantiated by some available systems, and those ongoing research efforts on the frontiers of robotic microsurgery are well accommodated within those autonomy limits.

Meaningful human control of surgery robot autonomy.
A key requirement advanced in [3] on each hierarchy level falling short of full robotic autonomy is that "the treating physician is still in control to a significant extent". Straightforward ethical justifications for this claim are usually advanced by appeal to the ideas of moral responsibility and accountability. Since present and prospective robotic systems are not morally responsible entities, they have no moral duties, and cannot be praised, blamed or punished for the outcomes of their actions. Only human operators have extensive duties to oversee robotic action and to avoid the occurrence of related harms. These duties must be sensibly distributed among involved human subjects, in accordance with their respective competences and professional roles -medical doctors, other members of medical staff and institutions, insurers, engineers, producers and designers of robotic equipment.
The requirement of exerting human control "to a significant extent" on increasingly autonomous medical robots parallels similar demands for the meaningful human control (MHC) of Autonomous Weapons Systems (AWS) originally advanced by the NGO Article 36 in a 2013 report [5]. It is claimed there that "... the exercise of control over the use of weapons, and concomitant responsibility and accountability for consequences are fundamental to the governance of the use of force and to the protection of the human person", and furthermore that "...it is apparent that having a person 'in', 'on' or 'touching' 'the loop' of a weapons system does not in itself ensure that meaningful human control is exercised -for example, if that person simply pressed a 'fire button' every time a light came on without having any other information. " Discussion of what MHC should amount to in the case of AWS is well under way in both academic [62][63][64] and international policy fora [65]. Moreover, applications of the MHC concept to increasingly autonomous driverless vehicles and other autonomous robotic and computational systems are being actively explored [6]. The concrete instantiation of the first three levels in the above hierarchy suggests the opportunity of examining in depth what MHC should amount to in the case of increasingly autonomous surgical robots too.
Level 0 MHC. MHC on surgical robots without autonomy is imposed and exerted by design. Surgeons govern in master-slave control mode the entire surgical procedure, from data analysis and planning, to decision-making and actual execution. By scaling motion and attenuating tremors, robots compensate for human sensory-motor limitations and improve the execution of imperfectly conveyed motion commands. This correction, however, is applied solely with the goal of bringing resulting actions closer to the actual intentions of human surgeons. Adherence to human intentions is similarly pursued in current experimental work on level 0 autonomy for microsurgery robots, with the proviso that microsurgical robots must reflect in this case human intentions that are mostly directed towards the performance of actions that are at the limits of or even beyond human sensory-motor capabilities. A significant case in point is the extension of the da Vinci platform towards sensory-motor functionalities enabling submillimeter execution of surgical gestures.
The word 'meaningful' in MHC is meant to exclude control modes that one may nominally argue to incorporate humans in the control loop, even though human control is reduced there to a perfunctory validation of robotic actions, that is, in the absence of sufficient time and rationale to make an informed human judgment and to undertake the attendant actions. At autonomy level 0, these provisos are straightforwardly satisfied, as surgeons are granted by design the sensory and mental processing windows that are needed to evaluate whether motion scaling and tremor attenuation result into actions faithfully reflecting their intentions, and to adopt countervailing measures otherwise. At higher autonomy levels, as we shall presently see, ensuring the "meaningfulness" of human control gives rise to more subtle issues.

Level 1 MHC.
In contrast with level 0 autonomy systems, robotic surgery assistants placed at autonomy level 1 are not invariably bound to bring about a better fit between human intentions on the hand and imperfectly conveyed human commands on the other hand. Indeed, robotic assistants applying VFs may act against some consciously entertained intentions of human surgeons, by actively constraining and modifying humanly determined trajectories of surgical instruments. Analogous VF functions are under development in robotic microsurgery, as testified by the micro-force guided, shared-control system actively guiding human operators in setting up virtual fixtures in eye surgery [54] and other microsurgical areas mentioned in the previous section.
Clearly, to exert MHC on level 1 autonomy surgery robots, humans must have the option to override robotic corrections to their actions, by enacting a second-level human control overriding first-level robotic corrections. But to let humans exert effectively this option, one must ensure sufficient temporal and informational resources for a considerate judgement to be expressed and overriding commands to be delivered. The modulation of these resources depends on the "how" of level 1 robotic autonomy. At this level, for example, VFs permitted and forbidden regions are identified either directly by surgeons or else by robotic assistants endowed with VF generation capabilities that are possibly attained by means of some learning procedure. In the presence of autonomous VF generation and the uncertainties that are inherent in the statistical modelling of machine learning processes, MHC requires human operators to apply more demanding (e.g. more frequent) perceptual and cognitive assessment of robotic VF action. Thus, one size of MHC-oriented human judgment does not fit all robotic systems placed at autonomy level 1.
The need for MHC at autonomy level 1 and the identification of corresponding human control burdens provide distinctively ethical motivations to introduce intralevel distinctions and to refine accordingly the interlevel hierarchical distinctions for surgical robots which are chiefly based on what the robot autonomously performs. The kind of refinement proposed here for autonomy level 1 is solely based on the "how" of surgical robot autonomy. We now turn to examine intralevel refinements for autonomy level 2 that are alternatively based on the "where" of surgical robot autonomy.

Level 2 MHC.
At this level, humans select a task that surgical robots perform autonomously. The surgeon's supervising role consists in the hands-free monitoring of robotic task execution, and prompt overriding if needed. Hence, the robotic system is under the surgeon's discrete, rather that continuous control. However, one size of discontinuous control does not fit -for MHC purposes -all systems enjoying this level of autonomy. This is readily brought out by comparing the level 2 task autonomies of the RO-BODOC and STAR systems, respectively. The former has been long endowed with supervised task autonomy as far as bone milling in designated sites is concerned. The latter performs autonomously the task of intestinal anastomosis on in-vivo tissue in an anaesthetized pig, demonstrating potential extensions to interventions on human patients. As noted in the previous section, these instances of level 2 task autonomy differ significantly from each other insofar as the where of task execution is concerned. ROBODOC's surgical sites are rigid anatomic structures, whereas STAR operates on soft tissues. In the latter case, occurring scene changes due to physiological blood flow and respiration are more difficult to predict.
ROBODOC and STAR surgical systems are two examples of different Technology Readiness Levels (TRLs) influenced by surgical environments and their predictability. The former system is used for a clinical standard procedure, while the latter is still at a research level. Rigid and structured environments where ROBODOC operates allow safe autonomous task execution thanks to the possibility of precise measuring and accurate scene changes predictions, namely detection and tracking of both tools and anatomical parts. On the contrary, the soft and deformable surgical sites where STAR operates raise higher challenges for proper autonomous task execution.
Thus, under the communal requirement of discrete human control and hands-free supervision, one finds MHC tasks that differ significantly from each other in the way of human perceptual and cognitive vigilance demands. Discrete perceptual sampling and cognitive evaluation for MHC purposes are arguably more demanding in the case of STAR-like systems, in view of less predictable and more likely occurring scene changes. There, additional MHC challenges concern the assessment of the robot's own adaptive response to unpredicted changes.
In prospective automation of microsurgical anastomosis, this MHC assessment concerns the motion of small tubular structures to be sutured and their slippery behaviour. If highly experienced microsurgeons only have the decision-making and operative competence to exert proper MHC on automated microsurgery anastomosis, then a tension arises between MHC demands and the bioethical beneficence motivation encouraging the automation of microsurgical anastomosis. The tension is specifically between the competences of surgeons that are required to exert effective MHC and the bioethical beneficence perspective of enabling less expert surgeons to take advantage of autonomous task execution in microsurgery: will these less expert surgeons be equally capable of exerting MHC on automated anastomosis? Can effective MHC training procedures for the wider groups of less experienced surgeons be put in place to defuse the tension between MHC and medical beneficence demands?
The need for MHC at autonomy level 2 and the corresponding identification of suitable human judgment and control burdens provide here too distinctively ethical motivations to introduce an intralevel refinement, which is based on the where of surgical robot autonomy, as distinct from the level 1 autonomy refinement advanced on the basis of its "how". We now turn to consider again the how of surgical robot autonomy, but presently in connection with the proper exercise of MHC at autonomy level 3.

Level 3 MHC.
Robotic systems endowed with level 3 autonomy generate task strategies under human supervision, and conditionally rely on humans "to select from among different strategies or to approve an autonomously selected strategy" [3]. To a limited extent, systems dynamically identifying VFs and generating optimal control parameters or trajectories already achieve this level of conditional autonomy. Indeed, the system NeuroMate [30] mentioned in the previous section performs trajectory planning based on high-resolution preoperative images. Other pertinent examples include Cyberknife, which generates execution plans that the surgeon is required to review and approve [4], and STAR as far as anastomosis strategies generation is concerned.
MHC for level 3 autonomy distinctively requires surgeons to decide competently whether to approve one of the robot generated strategies. This decision presupposes that surgeons understand the rationale for proposed strategies, are in the position to compare their respective merits, and to make up their mind in due time about which strategy to prefer over alternatives. Depending on the complexity of proposed strategies and surgical sites, MHC may incrementally raise human interpretability and decisionmaking challenges about robot generated strategies. Similar issues have already emerged in other areas of robotics and artificial intelligence: XAI (which is an acronym for eXplainable Artificial Intelligence) is a rapidly growing research area aiming at the development of intelligent systems that explain the rationale for their decisions and actions, thereby reducing the opacity of automatic decisionmaking while supporting human understanding and management. By the same token, in the future MHC on level 3 robotic autonomy might be profitably enhanced by endowing robots with explanation modules for generated strategies. This need may especially arise in connection with ex-planations for learned strategies, in view of interpretability problems that may affect machine learning results [66].
Today, the learning of surgical strategies is based on data sets formed by humanly generated strategies. In a more distant future, interpretability and explanation issues arising in the context of MHC for level 3 robotic autonomy may become increasingly acute if datasets for learning how to generate intervention strategies progressively shift from data concerning human-generated strategies to robot-generated strategies and corresponding clinical outcomes. In the end, whether and what kind of explanations for robot-generated strategies must be supplied to exert proper MHC is an important ethical policy issue. The RAS community must address this policy issue by reflecting on the information and reasoning steps that robotic systems use in strategy generation and their possible lack of transparency to human surgeons.
MHC in a broader bioethical context. We have so far considered kinds of human vigilance that are required at each autonomy level -and differentially within each level to ensure proper MHC. These reflections have chiefly to do with the surgeon's prospective responsibilities, that is, with what she ought to do or attend to when performing surgery with the help of robots that are endowed with some sort of autonomy. A thorough analysis of prospective responsibilities induced by the MHC requirement is important to shape training programs for surgeons. In particular, the non-maleficence bioethical principle requires proper training to provide conceptual tools countervailing positive machine biases, which may wrongly induce human surgeons to trust more what the robot does or proposes to do rather than their own contrasting judgment.
A thorough analysis of MHC-related duties plays an equally significant role in evaluating what are the surgeon's retrospective responsibilities, if any, whenever something goes wrong intraoperatively. Indeed, a surgeon might be held responsible for damages caused by an autonomously performing robot if she failed to exert MHC properly and if the harm in question might have been averted had she performed more carefully her MHC duties. By the same token, retrospective responsibility allegations against surgeons for damages caused by an autonomously performing robot might be correctly rebutted and possibly diverted towards other human agents by showing that the specified MHC duties were carefully complied with.
The discussion of MHC has been carried out so far under the assumption that throughout the examined autonomy levels one has to fulfil the requirement that "the treating physician is still in control to a significant extent" [3]. It is worth mentioning at this point some more speculative questions which may raise properly ethical challenges for this entrenched ethical requirement. It was noted above that the word 'meaningful' in MHC is meant to exclude control modes that one may nominally argue to incorporate humans in the control loop, but do not provide humans with sufficient processing time and data to make an informed judgment before licensing robotic autonomous action. However, one may sensibly wonder whether this kind of human judgement and approval on robot-generated task strategies is always compatible with medical beneficence. There might be expected benefits for patients flowing from faster-than-human task planning and execution capabilities which would recommend activating by default a robot-generated strategy and confining MHC to a veto power to exert during tight temporal windows. Even more drastically: Are there situations in which emergency considerations suggest leaving surgeons without any gating power and to endow robots with unconditional task performance and strategy selection powers? A candidate case in point is severe spontaneous bleeding, as occurring in aneurysms, where the preliminary crucial task of stabilizing patients by some endovascular intervention might be more rapidly performed by robots. Unconditional execution of restricted tasks on medical beneficence grounds suggests an ethically motivated refinement of the autonomy levels hierarchy introduced in [3]. The new autonomy level proposed for consideration is restricted unconditional autonomy for surgical robots, to be placed in between autonomy level 3 (conditional autonomy) and level 4 (high autonomy). The latter autonomy level 4 is exemplified in [3] by means of a robotic resident performing an entire surgery under human supervision. The new restricted unconditional autonomy level singles out unconditional strategy planning and execution of quite limited subtasks falling short of entire surgery execution. This new level has not been instantiated yet, but appears to be less technologically remote than levels 4 ad 5.
MHC and patient autonomy. Robotic surgery involving increasingly autonomous systems inherits and extends ethical issues in RAS concerning the respect for patient autonomy and its application to informed consent procedures. Aspects of patient autonomy that must be carefully addressed in RAS are confidentiality maintenance and the adequacy of technological information provision. Special care must be taken that confidentiality be observed by robotic engineers and other non-medical staff whose presence might be occasionally needed in the operating room [67]. Moreover, one must evaluate whether information disclosure must include selective information about the involved robotic systems and, if so, which amount of information is sufficiently rich and understandable for autonomous patient reflection and decision-making. In the case of increasingly autonomous robots, an integral part of this information may concern the overriding privileges of human surgeons and their MHC powers more generally. Indeed, this information may prove crucial for patient proper evaluation and acceptance of risk arising from the use of autonomous surgical robots, especially in the early stages of their introduction, when reliable statistical projections about their future behaviours are not available yet.
Comparative human-robot benchmarks have been additionally identified as significant elements of information disclosure about robotic technologies in surgery. The informed consent process, it has been claimed, "should go beyond the usual one when conventional surgery is contemplated. One of the critical ethical concerns is whether robotic surgery is at least equal to conventional surgery" [67]. This ethical concern may be put to rest in connection with task autonomous surgical robots by pointing to the evidence that the robot performs statistically better the task at hand in terms of various metrics, including execution and recovery times, precision, bleeding, pain and so on. A promising example is afforded by the experimental comparisons mentioned above between STAR and human performances in intestinal anastomosis on an animal model. In extending the use of STAR-like systems to operations performed on humans, an ethical concern may reasonably arise about unpredicted, abnormal and hurtful autonomous robotic behaviours in less structured soft tissue surgical sites. Proper information about MHC procedures to stop autonomous robotic action and available statistical data showing the unlikelihood of these adverse events may alleviate this concern, thereby enhancing the surgeon-patient trust relationship [18]. If the surgeon does not feel sufficiently competent to answer pressing patient's questions about models of autonomous robotic action and the related likelihoods of abnormal robotic behaviours, then the surgeon may still bring the other hidden pillar of RAS -the robotic engineer -to the forefront of information disclosure and preserve by this act of delegation her trust relationship with the patient.

Discussion
The increasing capability of robots to perform autonomous actions and complex tasks raises responsibility and accountability issues in a wide variety of application domains. In some military, industrial and service application domains, ethical analyses of these issues are well under way and are mostly oriented towards the development of ethical policies requiring MHC on robotic autonomous behaviours.
The situation is rather different in robotic surgery. Presently, the expression "surgical robot" is mostly used to denote teleoperated devices which reproduce human surgeon gestures and improve overall human performance by scaling and filtering, to achieve shorter recovery times for patients, in addition to reduced traumas, pain and postoperative infections. Surgical robot autonomy, presently rather limited in scope, is usually confined to structured bodily environments, that is, surgical sites in which proper autonomous execution takes advantage of precise measurements of bodily parts and highly predictive models of bodily changes. Significant cases in point illustrated above include MIS robots for orthopaedic and neurosurgery interventions. Recent research developments point to extensions of surgical robot autonomy to tasks performed in less structured environments, such as the soft tissue intestinal suturing mentioned above. And one finds powerful drivers of the same kind in microsurgery, where the automation of small vessel anastomosis and suturing in narrow surgical spaces promises to bring significant benefits to patients. These forthcoming developments suggest the opportunity to start an in-depth discussion of what MHC should distinctively amount to in robotic surgery and how one should modulate MHC considering autonomous task assignments, contexts of use and related robot skills.
Here, discussion of MHC was anchored to identified forms of present and imminent surgical robot autonomy or, in other words, to levels that are (or are likely to be soon) instantiated in proposed hierarchies of increasing autonomy for surgery robots. It was pointed out that the MHC normative requirement provides ethical motivations to refine these technologically oriented hierarchies and introduce finer intralevel distinctions. In particular, one size of MHC does not fit all systems that one finds at one and the same autonomy level. A consideration of the where and how dimensions of autonomous task execution is needed to modulate properly MHC within each hierarchical level. Intralevel classifications of systems based on these dimensions are crucial to adapt MHC to task environments and robot capabilities, thereby excluding control modes that one may nominally argue to incorporate humans in the control loop, without letting them have sufficient information and processing resources to make an informed judgment before approving robotic autonomous action. Also, the more remote possibility was noted of introducingin view of envisaged medical beneficence implicationsan autonomy category for surgical robots above level 3 and below level 4 in [3]. This new category is restricted unconditional autonomy on selected tasks planning and execution, that one might wish to grant to robots capable of identifying and dealing more rapidly than human surgeons with some designated emergency situations. Restricted unconditional autonomy on selected tasks planning and execution might be granted in the future by comparing human and robot performances and benchmarks. However, one cannot exclude that future advancement of robotics research and technologies may even lead to robots performing new practices that overcome human surgical capabilities, so that human benchmarks cannot exist. In microsurgery, these circumstances might arise from advanced miniaturization of endoluminal and endovascular applications in robot-aided surgery. More concretely, in the light of the current state of art, the next frontier is to automate selected tasks using sensors as real-time feedback for a surgical procedure and to develop models of humanrobot shared control taking in due account the ethically motivated requirement of meaningful human control on surgical robot autonomy.