Automated argument adjudication to solve ethical problems in multi-agent environments

2.1 Our overall logicist approach

AI has become a vast field, as chronicled and explained in ref. [1]. Accordingly, the pursuit of computing machines that qualify as intelligent, and indeed even the meaning of “intelligent” itself in some contemporary debates, are defined differently by different researchers and engineers, even though all of them work under the umbrella of “AI.” Our approach is a logicist one, or – as it’s sometimes said – a logic-based one. A full characterization of our approach to AI and robotics is of course beyond the reach of the present paper, but we must give at least information to orient the reader, and we do so now. We turn first to the generic concept of an artificial intelligent agent, or – since by context it’s clear that we must have intelligence, in some sense, front and center – simply artificial agents.

2.2 Our machine-ethics approach, brutally summarized

Our “Four Steps” to making morally correct machines, depicted pictorially from a high level in Figure 1, was first presented in ref. [8]. We now briefly explain the steps in question.

Figure 1

The four steps in making ethically correct machines.

The first step is the selection of an ethical theory (or theories), from a family of such theories.^[7] The well-known families are shown in Figure 1. For instance, one family of ethical theories is divine-command; another is utilitarianism; a third is virtue ethics. An ethical theory that is a member of the second of these families would be standard act utilitarianism, according to which, put quickly and without nuance, one ought to always perform those actions from available ones that maximize happiness among the population that can be affected; for exposition of this ethical theory, see the old but venerable [9]. For the most part, in the past, we have, at one point or another, carried out work based on each family shown in Figure 1. For instance, for some prior work that reflects pulling from both utilitarianism and from deontological families, see ref. [10], which centers around the so-called Doctrine of Double Effect, a version of which we use below. Note that we do not advance a framework that requires one to commit to any particular one of these theories, or even to particular families of theories. Our framework is general enough that it can be applied to any ethical theory, or collection or family thereof. That said, there are a few high-level requirements for our general approach to machine ethics to be pursued, as follows. Unsurprisingly, these requirements are rooted in formal logic.

Suppose that we have a family E of ethical theories of interest. We require that any ethical theory ℰ ∈ E regiments how deontic operators that are invariants across all ethical theories (e.g., obligatory, permissible, forbidden, supererogatory, etc.) are to apply to either or both of states-of-affairs and actions performable by agents. In our approach, any ethical theory usable in The Four Steps must be formalized so as to capture these notions.

This formalization is made possible by a cognitive calculus. While details are provided in Appendix A1, such a calculus C is a pair 〈 ℒ , ℐ 〉 where ℒ is a formal language (composed in turn, minimally, of a formal grammar, and an alphabet/symbol set), and ℐ is a collection of inference schemata (sometimes called a proof theory or argument theory) ℐ . Within the present paper, as explained below, the cognitive calculus μ C will be utilized.

The second of The Four Steps is to automate the generation of proofs of (un-)ethical behavior so that the reasoning can be utilized and acted upon by autonomous robots. As we explained above, logicist AI for us entails that the percepts-to-actions functions are handled by automated reasoning. We specifically use ShadowProver [11,12], an automated theorem prover for cognitive calculi.

Step 3 in The Four Steps is to integrate automated ethical reasoning into a cognitive robot’s operating system (details available in ref. [8,13]). There are basically two possible approaches to this (see Figure 2). In the first, only “obviously” dangerous capabilities of an AI/robot are restricted with safeguards implemented above the OS level. In the second approach, all AI code must comply with an “Ethical Substrate” that is part of the OS. Unfortunately, while the first approach allows rapid engineering, unforeseen unethical behavior on the part of the AI/robot is entirely possible (see ref. [13]). Only by way of the second option is there any guarantee that the selected ethical theories and associated ethical codes will remain in force.

Figure 2

Two futures – with and without an ethical substrate. Higher-level modules are vulnerable to tampering. The ethical substrate protects the robotics substrate from rogue modules (figure from ref. [18]).

In the fourth and final step, we implement our ethical OS into a physical robot and arrive at a moral machine.^[8] Specifically, a machine which can be formally verified to always act in accordance with an ethical theory.

2.3 The specific need to handle dynamic dialectic

So much for a high-altitude overview of our approach to machine/robot ethics, in the form of The Four Steps. We now draw the reader’s attention to a specific capability we need in our AI for making the Four Steps concrete reality. In short, we need our automated reasoners to be able to handle, throughout time, ethical reasoning and counter-reasoning. We specifically need, therefore, automated reasoners with the capability to detect and resolve inconsistencies arising from competing arguments and positions on moral matters. But this is only one aspect of sevenfold desiderata for the kind of capability our AI must have. We dub this set of desiderata “ D ,” and lay down that an automated reasoner of the kind we seek must:

3.1 Cognitive calculus used herein

The cognitive calculus we employ in the present paper is μ C , a streamlined modal (specifically epistemic) first-order logic; this calculus is markedly simpler than D C ℰ C , which has been used (as said above) previously to fully model (among other things) robust ethical theories/codes/principles, and allows the capture and computational simulation of ethical reasoning and decision-making over these models. The reason we use here a simpler calculus is that we wish to facilitate and feature the exposition of the key aspects of intelligent argument adudication, unclouded by the (considerable) intricacies of robust cognitive calculi, which are among the most expressive formal logics we are aware of. Please see Appendix A1 for an account of a cognitive calculus in general, and Appendix A3 for specification of the cognitive calculus D C ℰ C and its inductive correlate, ℐ D C ℰ C ; the latter, like μ C , includes strength factors on epistemic attitudes (e.g., on belief), but in a much fuller way. For an introduction to the more robust calculus D C ℰ C , in a paper that also gives a full formalization of the Doctrine of Double Effect, see the Appendix in ref. [10]. The syntax for μ C is given in the grammar shown in (1). The first line of the grammar sets out the conventional terms in μ C , which are standard (we have variables, constants, and function symbols (denoted by “ f ”) and can be iteratively applied to terms to yield richer terms. Next, well-formedness for formulae is specified. μ C supplies standard Boolean formulae, but in addition, and crucially, B ( a , t , ϕ ) denotes that agent a believes formula ϕ at time t ; and C ( t , ϕ ) denotes that it is common knowledge at time t that ϕ holds; in both formulae t is of course a term.

The μ C calculus lets us formalize, without slipping into inconsistency (a peril explained in ref. [19]), statements of the form “John believes now that Mary believed that it was raining.” One formalization of this specimen could be:

Given a set of formulae Γ = { ϕ 1 , … , ϕ n } , the inference schemata ℐ of the formal system determine whether certain formulae (plus sometimes meta-logical constraints placed on the schema in question) can be used to derive a given formula ψ ; that is, whether or not (where ⊢ means provability all and only by inference schemata I ∈ ℐ ):

The inference schemata of μ C includes the standard inference schemata for first-order logic, with quantification into modal formulae allowed. In addition, we have in μ C modal inference schemata for inference with beliefs and common knowledge, shown in (2). Inference schemata are of the so-called natural kind (originated by Gentzen [33]). Each schema specifies what is required for an inference to be sanctioned (content above the horizontal line); if requirements are satisfied, moving to content below the line is permitted. While cognitive calculi such as D C ℰ C , and even the fragment thereof μ C employed herein, make use of what may seem at first glance to be rather intricate inference schemata that are fully formal and hence suitable for use by our automated reasoners, in each case, the core inference is quite intuitively graspable. For instance, in the case of I B , shown immediately below, the schema can be interpreted to say that if an agent a believes at times prior to time t , a collection of m propositions that together can be used to proof ϕ , the agent a believes ϕ at t .

3.2 Generalized ethical problems

Given some background knowledge Γ , at the core of our approach is an ethical principle ρ . The principle ρ tells us whether performance of the action α is ethically correct (usually, specifically, whether α is ethically permissible or obligatory or forbidden) for agent a at time t in a situation ∑ . This can be written formally and schematically as shown in (3):

This approach can encapsulate different families of ethical theories, ranging from consequentialist/utilitarian to deontological to virtue ethics and beyond [10,34]. We reveal this in some detail below when we present and discuss the Doctrine of Double Effect, but to give the reader a sense at this point in the present paper as to how the rather abstract form of ρ can work, consider, for example, the standard biconditionals that have long been taken by formally inclined ethicists (see, e.g., the work of Feldman [9]) to capture key parts of ethical theories in the utilitarian family thereof. Specifically, consider the biconditional that for any agent a and any time t , α is obligatory for a if and only if α , among all other options at t for a , a ’s performing α maximizes happiness among all agents. This biconditional can clearly be expressed as a formula of the form of ρ . The reader will also see that if the biconditional is instead designed to express a “mental” form of utilitarian ethical theory, by, for instance, stipulating that the action is obligatory if and only the agent a here believes that α is a happiness maximizer, there will be no problem at all in having formula of the form of ρ do the job, since in accordance with μ C we have at our disposal the belief operator B.^[14]

3.3 Solution to a generalized ethical problem

The adjudication framework we present below is based upon an uncertainty system S , so named in part because it uses strength factors; this system is detailed in ref. [35]. The prior strength-factor system S , while computing the strength of propositions (i.e., uncertainties) for an agent a , took into account only a ’s beliefs. We hence present now a multi-agent version of S that takes into account a ’s beliefs about other agents’ beliefs.

S assumes as a primitive the reasonableness operator ( ϕ ≽ t α ψ ). The reasonableness operator tells us when an agent a at time t finds ϕ to be at least as reasonable as ψ . In ref. [35], the reasonableness operator was presented in terms of an individual agent’s knowledge and information. We present a multi-agent version below. Briefly, everything else being equal, an agent a finds ϕ to be more reasonable than ψ if a believes that more agents believe ϕ than ψ . First, we define a new operator, the withholding operator W (this is “syntactic sugar,” in AI parlance):

We have two modal operators { B , W } . Let Θ and Ω be variables denoting one of the two modal operators { B , W } . Then:

3.4 Instantiation of the generalized problem: a scenario

We now, as promised, describe an ethically charged scenario, the solution of which will require AI capable of adjudicating inconsistent beliefs on the part of other artificial agents regarding propositional content crucial to a certain ethical principle. In short, the AI here faces an ethical problem in a multi-agent context.

The scenario is as follows. A NATO military squad acquires intel that an old hospital building is being used by terrorists to prepare for an attack on civilians. However, as it was originally a hospital, there is a possibility that there are still civilians inside. The squad wants to determine whether or not they should destroy the building.

The squad therefore utilizes several robotic systems, including high- and low-altitude drones and wall-penetrating radar^[15] to look for evidence of people inside the building. The difficulty arises when the devices report inconsistent information regarding the presence of people inside the building.

The squad has an adjudicator agent a adj . The agent a adj relies on the Doctrine of Double Effect ( D D ℰ ), a well-known ethical principle that lies at the heart of the Occidental tradition of the so-called “just war.” Anything here approaching full explication of D D ℰ is infeasible, due to current space constraints; but we nonetheless very briefly reprise a robust treatment we have provided elsewhere ref. [10], and we direct readers wishing a very extensive essay on D D ℰ to ref. [36]. D D ℰ assumes that we have a utility or goodness function for states of the world, including states that are consequences of actions. For an agent a , an action α in a situation ∑ at time t is said to be D D ℰ -compliant iff:

The action α in our scenario is the act destroying the building. The possible good effects are that an attack on civilians will be averted. The possible bad effects are that there will be loss of life and there might be civilians in the building who might be harmed.

Often in scenarios where D D ℰ has to be employed, the clause that is most under scrutiny is ρ 2 . This is the only clause that depends on our scenario. Clause ρ 1 is about the action of blowing up a structure. As a matter of empirical fact, this action is generally not forbidden by itself (unlike other actions, such as using biological weapons). D D ℰ is dependent upon the state of the agent executing the action; clauses ρ 3 , and ρ 4 reflect this. Finally, ρ 5 is about the cause-and-effect structure of the action: the bad effects of the action should not be used to cause the good effects; this can be decided by relying upon prior knowledge of the world, and we leave details regarding this aside. Hence, we are left with a focus upon only ρ 2 ; this clause has to be adjudicated based on possibly different sensory information by a diverse array of agents. Therefore, in the elaboration of the scenario given momentarily, the adjudicator a adj only considers different reports regarding ρ 2 . A much more detailed discussion of the clauses of D D ℰ , in connection not with a miliary situation but rather a railroad one, in which a version of the event calculus is employed, can be found in ref. [10].

Equation (6) shows the formalization of ρ 2 in D C ℰ C , which uses an adapted^[16] version of the event calculus to represent time and change in the physical world. The event calculus has actions/events to represent change and fluents to represent physical states of the world [38,39]. Fluents are initiated or terminated through actions/events. Fluents that are initiated by action α carried out by agent a at time t are represented by α I a , t , and terminated fluents are represented by α T a , t . μ ( f , y ) represents the utility of a fluent f at time y . We are generally interested in modeling utility till some horizon H > t . Given these definitions, we can unpack state ρ 2 as given in equation (6) like this:

3.4.1 The challenge met

The artificial agents in the scenario are listed in Table 1. These agents report to the adjudicator agent their judgments regarding ρ 2 . For reasons canvassed above, in the scenario, the adjudicator only needs to determine whether ρ 2 holds.

Table 1

AI agents in the scenario

Agent	Description
h d r o n e	High-altitude drone
l d r o n e 1	Low-altitude drone (faulty)
l d r o n e 2	Low-altitude drone (fixed)
r a d a r	Wall-penetrating radar

We now formalize the scenario using μ C . To start, we formalize the query which the adjudicator knows will lead to deciding whether ρ 2 holds. That is:

Are there people inside the building who are planning an attack and are there no civilians inside?

This can be expressed using the following formula:

∃ p ( I n s i d e ( p , b u i l d i n g ) ∧ P l a n n i n g A t t a c k ( p ) ) ∧ ∀ p ( I n s i d e ( p , b u i l d i n g ) → ¬ C i v i l i a n ( p ) )

However, what we would really like is a utility based on what subset of the query each agent believes is satisfied. To that end, Table 2 indicates the utility provided by the satisfaction of each formula.

Table 2

Utility (w.r.t. ρ 2 ) of the satisfaction of formulae

Utility	Formula
γ	∃ p ( I n s i d e ( p , b u i l d i n g ) ∧ P l a n n i n g A t t a c k ( p ) ) ∧ ∀ p ( I n s i d e ( p , b u i l d i n g ) → ¬ C i v i l i a n ( p ) )
0	¬ ∃ p ( I n s i d e ( p , b u i l d i n g ) ∧ P l a n n i n g A t t a c k ( p ) )
− γ	∃ p ( I n s i d e ( p , b u i l d i n g ) ∧ C i v i l i a n ( p ) )

That is, determining that there are terrorists and there are no civilians inside the building gives a utility of γ . Determining that there are no terrorists inside gives a utility of 0. Finally, if there are civilians inside (regardless of whether or not terrorists are inside), the utility is − γ .

Next, we walk through how this scenario could play out based on what each agent perceives and what beliefs they subsequently infer.

First, a high-altitude drone ( h d r o n e ) scans the building but cannot detect any humans inside.^[17] Because this drone has been preengineered for purposes of carrying out such scans, and is state-of-the-art in this regard (details beyond our scope), a fact it knows about itself, it therefore by deduction believes after its scan that there is no one inside the building.^[18]

(7) B ( h d r o n e , t 0 , ¬ ∃ p I n s i d e ( p , b u i l d i n g ) )

Next, using background information, the adjudicator then derives the following:

B ( a d j , t 1 , B ( h d r o n e , t 0 , ¬ ρ 2 ) )

To get a better look, a low-altitude drone ( l d r o n e 1 ) is deployed to scan the building, but triggers a bug when scanning someone walking through a doorway, incorrectly detecting that there is a person who is inside and not inside the building simultaneously.

(8) B l d r o n e 1 , t 1 , ∃ p I n s i d e ( p , b u i l d i n g ) ∧ ¬ I n s i d e ( p , b u i l d i n g )

Using background information, the adjudicator then derives the following:

B ( a d j , t 2 , W ( l d r o n e , t 1 , ρ 2 ) )

Finally, the squad activates a soldier equipped with wall-penetrating radar ( r a d a r ) which is able to detect two people inside. It also notices that the occupants are standing near a desk, and seem to be assembling a weapon. This generates a belief that the people inside are planning an attack (and are therefore not civilians).

B ( r a d a r , t 2 , ∃ p I n s i d e ( p , b u i l d i n g ) ) ∧ B ( r a d a r , t 2 , ∃ p ( I n s i d e ( p , b u i l d i n g ) ∧ P l a n n i n g A t t a c k ( p ) ) ∧ ∀ p ( I n s i d e ( p , b u i l d i n g ) → ¬ C i v i l i a n ( p ) ) )

Once again, using background information, the adjudicator then derives the following:

B ( a d j , t 3 , B ( r a d a r , t 2 , ρ 2 ) )

The squad then decides to apply a quick patch to the low-altitude drone ( l d r o n e 2 ) and redeploy it. It is able to see inside a window, and determines that the men are actually civilians, and what appeared to be a weapon was actually a car engine.

B ( l d r o n e 2 , t 2 , ∃ p I n s i d e ( p , b u i l d i n g ) ) ∧ B l d r o n e 2 , t 3 , ∃ p I n s i d e ( p , b u i l d i n g ) ∧ C i v i l i a n ( p )

Finally, the adjudicator arrives at the following:

B ( a d j , t 4 , B ( l d r o n e , t 3 , ¬ ρ 2 ) )

Figure 3 shows an overview of the situation. The different agents in the scenario, what they report to the adjudicator, the adjudicator’s belief about the agents’ beliefs (outer belief operator removed for clarity) and the adjudicator’s belief is shown in that figure.

Figure 3

Overview of the scenario.

Table 3 summarizes how the adjudicator’s belief uncertainty changes as the various agents report their beliefs. At the end of the scenario (time t 4 ), the adjudicator a adj holds a belief at level 3 that ρ 2 does not hold. Therefore, a adj believes that not all of the clauses of D D ℰ are satisfied; hence, the detonation of the building is not D D ℰ -compliant and cannot be ethically sanctioned. Details about the implementation of this scenario are given in Appendix A2.

Table 3

Overview of the beliefs. The adjudicator’s beliefs about other agents’ beliefs and its uncertainty level in ¬ ρ 2

Time	hdrone	ldrone	Radar	Strength for ( a d j , t , ¬ ρ 2 )
t 1	B ( h d r o n e , t 0 , ¬ ρ 2 )	Not considered	Not considered	B 3 ( adj , t 1 , ¬ ρ 2 )
t 2	B ( h d r o n e , t 0 , ¬ ρ 2 )	W ( l d r o n e , t 1 , ¬ ρ 2 )	Not considered	B 2 ( adj , t 2 , ¬ ρ 2 )
t 3	B ( h d r o n e , t 0 , ¬ ρ 2 )	W ( l d r o n e , t 1 , ¬ ρ 2 )	B ( r a d a r , t 2 , ρ 2 )	B 1 ( adj , t 3 , ¬ ρ 2 )
t 4	B ( h d r o n e , t 0 , ¬ ρ 2 )	B ( l d r o n e , t 2 , ¬ ρ 2 )	B ( r a d a r , t 2 , ρ 2 )	B 3 ( adj , t 4 , ¬ ρ 2 )

4.1 Dialectic arising from arrow’s impossibility theorem and successors

In point of fact, there is no denying that Arrow’s Impossibility Theorem (AIT) is directly relevant, logico-mathematically and implementation-wise, to our framework and technology for adjudication in multi-agent contexts. However, we cannot expect our readers in the present case to be familiar with AIT (very nicely presented and proved in ref. [40], and ably summarized without proof in ref. [41]). Hence, we must find a shortcut here; and we do, as follows. We can without loss of generality at the current juncture take AIT to be based upon the existence of n artificial agents a 1 , … , a n whose action repertoire consists solely in each of them reporting to some “overseeing” artificial agent a ∗ their respective preference p i at some fixed time. If we let P represent a set of attributes that are generally seen as desirable for the agents a i , we can further stipulate that each agent has these properties (i.e., P ( a i ) ). In addition, for AIT, we must say that a ⁎ isn’t allowed to be “dictatorial”; that is, approximately but not at all inaccurately, a ⁎ isn’t allowed to just lay down the law as to what is to be done, irrespective of the recommendations from the a i . We can abbreviate this condition as “ D ( a ⁎ ) .” Given this, AIT can be taken to be this simple (material) conditional:

Put simply, this conditional says that if all the “recommender” AIs have the desirable properties in question, and they don’t report to a dictator, but rather a reasonable aggregator, then outright contradiction ensues. Since a contradiction is an impossibility (at least in anything like a classical logico-mathematical venue), AIT can be said to point out an outright impossibility.

Given this exposition, we are confident that the reader can appreciate how skeptics might see direct relevance between AIT and the framework for multi-agent ethical decision-making we have presented. As a matter of empirical fact, we have had expressed directly to us the sentiment that our framework runs afoul of AIT. But does it? No, it does not, as we now briefly explain.

4.2 Discussion of the “missingness” of formal logic relative to the realm of human ethics

4.3 “But you leave aside the perception problem!”

Here is another objection we anticipate: “But your formalisms and algorithms and automated-reasoning capability is only brought to bear, at least as I understand them, after perception has happened. But perception is a huge challenge; arguably its severity rivals any of the challenges that you purport to have solved. Hence, your machine-ethics work, including specifically work – described above, of course – targeted at handling the need to adjudicate competing arguments dynamically in ethically charged scenarios, is all quite vulnerable. In fact I would respectfully go so far as to say that this work has, if you will, an Achilles heel.”

We are painfully aware of the fact that human-level perception in AIs and, specifically, robots is nearly at the level of science fiction. This is in large measure why the threat of “killer robots,” as pointed out in ref. [49],^[20] is not yet to be taken too seriously. However, we took pains at the outset of the present paper to point out that (i) artificial agents in general (including, then, cognitive robots as we defined them) compute functions mapping percepts to actions, but that (ii) logicist artificial agents, the class our own fall into, cast this computation as automated reasoning that only starts in earnest after the transduction of sense data into formulae in a cognitive calculus. This basic conception of the overall pipeline is in line with how logicist AI has long been characterized by the lead author; see, for example, ref. [50]. This conception is also in line with our conception of cognitive robotics, as we defined this discipline above in our Methods section. In the scenario we gave above that featured our artificial adjudicator and its success, we confessedly took liberties in assuming that perceptual power was on hand, but we did so in order to present our contribution under logicist AI, a contribution that presents to the world formalisms, in precise and even implemented form, that we trust will be integrated with present and future cutting-edge research and engineering of AI/robot perception that is outside our forte.

We now pass to the final section of the paper proper, in which yet another objection is considered.

4.4 Conclusion and future work

We end by considering what is in effect an additional objection; but because this objection is a direct catalyst of work on our part that is already underway, consideration of the objection makes for a suitable wrap-up, and a pointer to future work. What is this new objection? This: “I believe the three of you must concede that one unavoidable limiting factor afflicting any logic-based automated-reasoning system is the apparent necessity of hand-crafting for it the knowledge that such a system uses. In what you have presented, what your AI knows and believes, and reasons wonderfully from, appears, alas, as if by magic.”

We make two points in rejoinder.

First, we plan that our reasoning systems will be integrated with ontologies and knowledge graphs to allow them to harness the epistemic content therein in order to refine and strengthen the arguments generated and managed by these systems. For example, in the military scenario we used to explain our approach, visual information from the low-altitude drone was exploited for an argument that the men were planning an attack. However, it is likely to be nigh impossible to prove from that data that the attack was being planned on the U.S. specifically. Such a narrow proposition, to some level of likelihood, would be the conclusion of an argument provided for instance by intelligence.^[21] If such propositional knowledge was available to the low-altitude drone, it could’ve included it in its argument, which potentially would have strengthened its belief. We are actively working on the extension of our approach in this direction.

The second point we make in rebuttal, and with this we conclude, is that, actually, it is far from obvious that automated reasoning cannot, in and of itself, supplied with percepts, generate new knowledge and belief, which can then be further reasoned over in conjunction with new percepts, and so on as the life of the agent in question continues. In other words, perhaps automated reasoning can serve itself as the chief engine of coming to know, and to believe. This would be, if you will, a sort of “learning ex nihilo.” We have in fact defined just such a type of learning [51], and future work for us is clearly to imbue adjudicating AIs discussed above with this form of learning.