Recognition of arm and body postures as social cues for proactive HRI

2.1 Recognition of body posture

Modern approaches for posture recognition often involve video stream analysis or tracking of body joint locations. Most of the literature focuses on motion capture of body parts during posture recognition. The other methods involve pose estimation using depth images. Recognition of postures through depth video streaming has also received recent attention. But such methods require a lot of pre-processing. Thus, this has become a problem to overcome in real time posture recognition applications.

In ref. [10], authors presented an approach using top view image of the human in order to avoid self-occlusion. Postures with similar top view, but slightly deviated from each other in the arrangement of body joints, are difficult to be identified in this manner. Furthermore, a top view camera is inapplicable for a moving robot in a domestic environment. Cucchiara et al. [11] used posture classification with the help of multiple cameras based on projection histograms implemented using Hidden Markov Model (HMM). Postures that are slightly deviated from each other but fall under a common major posture group (e.g., standing straight, standing while torso bent forward, standing while leaning to a wall, etc.) are difficult to be differentiated during this approach.

Pisharady and Saerbeck [12] have implemented a posture recognition algorithm based on geometric features of the body. In some postures, independent position of body parts as seen by a coordinate frame outside the body is required to identify a posture with a satisfactory accuracy. Buccolieri et al. [13] have used a recognition technique with active contours for moving action detection with HMM in fall detection. This requires intersection of inputs from all the cameras. Both the methods mentioned above require a large number of features for accurate functioning. A similar approach is explained in ref. [14], and this method analyzes an ellipse drawn around the human shape. Deviations in the ellipse are used to identify a certain posture. Identification of intermediate postures with similar geometry in the human shape is difficult with these systems. Visual surveillance of shapes is also a projection of this concept [15]. However, optimal matching of a silhouette will meet some difficulties when complex postures are considered. In such occasions, similarities between shapes will sometimes be misleading. A comparison of conventional 2D approach and modern 3D models for posture recognition has been presented in ref. [8]. The 2D model depends on the camera point of view. This gives erroneous results when the camera is in motion. Similar methods which use 3D models recognize only standard postures with high accuracy. Therefore, additional measurements have to be taken to identify intermediate postures accurately.

In summary, there are several body postures that cannot be recognized with a satisfactory accuracy. Hence, a requirement of approaches, which are able to recognize a higher number of postures, emerges.

2.2 Recognition of arm posture

We use gestures frequently to communicate nonverbally or enhance what is communicated verbally [16]. Moreover, human gestures can be considered a resemblance of actions where most of the gestures in nonhuman primates lack the representational elements to perceive gestures [17]. Perceiving a gesture is important in exploring emotional aspects in an encounter as well as a physical activity of an individual [18].

Most of the existing literature to determine the hand pose related to fingers and griping as in refs [19,20,21] was unable to identify the overall arrangement of the hand including upper and forearms. A hand posture recognition system which used discrete Adaboost learning algorithm with Lowe’s scale invariant feature transform (SIFT) features has been proposed in ref. [22]. Here, the authors have taken an effort to reduce the issues posed by background noise in existing approaches.

In ref. [23], an effort was taken to anticipate future activities of a human to generate reactive responses from a robot. Spatio-temporal behavior of selected subjects has been observed by a robot over time to anticipate future trends in human and object behavior. Each possible tendency has been predicted using an anticipatory temporal conditional random field (ATCRF) which simulated rich spatial-temporal relations through object affordances. Similar approaches can be utilized to identify spatial behavior of certain body parts such as hands which form gestures.

According to the literature, color cues contribute mostly to recognize hand pose. But the spatial orientation of limbs seems to be helpful in determining the overall body language of hands. This fact was suggested by ref. [24]. The study conducted in ref. [25] is an example for the effectiveness of perceiving nonverbal behaviors over voice during human–robot encounters in hospitals. In the work explained in ref. [26], the robot understands the meaning of human upper body gestures and expresses itself by using a combination of body movements, facial expressions, and verbal language. Twelve upper body gestures including four gestures with the association of objects have been considered here. One drawback of the approach is the deployment of wearable sensors to recognize gestures and people in domestic environment will not prefer being disturbed by this equipment for a long time. But here, the robot’s reaction toward a user was generated based on the emotions associated with the upper body gestures of its user. A robot teacher which could perceive affect of students and evaluate how that influences the engagement in a classroom is presented in ref. [27]. Affect was mostly deduced from four different facial expressions: “confused,” “distracted,” “engaged,” and “thinking,” and each expression was assigned with a positive/negative/neutral mood. This was combined with behavior recognition as well. For instance, bodily gestures such as “lower head,” “raise hand,” “sit still,” and “support chin” have been utilized to “map” people. People were mapped for their overall “state” of affect to be “confused,” “distracted,” “engaged,” and “thinking” in the end. This approach testifies to the fact that gestures contribute considerably to the internal state of a person, although a limited number of gestures have been considered for the study. Furthermore, this work is an example where a combination of cues interprets emotional state of people better than a single cue alone.

Most of the gestures involving hands were detected with a considerable accuracy using body worn sensors as in ref. [28]. But humans prefer other methods as they will find it cumbersome to wear sensors in daily encounters. In ref. [29], authors used HAAR-like features which associate digital image features similar to Haar wavelets. This method requires a relatively smaller set of training data compared to other similar methods. Even so, context-free grammar is required to reach a high level of accuracy to analyze the syntactic structure. Most of the probabilistic approaches have shown proven accuracy in gesture recognition [30]. But these involve complex algorithms which slow down the decisioning process which has to be made in real time, consuming more computational power. The spatial behavior of joints in the human hand has been monitored over time to identify friendly gestures in ref. [31]. However, this method requires monitoring several body joints for a minimum duration of time to confirm a gesture. Hence, much simpler mechanisms to recognize arm posture for socially interactive applications are required for present robotic systems.

Still there are several arm postures that could not be recognized with a satisfactory accuracy. Hence, a requirement of approaches that are able to recognize a higher number of arm postures emerges.

2.3 Nonverbal interaction initiation

An assistive robot is expected to possess intelligence to determine when to interact with the user and when not to. This natural human-like behavior enhances the relationship between robot and its nonexpert user. Therefore, the robot must replicate complex human behavior in decision making [32]. However, the focus of these approaches is to maintain the interaction by analyzing user feedback rather than initiating the interaction. Furthermore, postures are beneficial for a robot in assessing interaction demanding of a user prior to a conversation. Even so postures are excluded as a factor to determine user situation in most approaches.

Work explained in ref. [33] states an affect recognition technique based on body language and vocal intonation. This approach has been used to interpret human affective cues and respond appropriately via display of their own emotional behavior. But it conducts diet and fitness counseling during HRI, not an HRI scenario in general. Somehow this can be considered a mechanism where the robot could accurately guess the emotions associated with a situation. Mechanisms used in refs. [34,35] have evaluated emotions, gestures, pose, and movements of the human in his environment while interaction to measure interactive behavior. In ref. [36], nonverbal user engagement has been measured using initiator and responder gaze times, face orientation and feedback times for the two speakers. Bodily posture which is a critical parameter before and during an interaction is omitted in this method. In ref. [37], user’s pose and movements have been taken into account before interaction initiation. But this system lacks any input about the changes in user’s attention toward robot’s presence. Therefore, there is an emerging requirement for modern cognitive systems to perceive human situation prior to an interaction. In addition, although torso and lower body are good displays of internal state of humans, literature related to the assessment of these are scarce. Analysis of arm and body postures can be an excellent mediator to analyze human state in designing cognitive social robots.

The study conducted in ref. [25] is an example for the effectiveness of perceiving nonverbal behaviors over voice during human–robot encounters in hospitals. In the work explained in ref. [26], the robot understands the meaning of human upper body gestures and expresses itself by using a combination of body movements, facial expressions, and verbal language. Twelve upper body gestures including four gestures with the association of objects have been considered here. One drawback of the approach is the deployment of wearable sensors to recognize gestures and people in domestic environment will not prefer being disturbed by this equipment for a long time. But here, the robot’s reactions toward a user was generated based on the emotions associated with the upper body gestures of its user. A robot teacher which could perceive students’ affect and evaluate how it affects the engagement in a classroom is presented in ref. [27]. Affect was mostly deduced from four different facial expressions: “confused,” “distracted,” “engaged,” and “thinking,” and each expression was assigned with a positive/negative/neutral mood. This was combined with behavior recognition as well. For instance, bodily gestures such as “lower head,” “raise hand,” “sit still,” and “support chin” have been utilized to “map” people. People were mapped for their overall “state” of affect to be “confused,” “distracted,” “engaged,” and “thinking” in the end. This approach testifies to the fact that gestures contribute considerably to the internal state of a person, although a limited number of gestures have been considered for the study. Furthermore, this work is an example where a combination of cues interprets emotional state of people better than a single cue alone.

2.4 Problem statement and the proposed mechanism

Techniques for arm and body posture recognition which could recognize an adequate number of postures with a higher accuracy are scarce. Furthermore, the aforementioned systems evaluate a number of parameters for evaluating the possibility of an interaction. Even so the possibility of integrating posture and gaze factors in monitoring user behavior is not adequately considered within these works. Therefore, this article presents a model to interpret user situation before interaction by analyzing postural behavior and this behavior is recognized by means of arm and body-based geometry derived through postures encountered during each scenario.

The proposed system adopts a novel geometry-based approach for recognizing a set of human arm and body postures. The standard body postures considered here are sitting, standing, and bending, which are most frequently adopted by humans in domestic environments. In addition to that frequent relaxing postures are considered as well. The same approach was applied to recognize often encountered arm postures as well. The system can also be utilized in self-employee environments where the same posture prevails for extended periods. In addition to that, postural behavior of the user is monitored by the system to identify user’s interest toward interaction by evaluating his/her body language. This approach enables the robot make intuitive decisions during an encounter rather than adopting a behavior only based on a predefined set of actions.

This mechanism uses body joints and spatial arrangement of limbs to establish patterns in their arrangement. Spatial arrangement or the geometry of selected limbs with respect to each other is analyzed mathematically to differentiate various postures considered. This system is implemented on various activities which involve a number of postures to assess its functionality. A vision-based technique is used for posture identification in the system. Image processing techniques were equipped with extraction of feature points in 3D space using color and depth information. Spatial orientation of postures is mapped to the corresponding posture using a multi-layered feed forward neural network. Arm and body postures observed throughout a period were used to assess the emotional state of a certain encounter. This evaluation determines the appropriate behaviors for the assistive robot. We considered the nature of conversation and approach distance as the “approach behaviors” of the robot.

This article has two contributions.

1. A novel method to recognize a number of arm and body postures.

2. Evaluation of the effect of considering arm and body postures to generate human-aware responses by a robot prior to an interaction. In this context, the robot’s responses are related to its approach behavior which includes mutual distancing and the type of conversation to initiate with its user.

4.1 Most honest part of our body

People use their faces to bluff or conceal true sentiments and truthfulness reduces from feet to head [6]. Lower body must receive a significant level of attention when collecting nonverbal cues. For instance, we cross our legs when we feel comfortable. Then we uncross our legs in the presence of someone (e.g., a stranger or an official). When the whole body including legs is considered, individuals lean forward when they are thinking of something and they lean backward when relaxed. In addition, the degree to which we move arms is a considerable and accurate indicator of an individual’s attitudes and sentiments. Normally, arms move effortlessly when they interact. In contrast, when upset or exited we restrict arm movements. Gravity-defying arm movements are often generated from joy and excitement. Hence, arms are plausible cues to access for moods and feelings. Certain arm behaviors gesture attention and some gesture denial. Therefore, in this research, we consider various arm postures to be perceived by a robot for their emotional interpretations as well.

4.2 Changes in behavior

Changes in behavior can help reveal how a person is processing information or adapting to emotional circumstances. Changes in behavior can further reveal a person’s interest in an event. Careful observation of such changes can allow a robot predict the intensions upon an encounter in order to align robot’s behavior accordingly. Hence, the robot’s actions would not cause any intrusion or disturbance to the users. Knowing to distinguish between comfort (e.g., content, relaxation, happiness, excitement) and discomfort (e.g., displeasure, unhappiness, anxiety, stress) will help a robot decode user behavior and generate a response that will comfort that person. Learning to read cues of comfort and discomfort will help a robot decipher what their mind and the body are saying. For instance, people tend to keep their arms on hip, when there are issues or to show dominance.

To decode user behavior, the robot will observe them at a distance in our approach. Hence, neither the observation nor the interaction will be intrusive. For example, a person may distance herself from someone by leaning away. All such behaviors are controlled by the brain [39] and indicates whether a person is interested in or getting ready to avoid an encounter. Another similar example is that a person may rub his face or cheeks when in need of getting rid of a situation. Therefore, hands and the body are important to be considered in reading body language. In this research, we consider a number of arm and body postures to be distinguished by a robot before initiating an interaction with a human. A set of often encountered body and arm postures are shown in Figures 2 and 3, which we tried to recognize using the proposed concept. Hence, a robot may be able to interact with a human without territorial violations and disturbing behaviors. In addition, there are pacifying behaviors such as soothing necks or stroking necks. When we are in discomfort, our hands involuntarily respond to make us comfortable again. In addition, touching face, head, neck, and arms can originate in response to a negative stimulus such as stress.

Figure 2

A set of body postures encountered in daily chorus are shown. Each posture is called by an abbreviated form of their actual name and their actual names are explained in Section 4.2.

Set of body postures shown in Figure 2 identified by the proposed approach are abbreviated for the ease of referencing as follows.

Figure 3

A set of arm postures encountered in daily chorus are shown. Each posture is called by an abbreviated form of their actual name and their actual names are explained in Section 4.2.

Similarly, the set of arm postures shown in Figure 3 identified by the proposed approach are abbreviated as follows.

5.1 Mechanism used to differentiate postures

To detect a posture uniquely from a set of postures, orientation of vectors drawn along selected limbs in human body is considered. In other words, each limb that makes a posture, becomes a vector. A vector is formed by combining two adjacent joints in the skeleton. Once a person is tracked, his/her skeletal arrangement is extracted to analyze the positioning of a selected set of vectors in 3D space. This is demonstrated in Figures 4 and 5. Figure 4 shows two vectors: forearm vector and upper arm vector, which are used to determine arm posture. To determine the spatial orientation of the body, three major limbs, spine, femur bone, and knee, are considered. A vector is drawn across the limb, joining the end joints of that limb, according to the extracted skeleton. The angles this vector make with the X Z plane (horizontal) and Y are also measured. Y direction is marked along the vertical of the camera of the robot, as marked in Figures 4 and 6. Spine, femur, and tibia vectors shown in Figure 5 are the vectors used to determine the body posture. Front view of a skeletal while sitting is shown in Figure 5.

Figure 4

Body angles derived from the limbic vectors of a person with stretched hands are shown. Coordinate system as seen from the camera is shown as X-Y-Z. The angles formed by the three limbic vectors with the X Z -plane are marked as α and β , while the angles made with the X Y -plane are marked as θ and δ . These angles are used to recognize the arm posture. A, B, and C denote shoulder, elbow, and wrist joints.

Figure 5

Limbic vectors of a seated person are shown here. The limbic vectors are derived by joining A (spine-mid), B (spine-base), C (knee), and D (ankle) joints.

Figure 6

Body angles derived from limb vectors of a seated person as in Figure 5 are shown here. Coordinate system as seen from the camera is shown on bottom left. The angles formed by the three limb vectors with the xz plane are marked as α 1 , β 1 , α 2 , β 2 . (a) Orientation of the upperbody vectors is shown by the triangle ABC, (b) orientation of the lowerbody vectors is shown by the triangle BCD.

6.1 Posture change over a timeline

During certain activities, humans change their posture at least once. Therefore, posture changes must be analyzed for an adequate duration. The system observes the user for T seconds from the moment a user’s skeletal was identified. DR records raw information such as coordinate positions of specific joints and calculated parameters such as joint vectors throughout this period. At the end of T s, set of information recorded in DR is analyzed for posture identification and decision evaluation. An example of this evaluation for the period of observation equal to T s is shown in Table 1.

In Table 1, arm and body postures perceived by the model in each second throughout the period of observation are shown. This postural behavior is taken into consideration prior to the interaction. Since different postures have different emotions associated with them, an evaluation of postural behavior makes more sense than considering the postures observed in the beginning of a scenario. Hence, the decision making is affected by this “postural behavior” of the user. The “initial posture” was recorded at t = 1 s and the “final posture” was recorded at t = T s . A separate experiment was conducted to find the optimum value for the period of observation during this type of occasions. This is elaborated in Section 8.5.1.

6.2 Availability of the user

The intention of deploying an assistive robot is to uplift the mental condition of its user, through friendly interaction at user’s leisure or lonely hours. Therefore, the robot’s interaction approach must consider appropriate situations to achieve this outcome. IDMM assesses the availability of the user by analyzing the change of his/her behavior by means of posture over time. The idea behind this evaluation is as follows.

According to behavioral sciences, there is a trend in humans to change their behavior when somebody is around. Therefore, the user’s response toward the robot changes, when he/she sees the robot nearby. A common example for this type of situation is that humans slant their spine (SEAB in Figure 2) when seated for a long duration, but when someone comes near, they adjust their postures to standard sitting posture (SEA in Figure 2) with erect spine. Humans notice such changes in behavior in someone when they intend to interact with that person. Instead of the human, it is the assistive robot who assesses such incidents in domestic environment in this context. This is the motivation behind the use of postural behavior instead of only posture in this approach.

7.1 Valence of a situation

Once the postures observed in a single scenario are known, a variable called “valence” is calculated to assess the “internal state” of that person quantitatively. To calculate valence, we categorized postures into three groups namely, “relaxing poses,” “neutral poses,” and “stressed poses.” This is shown in Figure 7. Postures adopted during relaxing situations were categorized as “relaxing poses,” and are assigned a positive valence. Similarly, postures adopted during stressed situations were categorized as “stressed poses” and are assigned a negative valence. Standard postures were assigned the value zero for valence. This categorization was done after analyzing some simulated domestic encounters according to the generally accepted social norms. Assignment of a valence for each posture was influenced by the study shown in ref. [40]. Valence for each posture was determined after this analysis and the values assigned for the valence are given in Table 2. Assigning valence and categorizing postures according to their emotional state were empirical and based on how humans react in social encounters in general. Hence, we simulated such encounters and arranged arm and body postures in an order of their positive emotional state. Values for the valence were assigned after this step.

Figure 7

A semantic map illustrating the components that make valence: hand and body poses. Each pose is categorized into three groups: relaxing, neutral, and stressed. This categorization is based on the emotions associated with each pose in general human perception.

In a single scenario, a person adopts both body posture and hand posture. In such a case, the valence of the scenario was calculated as follows.

Valence Body Posture and Valence Hand Posture indicate the valence corresponding to body posture and hand posture as given in Table 2. k 1 and k 2 indicate the weights given to body posture and arm posture, respectively, in determining the valence of a situation.

Interaction decisions made by the robot based on the calculated valence are shown in Figure 8. The values to separate margins between each type of conversation and mutual distancing were empirical and were decided by repeating a few trial experiments before implementing the IDMM. Corresponding type of conversation and the mutual distancing appropriate for a particular instance were determined according to the value of the valence as shown in Figure 8. The margins for each approach behavior are set by simulating a number of occasions in domestic encounters and human responses if a human approaches another human instead of the robot. This simulation helped in determining the values that were used for the valence in each arm and body posture shown in Table 2. Postures were listed in order of their relaxing or stressed nature as per the human perception and then a numerical value for the valence is assigned.

Figure 8

Marginal values for valence to determine appropriate approach behavior of the robot. (a) Initial valence is marked as 0 and this corresponds to “Greeting” at a distance of 1.8 m as the appropriate approach behavior. (b) Final valence increased to +8 which resulted in an approach behavior of “asking for a service” and keeping a mutual distance of 1.5 m.

7.2 Decision-making in dynamic user behavior

In a dynamic user behavior, several changes in body and arm postures could be observed. In such occasions, the initial and final postures observed during the period of observation, T were considered in decision-making. If the difference between the two valences: initial valence and final valence, is an increment, the approach behavior is promoted by one set. The conversation types used in this work are listed as follows:

• Greeting

• Offering a service

• Small talk

• Long conversation

• No interaction

For instance, if the valence was increased during a certain occasion and the approach behavior corresponding to the final valence is “Greeting” and 1.5 m, this is promoted to “Offering a service” and 1.8 m. For this promotion, we referred to the chart shown in Figure 8. A combination of a mutual distance and the corresponding conversation is referred to as a “set.” Similarly, if the difference between the initial and final valences is a decrement, the approach behavior is demoted by one set. For instance, if the final valence corresponds to { Greeting, 1.5 m } , this is demoted to { No Interaction, 2 m } . This decision making criterion is given in the following algorithm.

8.1 Determination of T (Experiment 01)

Since a period of observation was necessary for the robot to continue with the interaction decisions, the first experiment was conducted to determine T in the first place. Therefore, a set of domestic activities were selected and a person was allowed to engage in a particular activity for a period of time. During that period, posture changes and the time gap between two consecutive different postures was recorded. For the experiment, 11 users aged from 26–59 years (SD: 12.83 and mean: 31.63) participated. The participants were given a task to perform, e.g., standing, without further instructions on how to perform the activity. This was because we tried not to influence people’s postures by instructing on how to perform a task. We observed the postures that they performed and therefore we could identify variations in posture while performing the same activity. This step was taken to retain the natural behavior of people although we conducted it in a simulated environment. The participants were given the chance to stop the activity whenever they feel like it. Details of the activities selected are given in Table 3. Each participant was asked to perform at least three tasks which resulted in 33 activities observed altogether. Average values for the variables observed during each activity are given. In Table 3, “minimum and maximum times between two consecutive postures” refer to the average time observed between two postures, if posture changes were observed during the period. Here, a single user was observed for 30 min unless the user walked away in the middle of observation. To determine T , we observed for only body postures because the hands are dynamic and change faster than the body in general. To determine T , postures were recognized using the mechanism explained in Section 5.

8.2 Recognition of arm and body postures (Experiment 02)

A feed forward neural network was trained to map the set of angles made by the limbic vectors to the corresponding arm posture. We used a feed forward neural network of three layers and 50 neurons in the first and second layers. Training data set consisted of posture information of 21 human subjects (SD: 11.25 and mean: 29.1). We tried several other hyper parameters such as more and lesser neurons and hidden layers as well. But in the end, we could observe an optimum behavior of the algorithm for recognizing these postures at the values given here. The inputs and the output of this network can be stated as follows:

Inputs: A ˆ , B ˆ , C ˆ , δ , θ , α , β

These are the inputs shown by the set H i in (3). We exempted the two angles A C → makes with the vertical and the horizontal, from the current set of inputs as we did not observe an improvement in the training and testing accuracies of the dataset with the additional data. This holds true for the inputs selected for the recognition of body postures as well. Therefore, we observed the current set of variables as the optimum set of inputs in our approach.

Output: Corresponding hand posture out of { DWN, FRO, UP, FOR, SID, WAI, CHK, DES, WAV, HEA }

The module called Arm Posture Identifier performs the aforementioned functions.

Another feed forward neural network was used to map the set of limbic vectors to the corresponding body posture. This feed forward neural network also consisted of three layers and 50 neurons in the first and second layers. Data related to body posture were also taken from the same subjects as for the hand posture.

Inputs: A ˆ , B ˆ , C ˆ , α 1 , β 1 , D ˆ , E ˆ , F ˆ , α 2 , β 2

These are the inputs shown by the set P i in (7).

Output: Corresponding body posture out of { STA, STDUP, LEA, PIK, CRO, SEA, SEAF, SEAB, MED, SEAUP, BEN }

Different seating arrangements make a body posture slightly different from each other. Furthermore, people tend to keep their legs at different angles when seated. Figure 9 shows three examples for this. Such instances were also considered while collecting data for the training data set.

Figure 9

A comparison of the limbic arrangement for regular sitting posture (SEA) when the seated chair is changed. This occurs due to the difference in dimensions of each chair type. (a), (b), and (c) represent the user posture, the corresponding skeletal image, and two angles β 1 and β 2 for side chair, stool and arm chair, respectively.

The module called Body Posture Identifier performs the above functions. In the end, we used this trained algorithm to recognize posture changes in Experiment 01.

8.3 Implementation of IDMM (Experiment 03)

After implementing the recognition algorithm and determining T , we used these findings to determine the internal state of the people before an interaction. Throughout the article, we refer to robot-initiated interactions in this regard. We instructed the participants to perform a selected set of activities listed under “description” in Table 4. IDMM was implemented and the interactivity of the situation was evaluated at this stage while the participants were performing these activities. The same set of participants in Experiment 02 participated in Experiment 03 as well. We asked each participant to perform at least two activities and altogether we considered 40 different scenarios for Experiment 03. Twenty scenarios out of these are shown in Table 4. All the participants were selected outside the university and over 80% of the participants had no technical background.

Before implementing IDMM, the robot was remotely navigated toward the participant adopting the approach behavior as “Offering a service” and 1.5 m. User was asked to give a feedback score for the robot’s behavior by considering user satisfaction. This is used as the ground truth for this experiment. Then the entire scenario was performed again and the valence of the set of postures was calculated for each scenario according to (8). Then the robot was commanded by a teleoperator to adopt the approach behavior determined by the IDMM. IDMM decided upon an approach behavior by evaluating the valence. For the experiments, we assigned 1 for k 1 and k 2 in (8), assuming that the arm and body postures contribute equally for the current user situation. Then the participant was asked to give a feedback score (rating from 0 to 10) for the approach behaviors of the robot during the two occasions. Feedback score was given by users based on how they preferred or how comfortable were the decisions of the robot during that particular situation. Users were asked the reason to give their corresponding feedback score and important remarks during reasoning are highlighted in the discussion.

8.4 Research platform

Proposed concept has been implemented on MIRob platform attached with a Microsoft Kinect sensor. This is a Pioneer 3DX MobileRobots platform equipped with a Cyton Gamma 300 manipulator. The robot required an initial map of the environment for navigation and path planning. Therefore, we initially defined a map of the simulated environment with the placement of furniture, doors, and walls. Dynamic obstacle avoidance was possible with the MobileRobots platform. Navigation maps required for locomotion were created with Mapper3 Basic software. Skeletal representation of human body was extracted as 3D co-ordinates of feature points in Kinect sensor. The experiment was conducted in an artificially created domestic environment with the participation of users in a broad age gap. MIRob is shown in Figure 10.

Figure 10

Two occasions during Experiment 03 is shown. (a) Occasion 1: Responses of the user when the approach behavior: “greeting at 1.8 m” was adopted by the robot. (b) Occasion 2: Robot observes the user at a distance. (c) Final position of the robot and the behavior of the user. Recognized postures and mutual distances are marked.

8.5 Results and discussion

8.6 Implications derived from the set of experiments

Implications derived from the set of experiments can be stated as follows. These implications could be utilized in the development of conceptual basis for response generation and the design of future social robots.

It can be seen from the experiments that an individual changes his/her behavior every once in a while. Therefore, it is important for robots to observe their users for at least a short duration before initiating an interaction with them. This can be proposed as the first design guideline derived from our experiments.

The implementation of IDMM received a higher feedback score when compared with a nonadaptive approach behavior used as the ground truth. Hence, considering the variables in the context such as postures could be advantageous for a robot in order to receive a higher user acceptance. Therefore, considering the factors within humans before making decisions can be stated as the second design guideline for intelligent robot assistants. Even though we considered only arm and body posture as a signal of internal state of human mind, there can be other factors that affect the internal state of mind. Therefore, a conceptual basis must be laid to identify such factors and use them for AI (Artificial Intelligence) systems in a theoretical manner. This can be stated as the third design guideline derived from these experiments.

There are some situations at which humans nonverbally seek attention from outside for various reasons. Therefore, unveiling the meanings of nonverbal behavior is an important aspect in present HRI. This can be considered the fourth design guideline proposed by these sets of experiments.

It can be seen that users preferred different approach behaviors at different occasions. The difference between the user feedback scores received for a user-aware approach behavior and static approach behavior testifies to this. Therefore, generating adaptive approach behavior which matches with the context can be proposed as the fifth design guideline derived from our experiments.