Abstract
Robots have been used for many years in therapeutic activities with people with Autism Spectrum Disorder. However, most robots presented in the literature have limited or no mobility, are made of rigid materials, or are too expensive for many care centers. We share the choices and the design rationale of the latest version of a soft, mobile, low-cost, autonomous robot that has successfully been used for 3 years in a care center for activities that include both free play and structured games. Moreover, the kind of activities that can be performed with this robot, and the feedback obtained from therapists about its application are reported.
1 Introduction
The aim of this article is to share experiences and qualitative insights obtained from the design and application of a robot, TeoG, designed for people with Neuro-Developmental Disorders (NDD). The article presents the design choices, introduces the activities that can be implemented with these choices, and discusses the experience acquired during more than 3 years of using the robot in a care center.
A few years ago, we developed Teo, a huggable, mobile, autonomous robot having a size comparable with that of the intended users and the ability to move on the floor [1,2]. Teo was designed in order to explore the opportunities that a robot with such characteristics could offer for therapy for children with NDD, such as Autism Spectrum Disorder (ASD). No robots with all its characteristics were reported in the literature. In particular, the focus was on two categories of activities: structured games and free play. Structured games were required by therapists to augment with the robot the interaction with the children, where question–answer sessions were mainly used to develop associations. The mobile robot was intended to attract the subjects and to keep them involved in the activity. An expected effect was the development of attention. Free play was an innovative activity possibility enabled by a robot offering different interaction channels and with some autonomy. The aim was to provide the possibility of free play [3] with the expectation of development of motion and cognitive competences. However free play was only implemented through an interface with a joystick used to move the robot by an assistant or the therapist. Teo was designed by a team of more than 20 engineers, designers, therapists, psychologists, and caregivers in a user-centered design setting.
Since then, the robot has evolved to match new requirements from caregivers, therapists, and actual users as well as to provide a wider set of integrated features. In particular, the third iteration, TeoG, was implemented for the care center “Il Sogno,” based in Castelnuovo di Garfagnana, Lucca, Italy. TeoG provides a wide range of functions, shows a good degree of autonomy, and supports multimodal interaction. Moreover, an application for mobile phones and touch pads makes it user-friendly for the caregivers. Autonomy is important as it does not require the therapists to always have full control of the robot, mostly in free play and for emotional feedback, although they always have the possibility to take control of the robot through the touch pad interface. A full description of the robot is provided in Section 2.1.
TeoG has been operating since 2018 and has been used in support of activities with children with NDD, exploiting both free play and structured games, as described in Section 2.2. From the experience with the robot during the first 15 months, some issues were pointed out by the therapists and then used as input to implement an improved version, in operation for more than 3 years, at the time of writing. The experiences with the two versions and the implemented improvements are reported in Section 2.3.
1.1 Background
A lot of articles about the therapy for subjects with NDD supported by robots have been published (e.g., [4,5,6,7,8]). Robots may introduce many positive aspects in these therapies. When they are present in a therapeutic setting, they are perceived as new objects, possibly triggering interest and curiosity. They have predictable reactions to actions performed by subjects, thereby enabling them to feel in control of the situation. They can be controlled by the therapists, thereby giving them the possibility to work with the subject through a physical, nonhuman avatar. Many robots can also record data that can be used for objective evaluation. We henceforth mention only the robots having similar or opposite characteristics with respect to Teo. Given the wide number of impairments included in NDD, a lot of different activities have been devised for robots. Many of them have been used to stimulate attention (e.g., [9]) or gesture replication (e.g., [10]). In most of these cases, the robot is expensive and has a rigid, humanoid body (e.g., NAO, Zeno, and Kaspar). This can prevent full interaction with the children, which may also produce violent actions. Tito [11], could move on the floor, had arms and head, and was mostly intended to elicit imitation and sharing behaviors. It was “sufficiently robust to sustain rough interplay situations” but rigid, although covered by soft skin. Different experiments have been done with the soft, huggable robot Probo [12], e.g., to develop attention. Probo cannot move around on its own so spatial interaction was not explored with it. Soft robots, intended to be hugged, such as Paro, have been mostly used to develop attention (e.g., [13]). Paro cannot move around on its own and only exploits touch and sound input channels.
Other robots have been designed and used to explore spatial movement possibilities. Roball [14,15] and QueBall [16] have spherical shape, can roll on the floor, are partially autonomous, and are able to react to some sensor input. Given their small dimensions (about 20 cm in diameter), they are intended to be manipulated. Their bodies are made of rigid plastic inappropriate for many natural actions, like hugs. Another mobile robot IROMEC can be used to work with people with ASD [17]. It was partially autonomous and could also be equipped with an optional sensitive fur. Unfortunately, at the end of the project, the prototypes were not reliable enough to be extensively tested [18].
Other important aspects are emotional expression and emotion recognition capabilities (e.g., [19,20]). Humanoid robots have been used to implement emotional gestures, while many robots could use sounds, lights, and body movements to express emotions. The development of dyadic and triadic interaction was also obtained by Keepon [21], a soft, small, fixed robot able to interact by contracting or expanding its body. Its dimension, mechanical movement, and sensors could not enable a full touch-based interaction.
Teo [1], the first version of TeoG, was designed to fully exploit the possibility to move on the floor and, at the same time, to show emotional reactions and to react to touch. The first experiments were done by integrating Teo, an external screen and a Kinect 3D camera in the setting. This setting was devised to involve children in therapeutic games, while free play capabilities were explored only by remotely driving the robot through a joystick. The new TeoG no longer needs Kinect and the big screen for structured games and can now be autonomous in free play activity and emotional reactions.
Most of the aforementioned robots can be used as therapeutic tools, included in specific protocols where activities have specific aims. It turns out that people with disabilities spend most of their time in “therapies” and do not have much time and proper tools to just “play.” Play is an important aspect of the human development [22,23], and, in particular, “play for sake of play” (or “free play”) [3] is an activity that can trigger unexpected behaviors and develop abilities. This is the type of play that best matches the definition given by Garvey [24]: “Play is a range of voluntary, intrinsically motivated activities normally associated with recreational pleasure and enjoyment.” This type of play is one of the rights recognized for all children by the United Nations [25]. TeoG has been designed with the goal of enabling free play by offering a wide set of possible interaction means that can safely be explored and managed without the need of caregiver activity, as described in the next section.
2 Methods
2.1 Robot description
TeoG was codesigned and implemented by a team of more than 20 engineers, designers, psychologists, and therapists along with the important contribution of subjects with NDD. Functionalities, shape, body, and technical aspects were developed incrementally and in an integrated way. In this section, we present the different aspects separately, in order to provide a structured description.
2.1.1 Shape
The original Teo was about 60 cm high and had an ovoid shape with a diameter of 45 cm at the footprint. It was designed to be a little bit smaller than the intended subjects, so that they could feel in control but not too small. The robot could be considered as a companion. The parameters of TeoG have been adjusted to make it more suitable for both standing and sitting play sessions, as shown in Figure 1. The height of the robot was defined to be about 75 cm, which also makes easier to adapt to kids of different ages, up to early teens. The final version of TeoG is depicted in Figure 2, together with its smaller version.
Example of play session in both sitting (L) and standing (R) positions.
Comparison between final (left, higher) and previous (right, smaller) robot bodies. From this picture, it is also possible to see some of the different eyes and mouth patches that can be attached to the body as well as the places for visual patches corresponding to answers to questions in structured games (see text). When in free play, these are left empty, and embedded LED lights inserted in the patches are used for emotional expression.
The robot can move freely on the floor, thanks to the motors, omniwheels, and electronics integrated in the lower 8 cm of its body, safely covered by a vacuum-formed, hard, flexible, protective shell, aimed, from one side at reducing the possible damage of the mechanical and electronic parts of the robot, and on the other side to prevent potentially dangerous contacts with the subjects. The rest of its body is soft, made like a pillow, in order to have a defined shape, while providing a comfortable, yet known, feeling when hugged. The body is covered by a soft synthetic textile, selected for tactile impressions and easy machine washing. This may be periodically performed for sanitation, even more in a pandemic situation as the one we are currently living. A light yellow color was selected as it is neutral, so reducing the possibility to induce specific, negative, personal reactions, but at the same time bright enough to emerge once placed in any living or caregiving space, inducing curiosity about the robot. Moreover, many recent cartoon characters are yellow so subjects can be more confident toward the new object since they are familiar with these cartoons. Using familiar colors for the robot body was a general recommendation provided by psychologists.
A special, white area is present on the front where different types of eyes and mouths can be freely attached and detached, in order to give to the subjects the possibility to personalize TeoG and get familiar with it. In addition, therapists consider this important to develop and recognize emotion stereotypes. The subjects can be stimulated to assemble a correct set of eyes and mouth to express the desired emotion. In Figure 2, two sets of eyes and a mouth are shown. Neutral, happy, angry, and sad sets are available, which are designed according to the typical cartoon traditions. Technically, this white area is made of a special cloth that allows male Velcro™ strips to attach quite strongly; being a full Velcro™ friendly area, no restrictions are placed for any face components. Such cloth is available in several colors, and we had to choose between two options: making it the same yellow as the body or adopt a different color. Finally our choice moved toward a strong, but still neutral, detachment from body color, in order to communicate to the user at first sight, where the parts could be attached. Subjects not liking faces can leave the area empty or attach any other Velcro™ component (option still to be explored). Kids playing with the robot never tried to attach face parts on the yellow body. This activity is meant to be a part of the familiarization with the robot, giving the children the possibility to personalize it and start to touch it. A moment where a child attaches the emotional elements to the face is presented in Figure 3.
Subject putting eyes on the robot’s face during a session.
In addition, four round areas have been created around the face area and are designed to allocate several types of visual contents required for the “structured game phase” described in Section 2.2.2. These areas initially used the same white Velcro-friendly cloth, but this solution showed several issues, and they were substituted by a magnetic attachment, included in our patented capacitive sensor, visible in Figure 4.
Magnetic application of visual contents on our patented capacitive sensors.
TeoG also has RGB LED lights, as visible in Figure 5, in order to attract more attention and provide extra effects to reinforce the robot’s reactions. These lights are hidden inside the vacuum-formed base, which is made of strong, transparent polycarbonate. The whole base can light up, while keeping the diodes in a safe, guarded area. In the first versions of the robot, LEDs were kept outside of the opaque plastic cover, but this showed to be a poor and ineffective solution as light was not diffused at all and diodes were exposed to user actions.
Detailed depicting of the LED diffused effect under the vacuum-formed PC shell.
2.1.2 Mechanics and sensors
The robot base includes three omnidirectional wheels driven by three DC motors and distributed every 120°, as shown in Figure 6. These give TeoG the possibility to move freely on the floor at a maximum speed of 1.4 m/s. These movement possibilities are similar to those of potential subjects in a limited environment, such as a session room. This was intended to provide the possibility of implementing behaviors that could engage the subjects as peers of the robot in a natural way: movement or speed limitations (such as in Nao) do not enable a realistic implementation of interactive actions such as “following” or “escaping.” Although the base is holonomic, most of the movements are programmed to be coherent with the position of the “face” of the robot, as expected by the subjects. So TeoG is almost always controlled by tangential and rotational speed. Three HC-SR04 sonar sensors are placed frontally, angled at 20° from each other, and one is placed in the back, hanged over the rear omniwheel (see Figure 6), so as to perceive obstacles and people, and navigate safely. These can also be used to perform pursuit/evade games and to establish relationships and devise games based on distance.
Robot’s core (bottom view): mechanics and computation.
The soft body hosts capacitive sensors. Three large capacitive areas are placed on the lower sides and back of the body (see Figure 7), places where this robot is typically touched, as proved during experimental sessions. A dedicated microcontroller considers the data from the three capacitive sensors and integrates them with the data from an accelerometer (MPU-6050) embedded in the body, in order to distinguish some types of touches and their position through signal analysis; namely, caress, hit, hug, punch, and pat. The first version of TeoG adopted some steel wire meshes with a basic hardware setup that suffered from external fluctuations, providing unreliable data. The final body implements the same size and positioning of sensors, made now with a very thin copper foil machine sewn into the internal lining, while the signal is preprocessed by a slave local board, in order to remove fluctuations along the cable. This setup proved to be much more reliable than the original one.
Distribution of capacitive sensors on TeoG body. Lower front left, lower front right, and upper rear center.
Four capacitive sensors are placed on the top part of the body and can be covered with magnetic fabric patches that correspond to visualizations of possible answers to questions that TeoG can ask for structured games. The patches can be easily replaced by different ones, so as to implement different games or exercises. In the first version of TeoG, these capacitive sensors were implemented with a typical, mesh wire, and basic setup. In the final version, we adopted a local signal preprocessing slave board dedicated to these areas only and replaced the basic sensors with our patented silicon-embedded capacitive sensors [26]. These include magnets to secure the visual patches; matching magnets can be easily inserted into the fabric lining and give a soft and nice tactile impression while lasting for a long time. An MP3 player and loudspeakers complete the actuator set of the robot. Over 150 custom audio tracks were recorded and morphed, so as to avoid time-consuming text-to-speech vocal expressions.
TeoG is controlled by a microcontroller (Arduino Mega – see Figure 6), and a second slave microcontroller (Arduino Nano) for body monitoring is placed above the motor driver. The main board has Bluetooth 4 connection capability, so that it can be remotely controlled by a dedicated application that runs on an Android 6.0 tablet, with a 7.3-inch screen. The functionalities of this application will be presented in Section 2.2.3.
2.2 Functionalities
2.2.1 Free play
Free play is intended as a free exploration of the interaction with the robot through the available channels. TeoG can autonomously react to the detected signals in a consistent way, so that exploration can bring the subject to control the interaction using movements and touch, so as to develop autonomy, motivate attention, and trigger curiosity. In the free play activity, TeoG is programmed to show emotional behaviors in reaction to some actions of subjects. For instance, if the subject hugs TeoG, it slightly moves from left to right showing a slowly fading cyan light and emitting a pleasure sound. If the subject hits TeoG, it runs away from the direction of the hit, as detected by the capacitive sensors and accelerometer, complaining about having been hurt and showing red lights blinking at an appropriate rate. The same visual and sound reaction happens when the robot is flipped, until it is taken back to its standing position; after few seconds, TeoG moves slowly toward the user, ready to start again. Other than touch reaction, a wide range of games and actions have been implemented, from dancing sessions with movements and lights (very engaging and eye-catching experiences) to more or less complex moving patterns that can attract attention of the subjects and can then be replicated by them.
Autonomous behaviors are important to improve animacy [27] and to make the subjects perceive the robot as a companion so as to elicit real relationships with the robot. Although the same behaviors could be triggered by the therapist through the developed app, this would take time, including the reaction time of the therapist, and may be perceived as less natural. Robot autonomy allows the therapist to take a more observational role useful to reduce personal burden and to improve the quality of reports, while full control of the interaction is always possible.
One interesting free play feature emerged from sessions with autistic teenagers. The remote control possibility, implemented as a backup for therapists, proved to be a nice exercise for kids who have an attraction toward digital items. A special navigation system was adopted; it uses four arrows to move around, and has five sets of emotive faces represented by simple emojis. The arrows are used to control the robot, which will always move with its face pointing forward, e.g., by turning left if the left arrow is pressed, in order to also exercise spatial coordination. Emotion expressions can be used in conjunction with the arrows, and set specific linear and angular speeds in order to make the robot act as it was sad, happy, or angry. Joining this feature with the custom face expression creates a nice setup to play with emotions.
Some free play activities are also stimulated by the possibility to attach elements onto the Velcro™ “face.” Eyes and mouths with different expressions can be combined. A completely free activity consists of leaving the subjects to select and attach whatever components they like. This encourages confidence to touch the robot while it is kept inactive. They often attach many components more or less randomly, just for the pleasure of doing so. If needed, the therapist can suggest the composition of an emotional expression by combining different elements, and eventually guide the subject to represent this expression. In other cases it is possible to ask the subjects to represent their own emotion or to represent the emotion that would be expected by the robot.
We also implemented a function that monitors the level of sounds around the robot as detected by a microphone. The robot can trigger scared reactions in response to high-volume sounds or strong sounds, such as screams. This behavior has been proved to be interesting both to involve the subjects, who can see the robot reacting to the sound they produce, and to moderate sound excess in single or group sessions.
All the presented behaviors can be inhibited or halted by the therapist through the app when issues arise for specific subjects. The system is designed to relieve the therapist of full control of the free play sessions; they could be guided the first few times and then become self-introducing and self-developing. This is crucial to enable a therapist to evaluate the subject’s activity from an external point of view.
2.2.2 Structured games
In the structured game phase, TeoG can propose questions or activities that require the subject to perform an action that can be perceived by the robot, e.g., touch a patch, jump, and scream. The structured game session is meant to be started and monitored by the therapists.
TeoG includes over 30 exercise scenarios divided by themes and answers, as defined by the therapists. First, the therapist starts the session via the app. TeoG replies vocally to the received command and asks to choose the theme for the exercise (e.g., animals, city, and forest). Once the input is received, the robot asks to choose an exercise that corresponds to the actual set of questions and answers. The next step requires the therapist to physically apply the visual content patches on the robot, as presented by the app in the screen shown in Figure 8. Once everything has been correctly setup, the session can start and can run autonomously until the predetermined end. The subjects have to only touch the visual content they believe is the correct answer to the question. TeoG will then determine whether it is correct and then respond accordingly with a small dance and rewarding speech on correct answers or a sad move and encouraging words on mistakes. The whole interaction is mainly driven by TeoG through vocal stimuli.
Example of visual content placing scheme.
During the exercises, in order to support the therapists and psychologists to check the subject’s improvements, we decided to implement a unique time-stamped log. TeoG, after each completed exercise session, sends to the tablet a string of numbers that contain the information about the subject’s performance and reactions. We store how much time passes between each question and the corresponding touch, and how many touches were needed to get the correct answer. Once the data are sent to the tablet, they are time stamped and saved in an Excel file ready to be shared and further analyzed.
2.2.3 Mobile application
A mobile app has been developed to match the therapists’ requirements and provide an easy and pleasant robot control for nontechnological users. This is essential for acceptance of the system. Before the delivery of the final app, a standard setup was initially used, including a barely functional app that could allow the care center to start testing the robot as soon as possible. The first app was functional but hard to use, very poor on graphics, and with a harsh Graphical User Interface (GUI) and navigation. The therapists were about to give up using TeoG with that app until we provided the final version (see Figure 9). The new interface was implemented and tested without the physical presence of the robot, using a specifically made software that simulates input signals. This allowed the developing team to refine the app while letting the care center experiment and get familiar with the new object.
First version of the app (above) compared to the final app (below).
The app has 12 screens divided into two main sections corresponding to the main functionalities of the robot. One for free play and one for structured exercises. In the free play section, the therapist can trigger over 15 predefined actions, from involvement aimed, such as “follow me,” “catch me,” or “let’s dance,” to free movements, as described in Section 2.2.1. Several predefined audio files are playable at any time during the play phase in order to give a feedback to unexpected actions from the subjects and to reinforce positive actions. Lights can be controlled independently by the therapist to match imaginary play or specific games as the situation may require. Therapists can also turn on or off several aspects of the robot, such as voice, lights, or movements, and use the combination that suits the specific subject the most (Figure 10).
TeoG being controlled by the android tablet.
A debug section is also implemented to check robot functionalities like sonar sensors, battery, microphone, and trigger single sensors to check the system for specific failures.
2.3 Activities
The robot has been used at the care center “Il sogno” since January 2018. About 60 children between 5 and 16 years old are hosted by the care center in daily activities, and TeoG is used at least once a week in several types of sessions. They have all been involved in different sessions held by the five therapists mentioned in the acknowledgment section.
During the first 15 months, the robot was used with single autistic subjects. Sometimes the therapist decided to have TeoG switched off in the room, so that the children could perceive its presence as any other object in the room and become acquainted with it. In this phase, the focus was on visual and tactile sensor aspects. The subject could then play with it, much like with a rather big teddy bear. Then TeoG was turned on in the “free play” modality, so that the children could feel noise, sounds, lights and movements and interact with it. After a free play period, usually a structured game session was started, always depending on the subject’s situation. Sessions must always adapt to the subject’s feelings and need, and TeoG can satisfy different needs.
The exercise sessions present several levels of difficulty, designed to fit several cognitive disorders. The exercise to be played must be selected by the therapist or the psychologist when planning the session. At present, the implemented exercises concern the following:
color matching and color recognition,
numbers and counting,
animals,
figure recognition and figure matching, and
cognitive associations (e.g., animals that fly).
Each exercise includes a specific set of questions and a corresponding graphic page in the app to define how to match the answers with the position of the patches on the sensible areas. Once the exercise is set, the subject is introduced to TeoG. After a self-presentation, it starts proposing questions to be answered by touching one of the capacitive sensors equipped with the patch that shows the visual representation of the answer. A correct answer produces a positive feedback accompanied by a dance, music, and LED patterns. A wrong answer produces a feedback and an encouragement. The game continues until all questions are played.
Before the development of the final app, the therapists had a hard time with these games as they had to place the visual content following an instruction leaflet. This led to long setting times, mistakes, and problems in matching the exercise number with the correct setup. Since the new app was released, the setting has become easier and faster. A lot of testing was done, leading to requests for further improvements, too. From the latest upgrade that included a better functioning body and a new custom-made app, TeoG has been used in different settings, also considering both the new and available options and the fact that several subjects had already been acquainted with TeoG. The most interesting innovation was the activation of group sessions, where groups of three to five children with similar characteristics had full familiarization and game sessions with TeoG. This inspired the therapists to set new goals, starting a new series of innovative sessions where, for instance, attention and turn taking were needed. Depending on the cognitive status of the group, TeoG was used to achieve the following goals:
activate the interest of the children for an object, which was simpler than to establish a relationship with another person, such as the therapist or a companion;
stimulate senses: touch, sound, movement, and lights are involved in the relationship;
develop relationships with the robot and, through the robot, with other subjects;
develop verbal and nonverbal communication;
develop visual following;
develop motor activity;
regulate emotions;
develop both single and shared attention;
develop turn-taking abilities in facing the exercises;
develop short-term auditory memory;
develop verbal understanding.
3 Results
Given the wide spectrum of disabilities treated at the care center and the relatively small number of the subjects, therapists usually treat each individual as a specific case. Therefore, the reports we obtained could only include specific experiences, and general feedback about how they could use TeoG. It is important to state that TeoG at the beginning was not always working properly, but even the early failures provided interesting insights. The final months of testing, after the final upgrade, led to very interesting feedback from the therapists, and this is now driving TeoG toward further improvements. Therefore, the feedback from therapists will be presented separately for the two testing periods in the next paragraphs.
3.1 The first months
The first months of TeoG at the care center were crucial to refining a robot that was intended to be used in a real situation, without the presence of lab staff, since the care center is more than 400 km away from the lab. Integrating the experience gained with the previous versions of the robot, a team composed by two engineers and two designers refined TeoG by interacting with the care center remotely. Even though the team figured out how to simulate all the possible failures and problems, at the beginning it was not possible to interact strictly with the therapists. Different people have different ways of thinking, and different situations have different problem setters. Therefore, despite the hard work done during the implementation and testing phase, some issues showed up and, even if the robot was used quite often, the best result that could be obtained was to open a sharing channel between the care center staff and the developers’ team. Sharing the experiences and the frustrations from both sides made it possible to design a better robot that could finally be used to support the subjects. Nevertheless, a lot of dancing and happiness was brought by TeoG into the care center even in this period.
3.2 The last months
Once the robot was upgraded and finally ready to be used, the therapists started to work seriously with TeoG, implementing new activities. They started to note down several positive aspects that changed in the final months, and reported that often, if the robot was not present in the room while taking a standard session, subjects looked for it and asked to bring TeoG in. If the robot was there, they looked at it, touched it, and asked the therapist, each according to her/his capability, either verbally or not, to interact with the robot.
Among the episodes that happened in the “free play” phase, we would like to mention the first meeting, between a 5-year-old child with mid-level ASD and the robot. Once in the room, the child noticed TeoG that was just speaking to try to trigger an interaction. After observing the situation for a few seconds, he started to run against TeoG and completely flipped it using his body weight (see sequence in Figure 11). TeoG is designed to withstand such harsh behaviors without any damage. In previous versions, it was punched many times, impersonated a wrestler in an imaginary match, and was even ridden like a little horse. The softness of the top part, and both the limited dimension and protection of the lower part, make it also intrinsically safe for subjects even in extreme situations. In this case, once flipped, the robot stopped moving, as it does for safety reasons, and started sobbing and crying, fading on and off the lights at a smooth pace. The child was surprised by the reaction and turned it up immediately with a quite impressive effort, since the robot weighs about 12 kg. This game happened a couple more times in about 5 min. Then he started to play with TeoG, trying to move it on top of objects that he could find in the room, having the same color of the robot’s body, caressing it, and hugging it freely. The attention dropped after about 20 min, and he was then ready to start his music therapy as it was scheduled.
Sequence depicting a full body, harsh approach.
As an example of a structured game, we were able to see a session where TeoG was facing three kids sitting in their small chairs about 2 m away. It was proposing questions about animals, which could be answered by pressing the corresponding patch. The three kids were able to play three consecutive sessions consisting of four questions each by keeping their turn and dancing with the robot when the answer was correct. The therapist was only checking that they took their turns correctly, since two of them were at a level lower than the third one. After the first round, the kids were all able to take their turns by themselves and gave a lot of attention to the questions, so as to be able to dance with the robot. At the end, a final dance triggered by the therapist involved everyone.
As already mentioned, the therapists were exploiting the characteristics of TeoG in different ways with the different subjects, considering the characteristics of each one, as is common in most real settings. In general, they were extremely satisfied with the possibilities provided by TeoG and provided analytical judgments about the results obtained so far. They could assess a stable improvement in both verbal and nonverbal communication abilities, visual following, attention, focus on the proposed activity, motor activity, and turn taking on all the subjects.
We report here a list of notes that were provided after the last 6 months of use of the upgraded TeoG. Children who cannot move follow the movement of the robot with their eyes. Children who usually have problems with touching want and do hug and caress TeoG to obtain its feedback. Children usually very sensitive to noise accept the sounds produced by TeoG since they are motivated. Children who usually do not communicate verbally communicate their desire to interact with the robot through gestures. Children with problems to control their movement respect breaks and turn taking. For some subjects, the presence of TeoG triggered overexcitement in the first session, but this reduced in the following sessions, generating emotional regulation without the need of any special intervention. Through exercises, children could increase their attention span encouraged by the feedback, which was also produced verbally by TeoG. The verbal feedback triggered expectation, and this was a sufficient driver to support performance of the requested actions. Most children can now remember the requests in the exercises, understand them, and do what is required.
4 Discussion
The reported results, i.e., the general satisfaction of the therapists and their deep involvement, support the validity of the hypothesis about the potential interest of a robot like TeoG. It demonstrated the possibility of being able to autonomously manage free interaction, thereby relieving the therapists of real-time control burdens and leaving them to observe the interactions from a third-party point of view while still maintaining control of the situation. They can use the robot as a tool, planning how to use it for each specific subject and exploiting the wide variety of possibilities that it provides. Moreover, therapists could always monitor the session and directly intervene by proposing variations and exploring possibilities through the robot. They could keep their relationship with the subjects at a level different from the one they had if interacting personally in the session. Thus, the subjects can gain autonomy in the development of their own interaction modality, without the limitations that a relationship with a person that has the specific role of therapist may introduce.
TeoG has been operational for almost 3 years, with its ups and downs, requiring three interventions, one to fix a hardware failure due to a cheap component, another to upgrade the system to the present status, and the latest to change the almost exhausted battery and apply some fixes and improvements requested after intensive use. One improvement concerned the songs that were played in different situations. Some of them were in English and therapists required that only Italian songs were included, so that they could be understood and possibly repeated by the subjects, or even used to trigger further activities. Some expressive movements, including those used as reinforcement, were not played with music. Therapists suggested to always play some music, so that a ballet could be played together with the robot. This feature may also engage subjects in the motor dimension and has been added. Originally, if the answer to a question was wrong, a feedback was provided, and the robot just waited to obtain the right answer. Sometimes, subjects could not remember the question. Now the robot repeats the question that was not correctly answered, to help subjects to remember what was required.
Overall, TeoG proved to be a stable yet useful tool at its final stage, much more than what was evident in the previous versions designed back in 2015, where the presence of technical staff, or at least specifically trained persons, was crucial to keep the robot running.
5 Conclusion
We have presented a new robot, TeoG, designed for activities with people with NDD. TeoG covers a niche not yet explored in the use of robots with these people, being both mobile, rather big, soft, and touch sensitive. Moreover, TeoG can autonomously support natural interaction, including mistreatment and can react appropriately, so relieving the therapists of direct involvement in the relationship with the subjects. In this way, the therapists can dedicate more attention to observe what is happening and also play, if needed, a role mediated by the robot, which could not be played in first person. For instance, the therapist could not cry when punched by the subject, but the robot can, and this, together with the high degree of animacy that is attributed to the robot, makes it possible to elicit interesting interactions.
We implemented in TeoG the possibility of both free and structured play. Playing for the sake of play is a need for all, and it is triggered by curiosity and a need for exploration. Subjects with NDD need to feel safe and be in control to be free to really play. The simple reactions of TeoG to their actions can support this feeling and the relative richness of the multimodal interaction possibilities offer a sufficiently wide range of experiences to explore.
Structured exercises are explicitly requested by therapists, since they feel confident when they can plan to achieve a specific result and act accordingly. This type of activity has been implemented by exploiting the movement and multimodal interaction possibilities of TeoG, thereby providing an approach different from the more static and even more structured and demanding activity that could be obtained by the interaction with a screen-based application. We did not implement any speech recognition system since at the moment this technology cannot provide enough natural and reliable interaction possibilities with the type of subjects in a typical care setting. Touch-based interaction is very basic and can be exploited by most subjects.
TeoG was tested for more than 3 years in two main versions, satisfactorily supporting single and group sessions. It became part of the tools used daily by the therapists. Only three interventions by the developers were needed in these years, thus showing that it is possible to implement low-cost robots, like TeoG, robust enough to be managed almost autonomously by a care center. The obtained results have been judged by the therapists as very satisfying.
As expected, with the use, therapists were stimulated to think of new ways to use the robot and had interesting suggestions about new games and features that will be implemented in the next version of the robot. We are also experimenting with low-cost hardware architectures including Graphical Processing Unit and Tensor Processing Unit to exploit deep learning models to recognize people, actions, and objects, which will enable the definition of new interaction ways and new games. New, cheap touch sensors have been developed (patent pending), covering the whole body. These enable the identification, together with accelerometer data, of a much wider set of touch types, through a neural network running on Raspberry PI. They will be integrated in the next versions of this robot.
Acknowledgments
We would like to thank the subjects involved in the activity with TeoG, and the therapists and professionals involved in this project, who tested and stressed the robot in the field, defined protocols, and suggested valued improvements: Giovanna Mariani, Development and Education Psychologist and Therapist of Neuro and Psychomotricity of the developmental age, Lucia Cavani, Speech Therapist, Sara Martinelli, Therapist of Neuro and Psychomotricity of the developmental age, Francesca Maggianetti, Psychopedagogist, Joyce Caresia, Music Therapist.
-
Conflict of interest: Andrea Bonarini is an editor of the Paladyn, Journal of Behavioral Robotics and was not involved in the review process of this article.
-
Data availability statement: All data generated or analysed during this study are included in this published article.
References
[1] A. Bonarini, F. Garzotto, M. Gelsomini, M. Romero, F. Clasadonte, and A. N. Ç. Yilmaz, “A huggable, mobile robot for developmental disorder interventions in a multi-modal interaction space,” in 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2016), IEEE, Piscataway, NJ, 2016, pp. 823–830, https://doi.org/10.1109/ROMAN.2016.7745214.10.1109/ROMAN.2016.7745214Search in Google Scholar
[2] A. Bonarini, F. Clasadonte, F. Garzotto, M. Gelsomini, and M. Romero, “Playful interaction with Teo, a mobile robot for children with neurodevelopmental disorders,” in Proceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, ACM, 2016, pp. 223–231, https://doi.org/10.1145/3019943.3019976.10.1145/3019943.3019976Search in Google Scholar
[3] S. Besio, “The need for play for the sake of play,” in Play development in children with disabilities, S. Besio, D. Bulgarelli, and V. Stancheva-Popkostadinova, Eds., Berlin: Walter de Gruyter GmbH & Co KG, 2016, pp. 9–52, https://doi.org/10.1515/9783110522143-003.10.1515/9783110522143-003Search in Google Scholar
[4] B. Scassellati, H. Admoni, and M. Mataric, “Robots for use in autism research,” Annu. Rev. Biomed. Eng., vol. 14, pp. 275–294, 2012, https://doi.org/10.1146/annurev-bioeng-071811-150036.10.1146/annurev-bioeng-071811-150036Search in Google Scholar PubMed
[5] J. J. Cabibihan, H. Javed, M. Ang Jr., and S. M. Aljunied, “Why robots? A survey on the roles and benefits of social robots in the therapy of children with autism,” Int. J. Soc. Robot., vol. 5, pp. 593–618, 2013, https://doi.org/10.1007/s12369-013-0202-2.10.1007/s12369-013-0202-2Search in Google Scholar
[6] J. J. Diehl, L. M. Schmitt, M. Villano, and C. R. Crowell, “The clinical use of robots for individuals with autism spectrum disorders: A critical review,” Res. Autism Spect. Dis., vol. 6, no. 1, pp. 249–262, 2012, https://doi.org/10.1016/j.rasd.2011.05.006.10.1016/j.rasd.2011.05.006Search in Google Scholar PubMed PubMed Central
[7] W. L. J. E. den Brok and P. S. Sterkenburg, “Self-controlled technologies to support skill attainment in persons with an autism spectrum disorder and/or an intellectual disability: a systematic literature review,” Disabil. Rehabil. Assist. Technol., vol. 10, no. 1, pp. 1–10, 2015, https://doi.org/10.3109/17483107.2014.921248.10.3109/17483107.2014.921248Search in Google Scholar PubMed
[8] M. Jouaiti and P. Hénaff, “Robot-based motor rehabilitation in autism: a systematic review,” Int. J. Soc. Robot., vol. 11, pp. 753–764, 2019, https://doi.org/10.1007/s12369-019-00598-9.10.1007/s12369-019-00598-9Search in Google Scholar
[9] S. Shamsuddin, H. Yussof, L. Ismail, F. A. Hanapiah, S. Mohamed, H. A. Piah, et al., “Initial response of autistic children in human-robot interaction therapy with humanoid robot NAO,” in 2012 IEEE 8th International Colloquium on Signal Processing and its Applications (CSPA), IEEE, 2012, pp. 188–193, https://doi.org/10.1109/CSPA.2012.6194716.10.1109/CSPA.2012.6194716Search in Google Scholar
[10] B. Robins, K. Dautenhahn, R. Te Boekhorst, and A. Billard, “Robotic assistants in therapy and education of children with autism: can a small humanoid robot help encourage social interaction skills?” Univ. Access Inf. Soc., vol. 4, pp. 105–120, 2005, https://doi.org/10.1007/s10209-005-0116-3.10.1007/s10209-005-0116-3Search in Google Scholar
[11] A. Duquette, F. Michaud, and H. Mercier, “Exploring the use of a mobile robot as an imitation agent with children with low-functioning autism,” Auton. Robot., vol. 24, no. 2, pp. 147–157, 2008, https://doi.org/10.1007/s10514-007-9056-5.10.1007/s10514-007-9056-5Search in Google Scholar
[12] R. E. Simut, J. Vanderfaeillie, A. Peca, G. Van de Perre, and B. Vanderborght, “Children with autism spectrum disorders make a fruit salad with Probo, the social robot: an interaction study,” J. Autism Dev. Disord., vol. 46, no. 1, pp. 113–126, 2016, https://doi.org/10.1007/s10803-015-2556-9.10.1007/s10803-015-2556-9Search in Google Scholar PubMed
[13] L. B. Bertel and D. M. Rasmussen, “PEERs at play: a case study on persuasive educational and entertainment robotics in autism education,” in Proceedings of the International Workshop on EuroPLOT Persuasive Technology for Learning, Education, and Teaching (IWEPLET 2013), 2013, pp. 161–168.Search in Google Scholar
[14] F. Michaud and S. Caron, “Roball, the rolling robot,” Auton. Robot., vol. 12, pp. 211–222, 2002, https://doi.org/10.1023/A:1014005728519.10.1023/A:1014005728519Search in Google Scholar
[15] Leka, available: https://www.leka.io/. [Last accessed: December 27, 2020].Search in Google Scholar
[16] T. Salter, N. Davey, and F. Michaud, “Designing and developing QueBall, a robotic device for autism therapy,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication, Edinburgh, 2014, pp. 574–579, https://doi.org/10.1109/ROMAN.2014.6926314.10.1109/ROMAN.2014.6926314Search in Google Scholar
[17] R. J. F. van den Heuvel, M. A. S. Lexis, R. M. L. Janssens, P. Marti, and L. P. de Witte, “Robots supporting play for children with physical disabilities: exploring the potential of IROMEC,” Technol. Disabil., vol. 29, no. 3, pp. 109–120, 2017, https://doi.org/10.3233/TAD-160166.10.3233/TAD-160166Search in Google Scholar
[18] R. J. F. van den Heuvel, M. A. S. Lexis, and L. P. de Witte, “Can the IROMEC robot support play in children with severe physical disabilities? A pilot study,” Int. J. Rehabil. Res., vol. 40, no. 1, pp. 53–59, 2017.10.1097/MRR.0000000000000200Search in Google Scholar PubMed
[19] L. Boccanfuso, E. Barney, C. Foster, Y. A. Ahn, K. Chawarska, B. Scassellati, et al., “Emotional robot to examine different play patterns and affective responses of children with and without ASD,” in 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), IEEE, Christchurch, 2016, pp. 19–26, https://doi.org/10.1109/HRI.2016.7451729.10.1109/HRI.2016.7451729Search in Google Scholar
[20] M. J. Salvador, S. Silver, and M. H. Mahoor, “An emotion recognition comparative study of autistic and typically-developing children using the zeno robot,” in 2015 IEEE International Conference on Robotics and Automation (ICRA), IEEE, Seattle, WA, 2015, pp. 6128–6133, https://doi.org/10.1109/ICRA.2015.7140059.10.1109/ICRA.2015.7140059Search in Google Scholar
[21] H. Kozima, C. Nakagawa, and Y. Yasuda, “Interactive robots for communication-care: a case-study in autism therapy,” in IEEE International Workshop on Robot and Human Interactive Communication (ROMAN 2005), IEEE, Nashville, TN, USA, 2005, pp. 341–346, https://doi.org/10.1109/ROMAN.2005.1513802.10.1109/ROMAN.2005.1513802Search in Google Scholar
[22] L. S. Vygotsky, “Play and its role in the mental development of the child,” Sov. Psychol., vol. 5, no. 3, pp. 6–18, 1967, https://doi.org/10.2753/RPO1061-040505036.10.2753/RPO1061-040505036Search in Google Scholar
[23] J. Piaget, Play, Dreams and Imitation in Childhood, Abingdon: Routledge, 2013.10.4324/9781315009698Search in Google Scholar
[24] C. Garvey, Play, Cambridge, MA, USA: Harvard University Press, 1990.Search in Google Scholar
[25] United Nations, Human Rights, Convention on the Rights of the Child, General United Nations Assembly Resolution, 1990.Search in Google Scholar
[26] A. Brivio, K. Rogacheva, M. Zambelli, A. Bonarini, M. Merigo, and M. Romero, Sensore Capacitivo e Metodo di Realizzazione di Tale Sensore Capacitivo, Patent WO 2019/123503A1, Dec. 19, 2018.Search in Google Scholar
[27] H. Fukuda and K. Ueda, “Interaction with a moving object affects one’s perception of its animacy,” Int. J. Soc. Robot., vol. 2, pp. 187–193, 2010, https://doi.org/10.1007/s12369-010-0045-z.10.1007/s12369-010-0045-zSearch in Google Scholar
© 2021 Andrea Brivio et al., published by De Gruyter
This work is licensed under the Creative Commons Attribution 4.0 International License.
