16 Human Computer Confluence in the Smart Home Paradigm: Detecting Human States and Behaviours for 24/7 Support of Mild-Cognitive Impairments

: The research advances of recent years in the area of smart homes highlight the prospect of future homes equipped with sophisticated systems that monitor the resident and cater for her/his needs. A basic prerequisite for this is the development of non-obtrusive methods to detect human states and behaviours at home. Especially in the case of residents with mild cognitive impairments (MCI), such systems should be able to identify abnormal behaviours and trends, supporting independent living and well-being through appropriate interventions. The integration of monitoring and intervention mechanisms within a home needs special attention, given the fact that after a period of time, these will be perceived from the resident as inherent home features, altering the traditional way that the notion of home is perceived by the mind, transforming it into a Human Computer Confluence (HCC) paradigm. Activity detection and behaviour monitoring in smart homes is typically based on sensors (e.g. on appliances) or computer vision techniques. In this chapter, both approaches are explored and a system that integrates sensors with resident vision-based location tracking is presented. Location tracking is based herein on low-cost depth cameras (Kinect), allowing for privacy preserving, unobtrusive monitoring. The focus is on detecting the MCI resident’s Activities of Daily Living (ADLs), as well as extracting parameters, toward identifying abnormalities within her/his behaviour. Preliminary results show that the sole use of user position trajectories has potential toward effective ADL and abnormality detection, whereas the addition of sensors further enhances effectiveness, with increase however in system cost and complexity.


Introduction
Promoting independent living and well-being of people with mild cognitive impairments (MCI) is a significant challenge.Due to increasing life expectancy, the world population is continuously aging.Aging is often accompanied by MCI, leading to functional limitations and impairments in daily life (Ahn, et al., 2009;Wadley, Okonkwo, Crowe, & Ross-Meadows, 2008), degrading the way activities of daily living (ADLs) are performed and the person's capability of effectively catering for her/his own needs.MCI is an intermediate stage between the expected cognitive decline of normal aging and the more serious decline of dementia (Ahn, et al., 2009); it can involve problems with memory, language, thinking and judgment that are greater than normal agerelated changes.MCI can thus affect the person's ability to perform ADLs consisting of a series of steps, involving cognitive functions, related for instance to telephone usage, meals preparation, medication, management of belongings etc. (Ahn, et al., 2009).As the patient's cognitive state is reflected on daily activities, the capacity of the patient to perform these activities, or significant changes in the way that they are performed, can provide valuable clues toward the evaluation of MCI and its progression into dementia.
Future smart homes can play a vital role toward promoting independent living and well-being of MCI and dementia residents, by providing sophisticated 24/7 support for both the proper execution of ADLs and the early identification of cognitive decline signs.Following the research and technological advances of recent years in the area of smart homes, future houses are expected to transform into living environments that monitor the behaviour of their residents, adjust over it and allow for residenthome interaction that eases daily life and promotes well-being.A prerequisite for this is the development of non-obtrusive methods to detect human states and behaviours at home.Especially in the case of residents with mild cognitive impairments (MCI), such systems should be able to identify abnormal behaviours and trends, supporting independent living and well-being through appropriate interventions.These interventions may range from stimulating the resident to undertake a forgotten step of a meal preparation process (e.g.turning the oven off) to informing her/him about e.g. a significant increase in the duration of meal preparation or eating activities, decrease in resting activities, pointless movement around the house etc.The integration of monitoring and intervention mechanisms within a home needs however special attention, given that after a period of time, these will be perceived by the resident as inherent home features, altering the traditional way that the notion of home is perceived by the mind, transforming it into a Human Computer Confluence (HCC) paradigm.
In this line, the present chapter examines how in-house activity detection systems can be applied in practice, providing the basis of HCC in the smart home paradigm by enabling the home to sense its resident's behaviour.Typically, ADL detection is either based on ambient sensors that monitor for instance the operational state of appliances, or on vision-based techniques that monitor the resident directly.In this chapter, both approaches are explored; a novel framework for resident location trajectory-based ADL detection, built from a network of Hidden Markov Models (HMMs) and Support Vector Machines (SVMs) is introduced and compared with an ambientsensor based approach, whereas an ADL monitoring system integrating sensors with resident location tracking is also presented.

Detecting Human States and Behaviours at Home
The present section overviews the current state of art in activity detection within smart homes.Typically, the goal of relevant systems is to detect whether the resident is engaged in some of the "Basic ADLs", such as eating (Katz, Ford, Moskowitz, Jackson, & Jaffe, 1963) or "Instrumental ADLs", e.g.cooking (Lawton & Brody, 1969) and, once an ADL is detected, to monitor its characteristics.

Ambient Sensor-Based Activity Detection
The capability of ambient sensors to capture the monitored person's state has been extensively exploited toward activity detection (Chen, Hoey, Nugent, Cook, & Zhiwen, 2012).One of the first examples was presented by Tapia, Intille, and Larson (2004), where a wireless network of unobtrusive state-change sensors was deployed at home and the naive Bayes algorithm was used to learn the resident's ADLs.In a similar fashion, van Kasteren and Krose (2007) experimented with static vs. dynamic Bayesian networks, trying to incorporate temporal information in the detection process.They also studied the effect of the total number of sensors on detection accuracy, concluding that increasing sensor count above some level does not necessarily improve performance, arguing instead in favour of using small sets of strategically located sensors.Van Kasteren, Noulas, Englebienne, and Krose (2008) studied Hidden Markov Models (HMM) and Conditional Random Fields.HMMs were also used by Singla, Cook, and Schmitter-Edgecombe (2008), with state duration explicitly modelled after a Gaussian distribution.Furthermore, van Kasteren, Englebienne, and Kröse (2011) exploited the inherent hierarchical nature of activities via a two-layer hierarchical HMM, where the top layer corresponded to activities and the bottom to unit actions.Discriminative classification was preferred by Fleury, Vacher, and Noury (2010), where a Support Vector Machine (SVM) was trained to automatically discriminate among activities using a multimodal set of sensors.
The fusion of information among heterogeneous sensors is an important practical issue.In this line, Medjahed, Istrate, Boudy, and Dorizzi (2009) used fuzzy logic for information fusion, together with fuzzy rules for activity inference.Liao, Bi, and Nugent (2011) used the Dempster-Shafer theory of evidence, accounting for the relative uncertainty of sensor readings.In a recent approach, Chen, Nugent, and Wang (2012) presented an ontological model of the smart home universe, addressing variability in sensor modalities by sensor abstractions.

Resident Movement-Based Activity Detection
Parallel to sensor-based activity detection are approaches that rely on monitoring the way the resident moves.Le, Di Mascolo, Gouin, and Noury (2008) monitored areas of interest by passive infra-red (PIR) sensors and represented activities as sequences of moving and stationary states.Duong, Phung, Bui, and Venkatesh (2009) used a two-layer Hidden Semi-Markov Model to infer activities solely based on resident location trajectories.Park and Kautz (2008) combined wide-field-of-view cameras with narrow-field-of-view cameras, for coarse-and fine-level activity detection respectively.Similarly, using only a fisheye camera, Zhou et al. (2008) collected activity statistics in a hierarchical manner at multiple levels of detail.Recent advances in depth sensing, resulting to low-cost depth sensors such as Microsoft Kinect (Zhang, 2012), have pushed research in vision-based activity recognition forward.Zhang and Tian (2012) utilized skeleton information.Noting that skeleton extraction methods perform poorly under occlusions or cluttered background, Zao, Liu, Yang, and Cheng (2012) proposed extracting local features from the RGB and depth channels instead.

Applications and Challenges of ADL Detection in Smart Homes
Accurate ADL detection is a necessary condition in order to build truly occupantaware houses.It allows for extracting ADL-related parameters, toward efficient modelling of resident behaviour.This enables the detection of long-term behavioural trends as well as abnormalities, allowing smart homes to be utilized for elder care and support, as well as for early diagnosis of otherwise unidentifiable symptoms of mental or physical deterioration.To this end, Lotfi, Langensiepen, Mahmoud, and Akhlaghinia (2012) presented a framework for detecting abnormal behaviour of early dementia sufferers, based on deviations from predicted long-term trends.Trustworthy abnormality detection can also enable smart home systems to be more than a sensing device and intervene when necessary, generate alerts, engage the resident in positive action and assist in decision making (Hossain & Ahmed, 2012).Ambient-intelligence context-aware environments are eventually transformed into an HCC paradigm in the service of human support and well-being.
For such a confluence between the smart environment and the resident to be made practically feasible, certain design and implementation issues need to be tackled.It is of vital importance that the smart system be integrated in the home smoothly and transparently, without disrupting normal resident behaviour or compromising the comfort one feels at home.Privacy is an important issue that must be respected and non-obtrusiveness is a strong prerequisite.This holds for both monitoring and intervention mechanisms, whereas the latter should be designed by also having in mind that it is better to assist residents in a way that helps exercising cognitive skills, instead of merely facilitating their everyday life.Another important issue is the ease of installation, as it should interfere as little as possible with the resident.
Following the above, the present work examines whether ADL detection can be effectively conducted in practice using only a limited set of low-cost depth cameras installed in a house, tracking only the person's location, compared to a more typical ambient-sensor based approach that requires a large amount of sensors installed throughout the house.However, given that ambient sensors can provide further resident behavioural information (e.g.appliances usage statistics), we further explore the potential of integrating the above two approaches toward a more effective ADL monitoring schema.

A Behaviour Monitoring System for 24/7 Support of MCI
In the present section, the system which has been developed in the present study is described, for monitoring six typical ADLs, namely cooking, eating, dishwashing, visiting the bathroom, sleeping and watching TV, on the basis of the two modalities under examination, i.e. resident trajectories and sensor-based features.

Activity Detection Based on Resident Trajectories
Our novel framework for ADL detection through resident trajectories is introduced herein.In order to record location trajectories we used a camera network consisting of a small set of Kinect25 depth sensors.Normally three or four cameras are enough to cover a small apartment, as the one of Figure 16.1.The cameras are calibrated with respect to the house, that is, each camera knows its 3D position and orientation inside the house, the latter serving as a common frame of reference.Each camera maintains a depth image of the house background.The resident, not being part of the background, is detected by comparing all incoming video frames with the stored background image.The difference of the two images is the depth image of the resident at a particular time, from which the relative location of the resident to the camera can be calculated.This procedure is simultaneously performed by all cameras to which the resident is visible at any time.Since for each camera, we know its location in the house and the relative location of the resident with respect to the camera, the exact location of the resident in the house can be computed.This location is continuously tracked by the system, producing a stream of 2D locations.As a last step, this stream of locations is processed to remove possible measurement noise and thus a highly accurate estimate of the resident's 2D trajectory in the house is extracted.
We hypothesize that 2D location trajectories as the ones of Figure 16.1 can be a sufficient indicator of the activity being performed.Our goal is therefore to model trajectories which are typically generated when a person performs a certain activity.Figure 16.1 suggests that this is possible, since different activities produce undoubtedly distinct trajectories.The trajectory of a resident is sequential and probabilistic in nature.As such, we consider HMMs (Rabiner, 1989) as appropriate for modelling it, using one HMM per activity.The HMM's states correspond intuitively to general regions in space, whereas the observed variables correspond to successive points of the 2D trajectory.Each HMM is trained using only trajectories of its respective activity found in the train set, using the Baum-Welch algorithm (Rabiner, 1989).Eventually, our framework (Figure 16.2) consists of an array of HMMs calculating the resemblance between the resident's movement and respective patterns related to the target ADLs.These HMMs provide input to binary SVMs, one per activity, which recognize in turn whether the HMM-based input trajectories resemblance to ADL patterns justifies the occurrence of any of the target ADLs within a given time period.Our novel use of these binary SVMs after the HMMs layer in trajectory-based ADL detection, allows detecting both the simultaneous occurrence of multiple activities in the same interval and no target ADL occurrence.2. For each decision interval t, the resident's trajectory is evaluated against all HMMs, producing a feature vector , where is the log-likelihood score against HMM j, denoting the probability that r(t) is produced by ΗΜΜ j. 3.For each activity i, the feature vector is fed to an SVM, in order to determine whether activity i was being performed during interval t.

Ambient Sensor-Based Activity Detection
Our approach for ambient sensor-based activity detection is based on a multimodal set of Phidget sensors (www.phidgets.com),strategically located in the house (Table 16.1).Sensor selection was driven by three factors: The monitoring system should be unobtrusive and able to integrate naturally in the home, as if it were part of it, without compromising the resident's privacy.The sensors should provide rich information relevant to the target ADLs.Last, the system should be cost-effective and easy to install.Assume again a set of M activities.In our approach, for each activity i є {1,2,…M}, in each decision interval t, a feature is calculated for each sensor j, as the proportion of time, or probability, that sensor j is activated during t, that is: where is the total amount of time that sensor j is activated during interval t, the latter having a total duration of T i .To accommodate the heterogeneity of sensors, we use a separate activation criterion for each sensor type, as shown in Table 16.2.An exception to the above rule was made for proximity sensors on cupboards and the IR receiver.Since activation of those sensors is typically instantaneous and therefore , the number of activations was used as feature instead (i.e.number of opening/closing for cupboards and number of detected IR codes for the TV).
Figure 16.3 depicts our methodology for ambient sensor-based ADL detection.Assume a set of N sensor units providing input data and M activities.The method proceeds as follows: 1.For each activity time is discretized in non-overlapping consecutive intervals of equal per-activity duration T i = aT i avg , by setting again a=0.1.2. For each interval t, a feature is extracted for each sensor and the feature vector is formed.The different per-activity extractor components of Figure 16.3 denote that feature extraction is conducted on time intervals of different per-activity duration.3.Each feature vector is classified with a binary RBF SVM (trained with 10-fold cross validation) to determine whether activity i occurred during interval t.
Activities are detected again independently, allowing detecting occasions where two or more activities occur at the same time, or no target activity is taking place.

Activity Detection Based on Both Sensors and Trajectories
The above two methodologies focus on (a) resident trajectories and (b) sensor readings alone.In this section we move a step forward, combining the two approaches into an integrated and flexible framework, in order to simultaneously exploit information from both modalities.Noting that in essence, both trajectory-based and sensor-based features describe probabilities, and thus the two modalities can be fused by feature concatenation, forming the feature vector: which is used to train and evaluate the final array of SVMs (Figure 16.4).This integrated framework is essentially a generalization of the afore-described two, since the absence of a certain modality easily produces the framework for the other one.

Data Collection
In order to evaluate our proposed ADL detection framework, two experiments were performed, a small-scale one held in a kitchen using only resident trajectories and a larger-scale one held in a real apartment, using both trajectories and sensor-based features.In the kitchen experiment, a single Kinect camera was used to record occupant trajectories (Figure 16.5).Three kitchen activities were involved: cooking, eating and dishwashing.Nine sessions were recorded in total from two subjects, each session consisting of a single person entering the room, performing the three activities once and leaving.Four sessions were used for system training and the remaining five for evaluation.The apartment experiment focused on both trajectory and sensor-based detection.A wireless sensor network of environmental sensors was deployed as described in Table 16.1 and shown in Figure 16.6, together with a network of three Kinect cameras, covering the living-room, the corridor and the kitchen (Figure 16.7).For privacy reasons, the bedroom and bathroom were not covered by cameras.Six activities were involved herein: cooking, eating, dishwashing, visiting the bathroom and watching TV.Three full days of annotated data were recorded, two of which were used for training and one for evaluation.

Experimental Results
In both experiments, our system's detection performance was evaluated by calculating the precision Pr=TP/(TP+FP) and recall Re=TP/(TP+FN) of the detection of each activity.TP is the total number of seconds where the activity was correctly detected.FP is the total number of seconds where the activity was being detected without happening.FN is the total number of seconds where the activity was happening without being detected.

Kitchen Experiment
The results of trajectory-based activity detection in the kitchen experiment are presented in Table 16.3.Furthermore, Figure 16.8 shows a temporal comparison between the actual activities and the detected ones.The results show considerably high detection accuracy, with precision/recall rates often above 90% and all activity instances successfully detected with no false positives.All errors were in fact observed near the start or the end of activities, where it is natural for activities to appear ambiguous even to a human observer.

Apartment Experiment
Table 16.4 and Figure 16.9 present the ADL detection results for the apartment experiment, using (a) sensor-based features, (b) resident trajectories and (c) fusion of both modalities.Since the bathroom and bedroom were excluded from the trajectory-recording process, the bathroom and sleeping activities are excluded from the trajectory-only evaluation.From the above results, it is evident that both modalities demonstrated significant ADL detection potential.When examining each modality alone, average precision and recall rates above 80% were found.Both modalities exhibited comparable performance to each other, with the trajectory-based approach having slightly lower recall.The sensor-based method performed worse for the eating activity, demonstrating precision of only 55.56% as well as a falsely detected instance.This can be explained by the fact that no sensors existed that monitored directly this activity; instead, the detection method inferred the eating activity from the pattern formed by all sensors in general.On the other hand, the trajectory-based method removed the eating false positive, albeit not improving the recall rate.The best performance was achieved by fusing the two modalities, where the average precision exceeded 90% and no false positives or false negatives occurred.Note that most erroneous detection appeared again near the start and end of activities.

Toward Behavioural Modelling and Abnormality Detection
Following ADL detection, our system records a set of parameters for each detected activity, such as its duration and time of occurrence.In case location trajectories are involved, they allow keeping track of locations that the resident visited during an activity and the time s/he spent at each of them.Moreover, the use of ambient sensors allows monitoring the state of house devices during activities.The locations visited or the devices operating during a detected activity can be directly or indirectly associated with it (e.g.state of kitchen stove during cooking or during watching TV respectively).
The set of recorded parameters is then used to model the resident's ADL-specific behaviour, allowing for long-term trend and abnormality detection.For instance, trends related to constant increase in the average duration of daily routine ADLs (e.g.cooking), or even resignation from them can be detected.Indicative detectable abnormalities refer to cases of pointless movement around the house (movement not associated with any specific ADL), or even potentially dangerous situations that call for immediate intervention, such as going to sleep without turning off the stove.Apparently, the joint use of the trajectory and sensor modalities broadens the range of detectable abnormalities.As a further example, through the IR sensor that monitors the TV remote control usage, the resident's difficulties in understanding TV programs can be inferred, by detecting constant continuous changes in TV programs.
The possibility for accurate abnormality detection enhances a smart home with the ability to intervene when necessary.Such intervention can be either handled by the smart home itself (e.g.automatically turn off the stove when the resident is not cooking or urge the resident to do so) or by automatically calling for external help when deemed necessary (e.g.inform the doctor upon detecting restlessness/lack of sleep).

Conclusion
The prospect of future smart homes capable of sensing their residents, adapting to their behaviour and catering for their needs, gives rise to a new form of interaction that is expected to emerge, altering the traditional way that the notion of home is perceived by the mind, transforming it into an HCC paradigm.The present chapter focused on the resident monitoring part of this interaction cycle, a basic prerequisite for HCC within smart homes, whose significance is further highlighted when considering MCI residents.Persons with MCI and dementia often face difficulties in conducting ADLs composed of a series of steps and cognitive tasks, whereas their capability of successfully performing them can be considered as an indicator of their cognitive state.Thus, systems monitoring MCI residents' behaviour and abnormalities can have a significant two-fold impact in their daily life; they will allow (a) interventions facilitating the proper execution of ADLs to take place and (b) assessment of the person's cognitive state to be conducted on the basis of how ADLs are being performed within the person's daily life.
In the present work, ambient sensor-based and vision-based monitoring approaches were examined toward future effective, yet unobtrusive, in-house activity monitoring systems.This resulted in the development of a system, capable of detecting ADLs through the use of a depth-based camera network installed in the house, the use of sensors, or the fusion of the two modalities.Our developed visionbased modality, due to the small number of low-cost cameras needed, allows for low installation costs, both financially and in human effort.Moreover, it utilizes solely streams of depth images, which are significantly less obtrusive than RGB, as they do not capture facial or other identifying characteristics of the resident.Although vision-based, this specific modality can be regarded as less obtrusive also than the sensors modality, since it does not require a large amount of sensors to be installed throughout the house, whereas the only information it processes regards the resident's location.Experimental results showed that the sole use of user location trajectories detected through the vision modality has potential toward effective ADL and abnormality detection, whereas the addition of sensors can further enhance effectiveness and provide further details regarding the resident's behaviour, with an increase however in system cost and complexity.
The present chapter focused on a highly important topic of the HCC field, that of unobtrusive methods to detect human states and behaviours in naturalistic contexts.Providing computer systems with such sensing capabilities can be considered as an important step toward establishing new forms of natural interactions; interactions that will build upon advanced knowledge of the user's state and behaviour, bringing the computer system closer to the human user and advancing their confluence.Herein, emphasis was paid on in-home monitoring of MCI persons' activities, taking into account both the importance of human-computer coexistence in the context of smart homes and the potential HCC applications in the health domain.Unobtrusive sensing of resident activities and behaviour will allow future homes to facilitate on their own, the establishment of daily activities and also cognitive skills monitoring, providing support that will be properly adapted to user needs which evolve over time.Nevertheless, it should be noted that this side of research can have strong impact on the overall HCC field as well, providing the latter with a basic prerequisite; that of advanced context-awareness through user state and behaviour sensing, driving adaptation of system properties to the current situational context of users, toward optimal personalized, adaptive, proactive and invisible interaction.

Figure 16 . 4 :
Figure 16.4:Overview of the generic framework, fusing sensor readings with trajectories

Figure 16 . 8 :
Figure 16.8:Graphical representation of the detection accuracy in the kitchen experiment.Actual activities (green-top), detected activities (red-bottom)

Table 16 . 1 :
Set of sensors used for ADL detection

Table 16 . 2 :
Activation criteria for each sensor type

Table 16 . 3 :
Activity detection performance results for the kitchen experiment

Table 16 . 4 :
Activity detection performance results for the apartment experiment