1 A Research Agenda for Human Computer Confluence

: HCI research over three decades has shaped a wide spanning research area at the boundaries of computer science and behavioral science, with an impressive outreach to how humankind is experiencing information and communication technologies in literally every breath of an individual’s life. The explosive growth of networks and communications, and at the same time radical miniaturization of ICT electronics have reversed the principles of human computer interaction. Up until now considered as the interaction concerns when humans approach ICT systems, more recent observations see systems approaching humans at the same time. Humans and ICT Systems apparently approach each other confluently. This article identifies trends in research and technology that are indicative for the emerging symbiosis of society and technology. Fertilized by two diametrically opposed technology trends: (i) the miniaturization of information and communication electronics, and (ii) the exponential growth of global communication networks, HCC over it’s more than two decades of evolution, the field has been undergoing three generations of research challenges: The first generation aiming towards autonomic systems and their adaptation was driven by the availability of technology to connect literally everything to everything (Connectedness, early to late nineties). The second generation inherited from the upcoming context recognition and knowledge processing technologies (Awareness, early twentyhundreds), e.g. context-awareness, self-awareness or resource-awareness. Finally, a third generation, building upon connect-edness and awareness, attempts to exploit the (ontological) semantics of Pervasive / Ubiquitous Computing systems, services and interactions (i.e. giving meaning to situations and actions, and “intelligence” to systems) (Smartness, from the mid twen-tyhundreds). As of today we observe that modern ICT with explicit user input and output are becoming to be replaced by a computing landscape sensing the physical world via a huge variety of sensors, and controlling it via a plethora of actuators. The nature and appearance of computing devices is changing to be hidden in the fabric of everyday life, invisibly networked, and omnipresent, with applications greatly being based on the notions of context and knowledge. Interaction with such globe spanning, modern ICT systems will presumably be more implicit, at the periphery of human attention, rather than explicit, i


Introduction
Human Computer Confluence (HCC) has emerged out of European research initiatives over the past few years, aiming at fundamental and strategic research studying how the emerging symbiotic relation between humans and ICT (Information and Communication Technologies) can be based on radically new forms of sensing, perception, interaction and understanding.HCC has been identified as an instrument to engage an interdisciplinary field of research ranging from cognitive neuroscience, computational social sciences to computer science, particularly human computer interaction, pervasive and ubiquitous computing, artificial intelligence and computational perception.
The first definition of HCC resulted out of a research challenges identification process in the Beyond-The-Horizon (FET FP6) effort: "Human computer confluence refers to an invisible, implicit, embodied or even implanted interaction between humans and system components.… should provide the means to the user to interact and communicate with the surrounding environment in a transparent "human and natural" way involving all sense…" A working group of more than 50 distinguished European researchers structured and consolidated the individual position statements into a strategic report.The key issue identified in this report addressed fundamental research: "Human computer confluence refers to an invisible, implicit, embodied or even implanted interaction between humans and system components.New classes of user interfaces may evolve that make use of several sensors and are able to adapt their physical properties to the current situational context of users.In the near future visible displays will be available in all sizes and will compete for the limited attention of users.Examples include body worn displays, smart apparel, interactive rooms, large display walls, roads and architecture annotated with digital information -or displays delivering information to the periphery of the observers' perception.Recent advances have also brought input and output technology closer to the human, even connecting it directly with the human sensory and neural system in terms of in-body interaction and intelligent prosthetics, such as ocular video implants.Research in that area has to cover both technological and qualitative aspects, such as user experience and usability" as well as societal and technological implications "Researchers strive to broker a unified and seamless interactive framework that dynamically melds interaction across a range of modalities and devices, from interactive rooms and large display walls to near body interaction, wearable devices, in-body implants and direct neural input and stimulation".
Based on the suggestions delivered with the Beyond-The-Horizon report, a consultation meeting was called for "Human Computer Confluence" in November 2007, out which resulted the FET strategy to "propose a program of research that seeks to employ progress in human computer interaction to create new abilities for sensing, perception, communication, interaction and understanding…", and consequently the implementation of the respective research funding instruments like (i) new forms of interactive media (Ubiquitous Display Surfaces, Interconnected Smart Objects, Wearable Computing, Brain-Computer Interfaces), (ii) new forms of sensing and sensory perception (New Sensory Channels, Cognitive and Perceptual Prosthetics), (iii) perception and assimilation of massive scale data (Massive-Scale Implicit Data Collection, Navigating in Massively Complex Information Spaces, Collaborative Sensing, Social Perception), and (iv) Distributed Intelligence (Collective Human Decision Making, The Noosphere).

Generations of Pervasive / Ubiquitous (P/U) ICT
The novel research fields beyond Personal Computing could be seen as the early 'seeds' of HCC.Preliminarily suffering from a plethora of unspecific, competitive terms like "Ubiquitous Computing" (Weiser et al., 1996), "Calm Computing" (Weiser et al., 1996), "Universal Computing" (Weiser, 1991), "Invisible Computing" (Esler et al., 1999), "Context Based Computing" (UCB, 1999), "Everyday Computing" (Abowd & Mynatt, 2000), "Autonomic Computing", (Horn, 2001), "Amorphous Computing" (Servat & Drogoul, 2002), "Ambient Intelligence" (Remagnino & Foresti, 2005), "Sentient Computing", "Post-Personal Computing", etc., the research communities consolidated and codified their scientific concerns in technical journals, conferences, workshops and textbooks (e.Weiser's seminal vision on the "Computer for the 21st Century" (Weiser, 1991) was groundbreaking, and still represents the corner stone for what might be referred to as a first generation of Pervasive/Ubiquitous Computing research, aiming towards embedded, hidden, invisible and autonomic, but networked information and communication technology (ICT) systems (Pervasive / Ubiquitous ICT, P/U ICT for short).This first generation definitely gained from the technological progress momentum (miniaturization of electronics, gate packaging), and was driven by the upcoming availability of technology to connect literally everything to everything (Connectedness, mid to late Nineties), like wireless communication standards and the exponentially growing Internet.Networks of P/U ICT systems emerged, forming communication clouds of miniaturized, cheap, fast, powerful, wirelessly connected, "always on" systems, enabled by the massive availability of miniaturized computing, storage, communication, and embedded systems technologies.Special purpose computing and information appliances, ready to spontaneously communicate with one another, sensor-actuator systems to invert the roles of interaction from human to machine (implicit interaction), and organism like capabilities (self-configuration, self-healing, self-optimizing, self-protecting) characterize this P/U ICT generation.
The second generation of P/U ICT inherited from the then upcoming sensor based recognition systems, as well as knowledge representation and processing technologies (Awareness, early Two Thousands), where research issues like e.g.context and situation awareness, self-awareness, future-awareness or resource-awareness reshaped the understanding of pervasive computing.Autonomy and adaptation in this generation was reframed to be based on knowledge, extracted from low level sensor data captured in a particular situation or over long periods of time.The respective "epoch" of research on "context aware" systems was stimulated by Schillit, Adams and Want (Schilit et al., 1994), and fertilized by the PhD work of Dey (Dey, 2001), redefining the term "context" as: "…any information that can be used to characterize the situation of an entity.An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and application themselves.".
One result out of this course of research are autonomic systems (Kephart & Chess, 2003).Later, 'autonomic elements' able to capture context, to build up, represent and carry knowledge, to self-describe, -manage, and -organize with respect to the environment, and to exhibit behavior grounded on knowledge based monitoring, analyzing, planning and executing were proposed, shaping ecologies of P/U ICT systems, built from collective autonomic elements interacting in spontaneous spatial/temporal contexts, based on proximity, priority, privileges, capabilities, interests, offerings, environmental conditions, etc.
Finally, a third generation of P/U ICT was observed (mid of the past decade), building upon connectedness and awareness, and attempting to exploit the (ontological) semantics of systems, services and interactions (i.e.giving meaning to situations and actions).Such systems are often referred to as highly complex, orchestrated, cooperative and coordinated "Ensembles of Digital Artifacts".An essential aspect of such an ensemble is its spontaneous configuration towards a complex system, i.e. a "... dynamic network of many agents (which may represent cells, species, individuals, nations) acting in parallel, constantly acting and reacting to what the other agents are doing where the control tends to be highly dispersed and decentralized, and if there is to be any coherent behavior in the system, it has to arise from competition and cooperation among the agents, so that the overall behavior of the system is the result of a huge number of decisions made every moment by many individual agents" (Castellani & Hafferty, 2009).

Beyond P/U ICT: Socio-Technical Fabric
Ensembles of digital artifacts as compounds of huge numbers of possibly heterogeneous entities constitute a future generation of socially interactive ICT to which we refer to as Socio-Technical Fabric (late last decade until now), weaving social and technological phenomena into the 'fabric of technology-rich societies'.Indications of evidence for such large scale, complex, technology rich societal settings are facts like 10 12 -10 13 "things" or "goods" being traded in (electronic) markets today, 10 9 personal computer nodes and 10 9 mobile phones on the internet, 10 8 cars or 10 8 digital cameras with sophisticated embedded electronics -even for internet access on the go, etc. Todays megacities approach sizes of 10 7 citizens.Already today some 10 8 users are registered on Facebook, 10 8 videos have been uploaded to YouTube, like 10 7 music titles haven been labeled on last.fm,etc. Next generation research directions are thus going away from single user, or small user group P/U ICT as addressed in previous generations, and are heading more towards complex socio-technical systems, i.e. large scale to very large scale deployments of ICT to large scale collectives of user up to whole societies (Ferscha et al., 2012).
A yet underexplored impact of modern P/U ICT relates to services exploiting the "social context" of individuals towards the provision of quality-of-life technologies that aim for the wellbeing of individuals and the welfare of societies.The research community is concerned with the intersection of social behavior and modern ICT, creating or recreating social conventions and social contexts through the use of pervasive, omnipresent and participative technologies.An explosive growth of social computing applications such as blogs, email, instant messaging, social networking (Facebook, MySpace, Twitter, LinkedIn, etc.), wikis, and social bookmarking is observed, profoundly impacting social behavior and life style of human beings while at the same time pushing the boundaries of ICT simultaneously.
Research emerges aiming at understanding the principles of ICT enabled social interaction, and interface technologies appear implementing "social awareness", like social network platforms and social smartphone apps.Human environment interfaces are emerging, potentially allowing individuals and groups to sense, explore and understand their social context.Like the human biological senses (visual, auditory, tactile, olfactory, gustatory) and their role in perception and recognition, the human "social sense" which helps people to perceive the "social" aspects of the environment, and allowing to sense, explore and understand the social context is more and more becoming the subject of research.
Inspired by the capacity of human beings acting socially, in that shaping intelligent societies, the idea of making the principles of social interaction also the design, operational and behavioral principle of modern ICT recently has led to the term "socio-inspired" ICT (Ferscha et al., 2012).From both theoretical and technological perspectives, socio-inspired ICT moves beyond social information processing, towards emphasizing social intelligence.Among the challenges are issues of modeling and analyzing social behavior facilitated with modern ICT, the provision access opportunities and participative technologies, the reality mining of societal change induced by omnipresent ICT, the establishment of social norm and individual respect, as well as e.g. the means of collective choice and society controlled welfare.

Human Computer Confluence (HCC)
Human Computer Confluence (HCC) has emerged out of European research initiatives over the past few years, aiming at fundamental and strategic research studying how the emerging symbiotic relation between humans and ICT can be based on radically new forms of sensing, perception, interaction and understanding (Ferscha, 2011).HCC has been identified as an instrument to engage an interdisciplinary field of research ranging from cognitive neuroscience, computational social sciences to computer science, particularly human computer interaction, pervasive and ubiquitous computing, artificial intelligence and computational perception.
In order to further establish and promote the EU research priority on Human Computer Confluence, research road-mapping initiatives were started (see e.g.HCC Visions White Book, 2014), aiming to (i) identify and address the basic research problems and strategic research fields in HCC as seen by the scientific community, then (ii) bring together the most important scientific leaders and industrial/commercial stakeholders across disciplines and domains of applications to collect challenges and priorities for a research agenda and roadmap, i.e. the compilation of white-books identifying strengths, weaknesses, opportunities, synergies and complementarities of thematic research in HCC, and ultimately to (iii) negotiate and agree upon strategic joint basic research agendas together with their road-mapping, time sequencing and priorities, and maintain the research agenda in a timely and progressive style.
One of these initiatives created the HCC research agenda book entitled "Human Computer Confluence -The Next Generation Humans and Computers Research Agenda" (HCC Visions White Book, 2014).It has been published under the acronym "Th.Sc.Community" ("The Scientific Community"), standing for a representative blend of the top leading scientists worldwide in this field.More than two dozens of research challenge position statements solicited via a web-based solicitation portal are compiled into the book, which is publicly available to the whole scientific community for commenting, assessment and consensus finding.Some 200 researchers (European and international) have been actively involved in this process.In addition, the HCC Research Agenda and Roadmap is presented also in the format of a smartphone app, available in online ("The HCC Visions Book"-app).In the past 3 years (2010-2013), the HCC community has become a 650 strong group of researchers and companies working together to understand, not only the technological aspects of the emerging symbiosis between society and ICT, but also the social, industrial, commercial, and cultural impact of this confluence.

The HCC Research Agenda
The HCC research agenda as collected in "Human Computer Confluence -The Next Generation Humans and Computers Research Agenda" (HCC Visions White Book, 2014) can be structured along the following trails of future research:

Large Scale Socio-Technical Systems
A significant trail of research appears to be needed along the boundaries where ICT "meets" society, where technology and social systems interact.From the observation how successful ICT (smartphones, mobile internet, autonomous driver assistance systems, social networks, etc.) have radically transformed individual communication and social interaction, the scientific community claims for new foundations for large-scale Human-ICT organisms ("superorganisms") and their adaptive behaviors, also including lessons form applied psychology, sociology, and social anthropology, other than from systemic biology, ecology and complexity science.Self-organization and adaptation as ways of harnessing the dynamics and complexity of modern, networked, environment-aware ICT have become central research topics leading to a number of concepts such as autonomic, organic or elastic computing.Existing work on collective adaptive systems is another example considering features such as self-similarity, complexity, emergence, self-organization, and recent advances in the study of collective intelligence.Collective awareness is related to the notion of resilience, which means the systemic capability to anticipate, absorb, adapt to, and/ or rapidly recover from a potentially disruptive event.Resilience has been subject to a number of studies in complex networks and social-ecological systems.By creating algorithms and computer systems that are modeled based on social principles, socioinspired will find better ways of tackling complexity, while experimenting with these algorithms may generate new insights into social systems.In computer science and related subjects, people have started to explore socially inspired systems, e.g. in P2P networks, in robotics, in neuroscience, and in the area of agent systems.Despite this initial work, the overall field of socio-inspired system is still in an early stage of development, and it will be one of future research goals to demonstrate the great potential of social principles for operating large-scale complex ICT systems.

Ethics and Value Sensitive Design
ICT has become more sensitive to its environment: to users, to organizational and social context, and to society at large.While ICT has largely been the outcome of a technology-push focused on core computational functionality in the previous century in the first place, it later extended to the users needs, usability, and even social and psychological and organizational context of computer use.Nowadays we are approaching ICT developments, where the needs of human users, ethics, systems of value and moral norm, the values of citizens, and the big societal questions are in part driving research and development.The idea of making social and moral values central to the matrices, identity management tools) first originated in Computer Science at Stanford in the seventies.This approach is now referred to as 'Value Sensitive Design' or 'Values in Design'.Among the most prevalent, ubiquitously recognized, and meanwhile also socially pressing research agenda items relate to the concerns humans might have in using and trusting ICT.Well beyond the privacy and trust related research we see already today, the claim goes towards ethics and human value (privacy, respect, dignity, trust) sensitivity already at the design stage of modern ICT (value sensitive design), on how to integrate human value building processes into ICT, and how to cope with ignorance, disrespectful, offending and violating human values.

Augmenting Human Perception and Cognition
To escape the space and time boundaries of human perception (and cognition) via ICT (sensors, actuators) has been, and continues to be among the major HCC research challenges.Argued with the "Total Recall" prospect, the focused quests concern the richness of human experience, which results not only from the pure sensory impressions perceived via human receptors, but mostly from the process of identifying, interpreting, correlating and attaching "meaning" to those sensory impressions.This challenge is even posed to be prevalent over the next 50 years of HCC research.Dating back to the "As we may think" idea (Bush & Think, 1945), claiming ICT to support, augment or even replace intellectual work, new considerations for approaching the issue are spawned by the technological, as well as cognitive modelling advance in the context of brain computer interfaces.Understanding the operational principles (chemo-physical), but much more than that the foundational principles of the human brain at the convergence of ICT and biology poses a 21st century research challenge.The ability to build revolutionary new ICT as a "brain-like" technology, and at the same time the confluence of ICT with the biological brain mobilizes great science not only in Europe (see e.g.FTE flagship initiative HBP), but in neuroscience, medical and computing research worldwide.

Empathy and Emotion
Humans are emotional, and at the same time empathic beings.Emotion and empathy are not only expressed (and delivered via a subtle channel) when humans interact with humans, but also when humans interact with machines.The ability to capture and correlate emotional expressions by machines (ICT), as well as the ability of machines (ICT) to express emotions and empathic behavior themselves is considered a grand challenge of human computer confluence, while at the same time realism is expressed on the potential success ever being possible towards it.

Experience and Sharing
With novel ICT, particularly advanced communication systems and ubiquitous networking, radically new styles of human-to-human communication -taking into account body movements, expressions, physiological and brain signals-appear technologically feasible.With cognitive models about individuals, and digital representations of their engagements and experiences abstracted and aggregated from a combination of stimuli (e.g.visual, olfactory, acoustic, tactile, and neuro-stimulation), a new form of exchanging of these experiences and "thoughts" seem possible.Novel communication experiences which are subtle, i.e. unobtrusive, multisensory, and open for interpretation could build on expressions and indications of thoughts as captured and related to a cognitive model on the sending side, encoded and transmitted to a recipient through advanced communication media, and finally translated into multimodal stimuli to be exposed to the receiving individual, and by that induce mental and emotional states representing the "experience" of the sender.

Disappearing Interfaces
Seemingly in analogy to what was addressed at the turn of the century as the ICT research challenge "The Disappearing Computer" (EU FP4, FP5), articulated as (i) the physical disappearance of ICT, observed as the miniaturization of devices and their integration in other everyday artefacts as, e.g., in appliances, tools, clothing, etc. and (ii) mental disappearance, referring to the situation that artefacts with large physical appearance may not be perceived as computers because people discern them as (ICT also mentally move into the background) -appears to have come to a revival as far as notions of interfaces are concerned.As of today we observe that modern ICT with explicit user input and output is becoming to be replaced by a computing landscape sensing the physical world via a huge variety of sensors, and controlling it via a plethora of actuators.Since the nature and appearance of computing devices has widely changed to be hidden in the fabric of everyday life, invisibly networked, and omnipresent, "everything" has turned to become the interface: things and objects of everyday use, the human body, the human brain, the environment, or even the whole planet.Systems that happen to be perceived, understood and used explicitly and intently as the interface tend to disappear.Implicit interaction replaces explicit interaction.

Conclusion
Human computer confluence is a research area with the union of human brains and computers at its center.Its main goal is to develop the science and technologies necessary to ensure an effective, even transparent, bidirectional communication between humans and computers, which will in turn deliver a huge set of applications: from new senses, to new perceptive capabilities dealing with more abstract information spaces to the social impact of such communication enabling technologies.Inevitably, these technologies question the notion of interface between the human and the technological realm, and thus, also in a fundamental way, put into question the nature of both.The long-term implications can be profound and need to be considered from an ethical/societal point of view.Clearly, this is just a preliminary, yet evidenced classification of HCC research from the solicitation process so far.As this research roadmap aims to evolve continuously, some next steps of consolidation may possibly rephrase and reshape this agenda, as new considerations, arguments and assessments emerge.