Adapting visualizations and interfaces to the user

: Adaptive visualization and interfaces pervade our everyday tasks to improve interaction from the point of view of user performance and experience. This approach allows using several user inputs, whether physiological, behavioral, qualitative, or multimodal combinations, to enhance the interaction. Due to the multitude of approaches, we outline the current research trends of inputs used to adapt visualizations and user interfaces. Moreover, we discuss methodological approaches used in mixed reality, physiological computing, visual analytics, and proficiency-aware systems. With this work, we provide an overview of the current research in adaptive systems.


Introduction
The first stage of implementing adaptive interaction systems, according to the definition, is to define the desired "goal."In a system (a set of connected variables), such a goal is a state or multiple states (specified values for those variables at a certain point in time) that are chosen above others [10].In order to achieve this, such systems require developing a user model that represents preferences, capacities, and affective processes and their relationship with the task at hand.This cybernetic process requires two components: a sensing component that detects the current state of the system and an actuation component that guides the system toward the desired goal depending on the sensing component.
The user provides multimodal information, both explicit and implicit, that can drive an interface or visualization adaptation toward a (shared) goal.To date, adaptive systems have mainly exploited direct user input as interaction modalities: the computer reacts only to explicit commands provided by the user, e. g., mouse, keyboard, speech, touch.However, recent approaches are increasingly considering implicit aspects of the user, such as their cognitive processing capabilities [32] and the user's physiological state [17].For example, in the field of visualization, a personalized adaptation based on various cognitive functions (such as perceptual speed and working memory) impacts the user's performance [59] and different modalities of information processing require different visualizations [58].This complements traditional approaches of considering implicit user data based on their profile, e. g., interests or prior knowledge.
Monitoring the user's physiological state to infer and adapt interaction will couple them to the user's goals and thus, enable developers to design biocybernetic loops.The user's physiological data are processed online and classified to trigger the adaptive system, which is then in charge of performing adaptive actions in the interface [17].Fairclough [16] proposed first and second-order adaptation loops.The first-order adaptation consists of a loop that begins with monitoring the user's condition.The loop completes by executing adaptive actions.This first-order adaptation necessitates a set of rules that link each user's current condition to at least one adaptive action.The secondorder adaptation encompasses detecting changes as a direct result of adaptation.It allows the system to acquire information on the user's state and preferences over multiple iterations.Thus, it allows the system to adjust the ac-tions to a single user.And after a phase of reciprocal coupling, it leads to system and user co-existence.
Biocybernetic approaches do not consider individual users as static entities.Therefore, we describe such dynamic systems as continuously updating systems using incoming environment information and, thus, changing user requirements and goals [14,62].Such dynamics are critical, especially when the update of the adaptive interface is not under the explicit control of the user but depends on the characteristics of the interaction itself.Thus, both the adaptive interface and the user learn from each other.Mutual adaptation dynamics can lead to complex interaction patterns, affecting the adaptive system's usability.A paradigmatic example consists of adaptive interfaces designed to improve the user's performance by automatically reducing the error associated with a given task, e. g., an adaptive touch keyboard [19].If an interface fails to incorporate the user's learning capability, the performance of the joint system will likely be even worse than that of a user in isolation.In such cases, the contribution of the adaptive interface may result in error overcorrection and hence in a potentially unstable interaction [4].In contrast, the system's features will change according to the characteristics of the user, for example, by providing only partial correction and taking into account the learning rate associated with the user over the entire course of the interaction.In the future, such a joint adaptation approach can positively improve the outcome of the interaction and the user's subjective experience.The next generation of intelligent systems will encompass increased autonomy and adaptability [26] facilitating proactive and implicit interaction with users [1].
This work provides an overview of applications in adaptive content, interaction, and visualization.We specifically address which information researchers used for adaptation, with which purpose of using such information, and lastly, which domains are feasible for adaptive systems in the human-computer interaction (HCI) and the visualization domains.

Technologies for adaptation
Systems nowadays may draw from various information to infer the user's current state and environment.Such information ranges from static adaptation using user profiles to context-aware systems [12,53], using the user's surroundings, and even ubiquitous sensing technologies.The last category potentially provides deep insights into a person's cognitive ability or motoric skills.In the following, we provide a short overview regarding currently common physiological measures leveraged in adaptive interfaces.

Extracting features for adaptation
Among the psychophysiological measures useful for adaptive interfaces, electroencephalography (EEG), eye tracking, electrodermal activity (EDA), and electromyography (EMG) have garnered the most interest.Their sensors' comparatively small size and ability to measure physiological activity non-invasively make them more likely to be incorporated into wearable consumer devices, such as glasses, wristwatches, and headbands.
EEG records electric potentials from the scalp, which reflects brain activity.Machine learning (ML) can extract event-related activity to estimate cognitive workload [33], attention allocation [60], or affective states [40].Brainmachine interfaces [8] and user state estimation systems use these ML-generated estimates.However, such systems have to be considered in light of current challenges such as the need for generalizable applications of classification methods online [41], improvement of transfer learning, and application of new approaches such as deep learning or Riemannian geometry-based classifiers [36].
Similarly, gaze behavior can indicate high-level cognitive processes, see early work by Deubel [11] and Hoffmann [25].Recent work analyzed specific eye movements and gaze patterns to infer, for instance, user activities and cognitive states.Jacob and Karn [29] and Duchowski [13] provide in-depth overviews of this domain.
When the goal is to infer responses to novel stimuli, cognitive workload, and stress, the choice of EDA measures might be preferable as a noninvasive and easy-to-use method.EDA measures are the joined pattern of its phasic and tonic components [21].Phasic Skin Conductance Responses (SCRs) reflect discrete and stimulus-specific responses to evaluate the novelty, importance, and intensity of the stimuli utilized [44].As indexed via Skin Conductance Level (SCL), tonic activity is an inertial and slow response particularly well suited to evaluate the effect of continuous stimuli, i. e., task.Therefore, HCI has used it to quantify, for instance, changes in arousal under high cognitive load [34] or stress [6].
Besides inferring cognitive processes, measuring the user's motoric responses can be especially useful in enabling a system to adapt to the user's abilities and potential actions [39].By measuring muscular activity, EMG can provide insights into the working mechanism of motor tasks.Using EMG measures for adaptation allows for providing user-tailored feedback, ranging from detecting emotional states through facial EMG [61], over gesture recognition [52], to an adaptive tutoring system for motor tasks [30].
Finally, sensor fusion multi-model adaptive systems can often achieve more robust adaptation.For example, Putze et al. [48] showed that combining EEG recordings with eye-tracking addresses the Midas-Touch problem in gaze-based selection by estimating whether a fixation was purposeful or not.Moreover, combining EMG with EEG, Haufe et al. [23] showed that this leads to faster automatic braking in a driving simulator than using EEG alone.

Adapting the interface and visualization
Although there have been earlier models for adaptive systems [22,54], we consider three critical adaptation elements: content, presentation, and interaction.When adaptive systems adjust their content, which relates to users' preferences and engagement, they must consider the user's prior knowledge and interest.Such dimension might involve notification design and recommendations, especially considering the exploratory visual analytics process [51].
Secondly, presentation adaptation affects user interfaces (UIs) or visualizations according to users' spare perceptual capacity, discomfort and, stress level by simplifying displayed information, luminance, or other properties.
Thirdly, interaction adaptation is a broader field as it might encompass different paradigms.For example, in multitasking environments, users might experience tasks being switched off [47], see the number of options change in a decision-making task [46], or modify the interaction modality, i. e., from gesture to hand-free interaction.

Use cases and applications
Here, we provide a brief overview of adaptive visualization and interfaces with use cases and applications from our work, specifically targeting content-based adaptation from physiological data, adaptation of visualization presentation from physiological data, and interaction adaptation.

Content adaptation
In the following, we present and discuss systems based on eye-tracking features, that adapt to support language proficiency, increase recommender systems' performance based on inferred users' interest, or help visual analytics.

Adaptive displays based on language proficiency
Globalization means that interfaces are prevalent in a multitude of different languages.Hard-to-access language correction can lead to user aversion.Consequently, there is merit in creating systems capable of estimating a user's language proficiency and displaying content appropriate to the user's abilities.
Recently, Karolus et al. [31] explored the potential of using a user's gaze properties to detect whether the information is presented in a language the user understands (see Figure 1).Robustness and feasibility with lowgrade eye-tracking equipment were important aspects of this work.They proposed technical specifications for the recording equipment and the interaction period using robust gaze features, including fixation and blink duration.They found that a few seconds of recorded gaze data is sufficient to determine if a user can speak the displayed language.

Gaze as input for recommender systems
Silva et al. [56] sketched the possibility of back-propagating eye-gaze through the visualization pipeline and mapping it onto the underlying data.According to the eyemind hypothesis, this viewed data is of interest to the user.Recommender systems, such as the one in Figure 2, can take advantage of such implicitly selected data to suggest helpful visualizations [50].Recommendations based on such data fit the user's current interest and might, by extension, also fit their current task.However, a robust inference of an explicit task is not trivial, but a recommender system based on data interest can suggest the correct views for any generic and unidentified task.

Eye tracking support in visual analytics systems
Visual analytics is a design framework for interactive visual displays to facilitate the exploration of, and insight into, data sets.They rely on a loop that includes the viewer with all their prior knowledge, interests, and tasks.This allows the user to alter the selection of data, adjust parameters for data processing, and adapt the visualization onthe-fly to cater to current needs.
With the added information from eye-trackers, such visual analytics systems can augment existing interaction techniques [56].This can include, for instance, gaze as additional cursors for interaction through speech or disambiguation of targets when pushing buttons on handheld controllers.In addition, with the advent of coarse eyetracking for devices with front-mounted cameras (e. g., tablets and phones), existing visual analytics software can "retro-fit" gaze data without changing the actual hardware.For example, law enforcement agents already use software on car-mounted tablets to provide them with overviews of occurrences in their districts.In such a scenario, even coarse gaze data can check whether relevant events have been overlooked and provide adaptive visualizations to attract the agent's attention.

Presentation adaptation
In this section, we highlight the work of adaptive systems that adapts presentation based on users' physiological input, such as EDA, to support user experience, or to support processing of relevant information, i. e., notifications.

Adaptation of virtual reality visual complexity based on physiological arousal
Virtual reality (VR) is rapidly gaining popularity for social or collaborative virtual environment applications.Such settings envision the involvement of realistic Non-Player Characters (NPCs), such as virtual crowds with human-like behavior.However, highly dynamic environments could provide task-irrelevant elements that negatively increase a user's cognitive load and distractibility.Thus, monitoring users' physiological activity and adapting the interaction is an emerging research trend to optimize user experience or performance.The goal of physiological control loops is to detect deviations from the optimal physiological state that influence the adaptation of the features of the environment or tasks to drive users towards a more desirable state.Here, Chiossi et al. [9] focused on a peripheral measure of physiological arousal, i. e., EDA.Physiological arousal correlates with task demands and engagement in a multicomponent task [18] and can be affected by proxemics of NPCs both in VR [35] and augmented reality (AR) [27].Hence, the stream of NPCs was adapted in response to changing EDA levels of users while being engaged in a dual-task setting.They processed the EDA data only using an average moving window of 20 sec.For user-dependent adaptation, the adaptive algorithm adjusts the visual complexity to a baseline slope recording recorded at the beginning of the experiment.Thus, when the EDA slope was larger than the baseline slope, 2 NPCs were removed, indicating increased arousal.On the contrary, 4 NPCs are added to the environment if the system detected decreased arousal.Figure 3 visualizes the adaptation algo-  [42] collected perceivability and behavioral data on realistically looking synthesized desktop images.They used this data to identify the factors that impact the noticeability of notifications.This allowed them to develop a computational model of noticeability that can predict noticeability maps for a given desktop image and user attention focus.These maps visualize the locations at which a notification is likely to be missed (red) or likely to be seen (green).
rithm.Thus, they supported the user experience by leveraging visual complexity, i. e., increased system acceptance, competence, and immersion.
On the surface, the findings of Chiossi et al. [9] only impact the design of social VR scenarios.However, such results can generalize to the design of information visualization patterns to help users in mixed reality (MR) [57].Thus, physiologically-aware systems can potentially personalize environments for the user, improving the user experience.Furthermore, in the visualization domain, VR complexity can burden users; thus, a physiological-adaptive system varying the information load according to users' workload could foster the viability and usefulness of applications.

Adapting notifications to visual appearance and human perception
Users benefit from desktop notifications showing them their incoming messages, upcoming calendar events, or other important information.Notifications need to attract and divert attention from a primary task effectively to ensure that users notice important information.At the same time, notifications are embedded into the visual design of the user interface and are subject to aesthetic considerations.However, design decisions that are also currently static, i. e., do not adapt at runtime, can severely impair the user's ability to perceive notifications.Müller et al. [42] presented a software tool to automatically synthesize realistically looking desktop images for major operating systems and applications.These im-ages allowed them to systematically study the noticeability of notifications during a realistic interaction task.They found that the visual importance of the background at the notification location significantly impacts whether users detect notifications.Their work also introduced the idea of noticeability maps: 2D maps encoding the predicted noticeability across the desktop.The maps inform designers how to trade-of notification design and noticeability.In the future, such automatically predicted noticeability maps could be used in UI design and during runtime to adapt the appearance and placement of desktop notifications to the predicted user noticeability.

Interaction adaptation
In the last section, we present relevant work that shows how adaptive systems can support interaction for midair or multimodal interactions in immersive MR environments.

Adapting the 3D user interfaces for improved ergonomics
Interactive MR applications surround the user with virtual content that can be manipulated directly by reaching for it with the tracked hand or controllers.Such mid-air interaction techniques are beneficial, as they feel natural, but they may lead to physical strain, muscle fatigue, and challenging postures [5,37].The XRgonomics toolkit [15] addresses these issues by visualizing the ergonomics of The XRgonomics toolkit [15] visualizes the cost of interaction for each reachable point in the user's interaction space, through color coding (K) from blue (most comfortable) to red (least comfortable) (L).The applied metric is selected in a dropdown menu (A), and the computed value can be adapted for the user's arm dimensions (C).For a better visibility, the voxel size can be adapted (B), and the range of values to visualize can be limited along all three axes (E-G) to show only individual regions or slices of the space.Further, the user can retrieve the "optimal" voxel with the lowest ergonomic cost (D).Finally, the visualization of the avatar can be deactivated (H), and three sliders enable control of the perspective (I).
the user's interaction space (see Figure 5), allowing UI designers to create interfaces that are convenient and easy to manipulate.Further, it supports the automatic adaptation of UIs so that interactive elements remain within easy reach while the user moves about in a changing physical environment.The ergonomics metrics currently supported in XRgonomics are RULA [38], Consumed Endurance [24], and muscle activation [3].
Prior research has explored ergonomics [3,24,38] and while the resulting metrics help evaluate existing UIs, it is difficult to use them for generating novel UI layouts.Further, the formulated design recommendations can be challenging to interpret and apply, particularly if the ideal interaction space is unavailable, e. g., due to the user's physical environment.
To address this, Belo et al. [15] present a toolkit to visualize the interaction cost in the user's entire interaction space by computing ergonomics metrics for each reachable point in space.Their work shows a half-sphere of voxels around the user, color-coded to reveal the ergonomics of reaching for that position.Thus, the toolkit allows UI designers to inspect the interaction space and identify ideal placements for various interactive elements.The toolkit further allows the definition of constraints, e. g., allowing the designer to define areas of the interaction space that are not available for placement of interactive virtual content, for example, due to physical obstacles in the user's environment.Based on that, the toolkit can recommend the ideal position with the best ergonomic properties for reaching with the hand.As this computation is feasible in real-time, the toolkit API can be used for dynamic adaptation of UIs, depending on the user's changing physical environment or varying visible space.For example, consider a UI element that should always remain within the user's field of view in an AR scenario.The user is wearing a head-mounted display (HMD), and as they turn their head to look around, using the view frustum of the HMD, constraints arise in the available interaction space.The toolkit automatically computes the most ergonomic placement for the respective UI element within this available volume, keeping it in easy reach for the user.
Beyond improving the ergonomics of mid-air interaction, this approach may be applied to achieve the opposite goal of increasing physical effort to reach a UI element or virtual object, e. g., with the aim to train particular muscles.This may contribute to rehabilitation or be applied in exergame scenarios, as proposed by [43].

Hybrid user interfaces for augmented reality
The complexity of interaction in AR environments provides many opportunities for adaptation, such as adapting visualizations based on the user's physical surroundings (e. g., Shin et al. [55]), within situated analytics (e. g., Fleck et al. [20]), or by considering the devices available in the user's workspace (e. g., STREAM [28]).One possibility of adapting visualizations and interfaces to the user can be realized through hybrid user interfaces that combine the advantages of heterogeneous devices (e. g., head-mounted AR devices and handheld tablets), creating the ability to facilitate multiple coordinated views across different realities for visual analytics.
For example, STREAM [28] combines an immersive AR environment using an AR headset with a spatially-aware tablet for interacting with 3D parallel coordinates visualizations, consisting of linked 2D scatter plots.Here, the AR headset allows users to see the visualization in stereoscopic 3D space.At the same time, the tablet provides familiar touch interaction on individual 2D components of the visualization, e. g., 2D scatter plots, see Figure 6.Furthermore, to reduce the cognitive demand when switching between both interfaces, STREAM automatically adapts the representation of both interfaces to the user's implicit interaction by tracking the tablet's position in space: Once a user holds their tablet in front of them (i.e., indicating that the user wants to switch between devices), the selected 2D component of the visualization in AR (e. g., a 2D scatter plot) rotates toward the user's viewing direction, while the tablet adapts its content to show the same 2D component on screen-effectively merging both visual spaces into one interaction space.This adaptation allows users to seamlessly switch between AR and tablet visualization without losing context.

Outlook
Adaptive interfaces play a crucial role in developing new interaction paradigms, especially when improving performance or user experience.With this overview, we introduced how adaptive interfaces might leverage various user inputs for improved performance and UX, easier learning, and improved information engagement.Our overview presents current adaptive contents, interactions, and presentations applications.It serves as an initial design space to showcase how current systems use inputs for adaptation and hints at how future systems might adapt to the users' actions.For adaptation purposes, current work focuses on the usage of physiological input for either content or presentation, motion tracking data and ergonomic metrics for interaction.
We highlight that in the future, especially with sensor fusion, multi-model adaptive systems are important to consider as they have a high chance of capturing the full context for the user.Therefore, such systems have higher success potential.Future work will aim to formalize interaction paradigms that can generalize across application domains and classify which combinations of inputs might be more suitable for multi-model adaptive systems.
With this goal in mind, it is first necessary to clarify that the basis for adaptation accurately represents and relates to users' states, e. g., physical discomfort and focused attention.This relationship is especially troublesome for physiological measurements and their construct validity [7].However, we cannot claim a one-to-one explanatory relationship with the construct of interest.Therefore, the amount of diagnostic accuracy necessary for adaptive systems will inevitably vary between systems until an acceptable cost-benefit ratio is achieved [17].
Second, current systems focus on a single user; however, in the future, we envision that dynamically adapting to multi-user scenarios will improve collaboration [2].It is unclear if the results from the single-user investigation will hold reliably in multi-user scenarios, e. g., using the user's EDA for adaptation.
Third, data privacy is a central concern for users of adaptive systems that rely on input from users who do not control most of their physiological activity, i. e., physiological computing systems.Adaptive systems present significant potential for asymmetry in data protection [49], i. e., the system may not disclose to the user where his or her data are stored or who has access to this data.Moreover, adaptive control loops aim to manipulate users' states toward a positive goal.Here, the debate is not over on the adaptation's direction but on who keeps control over the adaptive process [45].These considerations bolster claims that mutual accountability [45] and giving users authority over the system are fundamental conditions that necessitate future work.

Figure 2 :
Figure 2: Screenshot of a recommender system with eye-tracking support [50].Gaze is used as indicator for interest and mapped to the underlying data.

Figure 3 :
Figure 3: Flowchart of the physiological loop used by Chiossi et al. [9].The visual complexity (in the form of NPCs) adapts according to changes in the EDA calibrated from a baseline recording (Δb).The adaptation function is called every 20 secs.

Figure 4 :
Figure 4: Müller et al.[42] collected perceivability and behavioral data on realistically looking synthesized desktop images.They used this data to identify the factors that impact the noticeability of notifications.This allowed them to develop a computational model of noticeability that can predict noticeability maps for a given desktop image and user attention focus.These maps visualize the locations at which a notification is likely to be missed (red) or likely to be seen (green).

Figure 5 :
Figure 5:The XRgonomics toolkit[15] visualizes the cost of interaction for each reachable point in the user's interaction space, through color coding (K) from blue (most comfortable) to red (least comfortable) (L).The applied metric is selected in a dropdown menu (A), and the computed value can be adapted for the user's arm dimensions (C).For a better visibility, the voxel size can be adapted (B), and the range of values to visualize can be limited along all three axes (E-G) to show only individual regions or slices of the space.Further, the user can retrieve the "optimal" voxel with the lowest ergonomic cost (D).Finally, the visualization of the avatar can be deactivated (H), and three sliders enable control of the perspective (I).

Figure 6 :
Figure6: STREAM[28] combines spatially-aware tablets with head-mounted AR displays for visual data analysis using a 3D parallel coordinates visualization.STREAM's adaptation mechanisms allow users to seamlessly switch between the AR visualization and the tablet visualization without losing context.For example, the user on the right holds their tablet vertically, allowing STREAM to adapt their AR scatter plot with the tablet's visual space.In contrast, the collaborator's (left user) AR visualization is unaffected.