Human–AI interaction and art

Perceiving human motion and intent at long range is a prerequisite for socially intelligent aerial robots, yet the data to learn it barely exists. We introduce Drones2BodyLanguage, a dataset grounding human motion in real UAV footage: avatars manifesting ten communicative intents are placed into unmodified 4K drone scenes with metrically correct position, scale and orientation, maintained over hundreds of frames of camera motion. Enabling it is a lightweight geometric world model of the local scene - semantically selected anchors lifted to 3D through streaming monocular depth - in which a placement point is predicted as an affine anchor combination with provably rigid-invariant weights, and re-rendered under an SVD-fitted ground rotation. Across twelve architectures on scene- and motion-disjoint splits, training on placed data lifts mean intent accuracy by a wide margin for real, retargeted and generated motion alike, with gains confirmed on two in-the-wild scenes.

Body movement communicates intent at distances and in conditions where neither the face, nor speech can be captured. We study the recognition of communicative intent from 2D body pose alone. We argue that body motion is a reliable signal especially in scenarios that require real time low-cost on-device person-to-robot communication in long distance environments, such as rescue missions. However, existing resources do not isolate this signal. Affective corpora combine body, face, voice and text, while skeleton action-recognition benchmarks label the action performed rather than the message conveyed. We release a dataset of real frames of full-body pose covering ten communicative intents and we compare it against other real (IPC) and synthetic (MotionLCM, VEO3.1, Kimodo) ones that span a range of difficulty. We target systems that can run on a robot’s limited onboard hardware. We benchmark multiple models, from skeleton graph classifiers to joint motion-forecasting networks, and report performance metrics together with frame rate on an embedded GPU (NVIDIA Orin~Nano), since speed matters as much as accuracy in our scenario. Finally, we show that a model’s own autoregressive self-consistency works as an unsupervised reliability signal. We give a short proof that bounds the probability that a self-consistent prediction is correct, show that this probability grows with the number of consistent steps, and identify the conditions under which a confident prediction can still be false, benchmarked against industry-standard metrics.

We study the ongoing debate regarding the statistical fidelity of AI-generated data compared to human-generated data in the context of non-verbal communication using full body motion. Concretely, we ask if contemporary generative models move beyond surface mimicry to participate in the silent, but expressive dialogue of body language. We tackle this question by introducing the first framework that generates a natural non-verbal interaction between Human and AI in real-time from 2D body keypoints. Our experiments utilize four lightweight architectures which run at up to 100 FPS on an NVIDIA Orin Nano, effectively closing the perception-action loop needed for natural Human-AI interaction. We trained on 437 human video clips and demonstrated that pretraining on synthetically-generated sequences reduces motion errors significantly, without sacrificing speed. Yet, a measurable reality gap persists. When the best model is evaluated on keypoints extracted from cutting-edge text-to-video systems, such as SORA and VEO, we observe that performance drops on SORA-generated clips. However, it degrades far less on VEO, suggesting that temporal coherence, not image fidelity, drives real-world performance. Our results demonstrate that statistically distinguishable differences persist between Human and AI motion.

Modeling face-to-face communication in computer vision, which focuses on recognizing and analyzing nonverbal cues and behaviors during interactions, serves as the foundation for our proposed alternative to text-based Human-AI interaction. By leveraging nonverbal visual communication, through facial expressions, head and body movements, we aim to enhance engagement and capture the user’s attention through a novel improvisational element, that goes beyond mirroring gestures. Our goal is to track and analyze facial expressions, and other nonverbal cues in real-time, and use this information to build models that can predict and understand human behavior. Operating in real-time and requiring minimal computational resources, our approach signifies a major leap forward in making AI interactions more natural and accessible. We offer three different complementary approaches, based on retrieval, statistical, and deep learning techniques. A key novelty of our work is the integration of an artistic component atop an efficient human-computer interaction system, using art as a medium to transmit emotions. Our approach is not art-specific and can be adapted to various paintings, animations, and avatars. In our experiments, we compare state-of-the-art diffusion models as mediums for emotion translation in 2D, and our 3D avatar, Maia, that we introduce in this work, with not just facial movements but also body motions for a more natural and engaging experience. We demonstrate the effectiveness of our approach in translating AI-generated emotions into human-relatable expressions, through both human and automatic evaluation procedures, highlighting its potential to significantly enhance the naturalness and engagement of Human-AI interactions across various applications.

Phobia is an anxiety disorder that affects 13% of the world’s population and it is manifested through an extreme and irrational fear toward objects or situations. A lot of research has been done on studying the contributing factors to the onset, development, and maintenance of phobias, underlying cognitive and behavioral processes, physical manifestation, and treatment methods. In this chapter, we present a new automated approach to phobias therapy, based on the integration of virtual reality technology, artificial intelligence, and affective computing—the ability of computational machines to recognize, adapt, and respond intelligently to human emotions. First, we introduce the results obtained by applying various machine learning algorithms for classifying the six basic emotions—anger, disgust, fear, joy, sadness, and surprise—based on the physiological recordings and self-reported ratings from the DEAP database. Then we present the stages of development and evaluation of a virtual environment for treating acrophobia that relies on gradual exposure to stimuli, accompanied by physiological signals monitoring. In a pilot experiment, we proposed a model for automatically adapting exposure scenarios according to the users’ emotional states, which has been tested on four acrophobic subjects. Here, the results obtained pass through a critical analysis in which we discuss the progress of the research and current limitations. Finally, we introduce the conceptual design of a system for phobias treatment that can be used as a therapy tool in clinical setups and also for home training, integrating virtual reality, machine learning, gamification techniques, biophysical data recording, emotion recognition, and automatic scenario adaptation. The system uses an intelligent virtual therapist that can understand emotions based on physiological signals, provides encouragement, and changes the level of exposure according to the user’s affective states.

In this paper, we investigate various machine learning classifiers used in our Virtual Reality (VR) system for treating acrophobia. The system automatically estimates fear level based on multimodal sensory data and a self-reported emotion assessment. There are two modalities of expressing fear ratings: the 2-choice scale, where 0 represents relaxation and 1 stands for fear; and the 4-choice scale, with the following correspondence: 0-relaxation, 1-low fear, 2-medium fear and 3-high fear. A set of features was extracted from the sensory signals using various metrics that quantify brain (electroencephalogram-EEG) and physiological linear and non-linear dynamics (Heart Rate-HR and Galvanic Skin Response-GSR). The novelty consists in the automatic adaptation of exposure scenario according to the subject’s affective state. We acquired data from acrophobic subjects who had undergone an in vivo pre-therapy exposure session, followed by a Virtual Reality therapy and an in vivo evaluation procedure. Various machine and deep learning classifiers were implemented and tested, with and without feature selection, in both a user-dependent and user-independent fashion. The results showed a very high cross-validation accuracy on the training set and good test accuracies, ranging from 42.5% to 89.5%. The most important features of fear level classification were GSR, HR and the values of the EEG in the beta frequency range. For determining the next exposure scenario, a dominant role was played by the target fear level, a parameter computed by taking into account the patient’s estimated fear level.

Summary (the publisher’s abstract is not openly available): Within a holonic, agent-based architecture for virtual-reality phobia therapy, this work classifies users’ fear levels from physiological signals (EEG, galvanic skin response, heart rate) recorded during VR exposure, comparing several machine-learning and deep-learning classifiers in order to automatically adapt the exposure intensity to the patient’s current affective state.

Emotions constitute an indispensable component of our everyday life. They consist of conscious mental reactions towards objects or situations and are associated with various physiological, behavioral, and cognitive changes. In this paper, we propose a comparative analysis between different machine learning and deep learning techniques, with and without feature selection, for binarily classifying the six basic emotions, namely anger, disgust, fear, joy, sadness, and surprise, into two symmetrical categorical classes (emotion and no emotion), using the physiological recordings and subjective ratings of valence, arousal, and dominance from the DEAP (Dataset for Emotion Analysis using EEG, Physiological and Video Signals) database. The results showed that the maximum classification accuracies for each emotion were: anger: 98.02%, joy:100%, surprise: 96%, disgust: 95%, fear: 90.75%, and sadness: 90.08%. In the case of four emotions (anger, disgust, fear, and sadness), the classification accuracies were higher without feature selection. Our approach to emotion classification has future applicability in the field of affective computing, which includes all the methods used for the automatic assessment of emotions and their applications in healthcare, education, marketing, website personalization, recommender systems, video games, and social media.

Summary (the publisher’s abstract is not openly available): eTher is an assistive virtual agent for acrophobia therapy in a VR mountain landscape with a cable-car ride. The agent talks to the user, adapts its voice pitch, tempo and volume to the patient’s emotional state estimated from biophysical signals relative to a resting baseline, and automatically lowers the exposure level when relaxation techniques do not work. A pilot study with 10 users showed faster task completion and significantly improved relaxation when assisted by the agent.

This paper proposes a protocol for the acquisition and processing of biophysical signals in virtual reality applications, particularly in phobia therapy experiments. This protocol aims to ensure that the measurement and processing phases are performed effectively, to obtain clean data that can be used to estimate the users’ anxiety levels. The protocol has been designed after analyzing the experimental data of seven subjects who have been exposed to heights in a virtual reality environment. The subjects’ level of anxiety has been estimated based on the real-time evaluation of a nonlinear function that has as parameters various features extracted from the biophysical signals. The highest classification accuracy was obtained using a combination of seven heart rate and electrodermal activity features in the time domain and frequency domain.

This paper proposes a real-time Virtual Reality game for treating acrophobia that automatically tailors in-game exposure to heights to the players’ individual characteristics – affective state and physiological features. The elements of novelty are the automatic estimation of fear and the prediction of the next game level based on the electroencephalogram (EEG) and biophysical data – Galvanic Skin Response (GSR) and Heart Rate (HR). Two neural networks have been trained with the data recorded in an experiment where 4 subjects have been in-vivo and virtually exposed to various heights. In order to test the validity of the approach, the same users played the acrophobia game, using two modalities of ex-pressing fear level. After completing a game level, the EEG and biophysical data were averaged and one neural network estimated the current fear score, while the other predicted the next game level. A measure of similarity between the self-estimated fear level during a game epoch and the fear level predicted by the first neural network showed an accuracy rate of 73% and 42% respectively for the two modalities of expressing fear level. 3 out of 4 users succeeded to obtain a fear level of 0 (complete relaxation) in the final game epoch.

The field of affective computing has significantly emerged over the recent years, either we talk about systems that recognize and interpret human emotions or applications with assistive or therapeutic purposes that aim to treat various psychological disorders. Our goal is to design and develop a system that would aid people suffering of a common anxiety disorder of our times - the phobia. Thus, we propose a system for treating acrophobia - fear of heights - that exposes the patients in a Virtual Reality scenario depicting a mountain landscape, while their physiological signals (electrodermal activity, heart rate and respiration rate) are continuously recorded. However, during exposure, certain artefacts can appear, caused by head or hand movements, controller or mouse clicks, body rotations - and thus affect the quality of the recording. As artefact identification is a fundamental step in obtaining clean data, in this paper we present a method of recognizing artefact patterns in the signals recorded during gameplay sessions, which consists of aligning the data segments corresponding to the moments when the users performed head/hand/body movements with the artefact signals recorded during a reference procedure. The results obtained in pilot experiment with 5 users showed that the bias is lower on the perfectly aligned segments than on the segments located a few steps before and after the moment when the user introduced the artefact in the game signal, but without statistical significance. In addition, we propose a new method of estimating respiration rate based on the distance between two HTC Vive trackers placed on an elastic band, on the chest and on the upper back torso of the user.

This paper presents a human-centered methodology for designing and developing Virtual Reality Exposure Therapy (VRET) systems. By following the steps proposed by the methodology – Users analysis, Domain Analysis, Task Analysis and Representational Analysis, we developed a system for acrophobia therapy composed of 9 functional, interrelated modules which are responsible for patients, scenes, audio and graphics management, as well as for physiological monitoring and events triggering. The therapist visualizes in real time the patient’s biophysical signals and adapts the exposure scenario accordingly, as he can lower or increase the level of exposure. There are 3 scenes in the game, containing a ride by cable car, one by ski lift and a walk by foot in a mountain landscape. A reward system is implemented and emotion dimension ratings are collected at predefined points in the scenario. They will be stored and later used for constructing an automatic machine learning emotion recognition and exposure adaptation module.