Oana Mitruț (Bălan)

The Rashômon Effect, applied in Explainable Machine Learning, refers to the disagreement between the explanations provided by various attribution explainers and to the dissimilarity across multiple explanations generated by a particular explainer for a single instance from the dataset (differences between feature importances and their associated signs and ranks), an undesirable outcome especially in sensitive domains such as healthcare or finance. We propose a method inspired from textual-case based reasoning for aligning explanations from various explainers in order to resolve the disagreement and dissimilarity problems. We iteratively generated a number of 100 explanations for each instance from six popular datasets, using three prevalent feature attribution explainers: LIME, Anchors and SHAP (with the variations Tree SHAP and Kernel SHAP) and consequently applied a global cluster-based aggregation strategy that quantifies alignment and reveals similarities and associations between explanations. We evaluated our method by weighting the $$:k$$ -NN algorithm with agreed feature overlap explanation weights and compared it to a non-weighted $$:k$$ -NN predictor, having as task binary classification. Also, we compared the results of the weighted $$:k$$ -NN algorithm using aggregated feature overlap explanation weights to the weighted $$:k$$ -NN algorithm using weights produced by a single explanation method (either LIME, SHAP or Anchors). Our global alignment method benefited the most from a hybridization with feature importance scores (information gain), that was essential for acquiring a more accurate estimate of disagreement, for enabling explainers to reach a consensus across multiple explanations and for supporting effective model learning through improved classification performance.

Phobia is an anxiety disorder that affects 13% of the world’s population and it is manifested through an extreme and irrational fear toward objects or situations. A lot of research has been done on studying the contributing factors to the onset, development, and maintenance of phobias, underlying cognitive and behavioral processes, physical manifestation, and treatment methods. In this chapter, we present a new automated approach to phobias therapy, based on the integration of virtual reality technology, artificial intelligence, and affective computing—the ability of computational machines to recognize, adapt, and respond intelligently to human emotions. First, we introduce the results obtained by applying various machine learning algorithms for classifying the six basic emotions—anger, disgust, fear, joy, sadness, and surprise—based on the physiological recordings and self-reported ratings from the DEAP database. Then we present the stages of development and evaluation of a virtual environment for treating acrophobia that relies on gradual exposure to stimuli, accompanied by physiological signals monitoring. In a pilot experiment, we proposed a model for automatically adapting exposure scenarios according to the users’ emotional states, which has been tested on four acrophobic subjects. Here, the results obtained pass through a critical analysis in which we discuss the progress of the research and current limitations. Finally, we introduce the conceptual design of a system for phobias treatment that can be used as a therapy tool in clinical setups and also for home training, integrating virtual reality, machine learning, gamification techniques, biophysical data recording, emotion recognition, and automatic scenario adaptation. The system uses an intelligent virtual therapist that can understand emotions based on physiological signals, provides encouragement, and changes the level of exposure according to the user’s affective states.

In this paper, we investigate various machine learning classifiers used in our Virtual Reality (VR) system for treating acrophobia. The system automatically estimates fear level based on multimodal sensory data and a self-reported emotion assessment. There are two modalities of expressing fear ratings: the 2-choice scale, where 0 represents relaxation and 1 stands for fear; and the 4-choice scale, with the following correspondence: 0-relaxation, 1-low fear, 2-medium fear and 3-high fear. A set of features was extracted from the sensory signals using various metrics that quantify brain (electroencephalogram-EEG) and physiological linear and non-linear dynamics (Heart Rate-HR and Galvanic Skin Response-GSR). The novelty consists in the automatic adaptation of exposure scenario according to the subject’s affective state. We acquired data from acrophobic subjects who had undergone an in vivo pre-therapy exposure session, followed by a Virtual Reality therapy and an in vivo evaluation procedure. Various machine and deep learning classifiers were implemented and tested, with and without feature selection, in both a user-dependent and user-independent fashion. The results showed a very high cross-validation accuracy on the training set and good test accuracies, ranging from 42.5% to 89.5%. The most important features of fear level classification were GSR, HR and the values of the EEG in the beta frequency range. For determining the next exposure scenario, a dominant role was played by the target fear level, a parameter computed by taking into account the patient’s estimated fear level.

Summary (the publisher’s abstract is not openly available): Within a holonic, agent-based architecture for virtual-reality phobia therapy, this work classifies users’ fear levels from physiological signals (EEG, galvanic skin response, heart rate) recorded during VR exposure, comparing several machine-learning and deep-learning classifiers in order to automatically adapt the exposure intensity to the patient’s current affective state.

Emotions constitute an indispensable component of our everyday life. They consist of conscious mental reactions towards objects or situations and are associated with various physiological, behavioral, and cognitive changes. In this paper, we propose a comparative analysis between different machine learning and deep learning techniques, with and without feature selection, for binarily classifying the six basic emotions, namely anger, disgust, fear, joy, sadness, and surprise, into two symmetrical categorical classes (emotion and no emotion), using the physiological recordings and subjective ratings of valence, arousal, and dominance from the DEAP (Dataset for Emotion Analysis using EEG, Physiological and Video Signals) database. The results showed that the maximum classification accuracies for each emotion were: anger: 98.02%, joy:100%, surprise: 96%, disgust: 95%, fear: 90.75%, and sadness: 90.08%. In the case of four emotions (anger, disgust, fear, and sadness), the classification accuracies were higher without feature selection. Our approach to emotion classification has future applicability in the field of affective computing, which includes all the methods used for the automatic assessment of emotions and their applications in healthcare, education, marketing, website personalization, recommender systems, video games, and social media.

Summary (the publisher’s abstract is not openly available): eTher is an assistive virtual agent for acrophobia therapy in a VR mountain landscape with a cable-car ride. The agent talks to the user, adapts its voice pitch, tempo and volume to the patient’s emotional state estimated from biophysical signals relative to a resting baseline, and automatically lowers the exposure level when relaxation techniques do not work. A pilot study with 10 users showed faster task completion and significantly improved relaxation when assisted by the agent.

This paper proposes a protocol for the acquisition and processing of biophysical signals in virtual reality applications, particularly in phobia therapy experiments. This protocol aims to ensure that the measurement and processing phases are performed effectively, to obtain clean data that can be used to estimate the users’ anxiety levels. The protocol has been designed after analyzing the experimental data of seven subjects who have been exposed to heights in a virtual reality environment. The subjects’ level of anxiety has been estimated based on the real-time evaluation of a nonlinear function that has as parameters various features extracted from the biophysical signals. The highest classification accuracy was obtained using a combination of seven heart rate and electrodermal activity features in the time domain and frequency domain.

This paper proposes a real-time Virtual Reality game for treating acrophobia that automatically tailors in-game exposure to heights to the players’ individual characteristics – affective state and physiological features. The elements of novelty are the automatic estimation of fear and the prediction of the next game level based on the electroencephalogram (EEG) and biophysical data – Galvanic Skin Response (GSR) and Heart Rate (HR). Two neural networks have been trained with the data recorded in an experiment where 4 subjects have been in-vivo and virtually exposed to various heights. In order to test the validity of the approach, the same users played the acrophobia game, using two modalities of ex-pressing fear level. After completing a game level, the EEG and biophysical data were averaged and one neural network estimated the current fear score, while the other predicted the next game level. A measure of similarity between the self-estimated fear level during a game epoch and the fear level predicted by the first neural network showed an accuracy rate of 73% and 42% respectively for the two modalities of expressing fear level. 3 out of 4 users succeeded to obtain a fear level of 0 (complete relaxation) in the final game epoch.

The field of affective computing has significantly emerged over the recent years, either we talk about systems that recognize and interpret human emotions or applications with assistive or therapeutic purposes that aim to treat various psychological disorders. Our goal is to design and develop a system that would aid people suffering of a common anxiety disorder of our times - the phobia. Thus, we propose a system for treating acrophobia - fear of heights - that exposes the patients in a Virtual Reality scenario depicting a mountain landscape, while their physiological signals (electrodermal activity, heart rate and respiration rate) are continuously recorded. However, during exposure, certain artefacts can appear, caused by head or hand movements, controller or mouse clicks, body rotations - and thus affect the quality of the recording. As artefact identification is a fundamental step in obtaining clean data, in this paper we present a method of recognizing artefact patterns in the signals recorded during gameplay sessions, which consists of aligning the data segments corresponding to the moments when the users performed head/hand/body movements with the artefact signals recorded during a reference procedure. The results obtained in pilot experiment with 5 users showed that the bias is lower on the perfectly aligned segments than on the segments located a few steps before and after the moment when the user introduced the artefact in the game signal, but without statistical significance. In addition, we propose a new method of estimating respiration rate based on the distance between two HTC Vive trackers placed on an elastic band, on the chest and on the upper back torso of the user.

This paper presents a human-centered methodology for designing and developing Virtual Reality Exposure Therapy (VRET) systems. By following the steps proposed by the methodology – Users analysis, Domain Analysis, Task Analysis and Representational Analysis, we developed a system for acrophobia therapy composed of 9 functional, interrelated modules which are responsible for patients, scenes, audio and graphics management, as well as for physiological monitoring and events triggering. The therapist visualizes in real time the patient’s biophysical signals and adapts the exposure scenario accordingly, as he can lower or increase the level of exposure. There are 3 scenes in the game, containing a ride by cable car, one by ski lift and a walk by foot in a mountain landscape. A reward system is implemented and emotion dimension ratings are collected at predefined points in the scenario. They will be stored and later used for constructing an automatic machine learning emotion recognition and exposure adaptation module.