Advancing Affect Modeling via Preference Learning and Unsupervised Feature Extraction

Recognizing and reacting to emotions are fundamental elements in communication among humans. Transferring these skills to computers is an exceptionally complex task, in part, due to the subjective nature of emotions and the subtle, context-dependent and disperse properties of their manifestations. This thesis investigates methods to uncover the mapping between emotions and their manifestations based on observations of humans experiencing specific affective states.

The first challenge is to annotate and, in turn, recognize the affective
states experienced. While posing interesting computational difficulties, ordinal
reports such as rankings and ratings can yield more reliable affect annotations
than alternative tools. This thesis explores preference learning methods to
automatically learn computational models from ordinal annotations of affect.
In particular, an extensive collection of training strategies (error functions and
training algorithms) for artificial neural networks are examined across synthetic
and psycho-physiological datasets, and compared against support vector machines and Cohen’s method. Results reveal the best training strategies for neural networks and suggest their superiority over the other examined methods.

The second challenge addressed in this thesis refers to the extraction of
relevant information from physiological modalities. Deep learning is proposed
as an automatic approach to extract input features for models of affect from
physiological signals. Experiments on psycho-physiological datasets show
that these methods, in combination with automatic feature selection, can
reveal information that yields more accurate predictors of affect than typical
hand-crafted feature extractors examined in Affective Computing research.

The third challenge arises from the complexity of hand-crafting feature
extractors that combine information across dissimilar modalities of input.
Frequent sequence mining is presented as a method to learn feature extractors
that fuse physiological and contextual information. This method is evaluated
in a game-based dataset and compared against ad-hoc extracted features. The evaluation reveals that this unsupervised method, combined with appropriate feature selection algorithms, yields more accurate predictors of affective player experiences than hand-crafted single-modality features.

In summary, this thesis proposes and validates a complete methodology
for building models of affect from ordinal annotations with minimal expertknowledge, advancing affect modeling towards an automated data-driven
process. The generality of the thesis’ key findings presented along with the
limitations and the extensibility of the proposed components are discussed.