In the last decade, an increasing need for affective and socially intelligent technology has been seen, partly caused by upcoming interactive technology that is enhancing our daily lives in our homes and at work. This has led to a significant increase of research in Social Signal Processing (SSP) in which the aims are to model, analyse, and synthesize social signals (including affective signals) and to develop socially intelligent machines. This body of work is inherently multimodal (e.g., eye gaze, touch, vocal, and facial expressions) and multidisciplinary (e.g., psychology, linguistics, computer science). Major research foci include the automatic understanding and generation of emotional and social behavior in specific situations. Applications are plentiful: the development of social robots, intelligent virtual agents, and smart environments are some of the application areas that will benefit from SSP research. 

SSP research involves studying human-human interactions, as well as human-machine interactions. Large corpora consisting of spontaneous human-human interactions offer SSP researchers the opportunity to analyse and understand multimodal human behaviors, and to develop detectors and data mining algorithms. Mining large amounts of human-human interaction data can unravel relations between modalities that were initially hidden from the naked eye. Human-machine interactions on the other hand can be studied in order to understand how the socially intelligent technology developed affects how humans interact with machines.

Although many SSP-related applications already exist, the puzzle is far from solved. Major challenges include robustness of the applications and algorithms, the role of situational and user context in SSP, data collection and annotation, and unknown relations among multiple modalities. SSP is a continuously developing and lively multidisciplinary research domain, bringing along new challenges, methods, application areas and emerging fields of research.