The face plays a key role in many real-world applications such as security systems, human computer interaction, remote monitoring of patients, video annotation, and gaming. Having detected the face, pattern recognition techniques and machine learning algorithms are applied to facial images, for example, to find the identity of a subject or analyze her/his emotional status. Though face and facial expression recognition in still images and in ideal imaging conditions have been around for many years, they have been less explored in video sequences in uncontrolled imagining conditions. Developing face and facial expression recognition algorithms for real-world scenarios, for instance, for remote patient monitoring or for identification in surveillance videos are still challenging tasks.