Scene understanding started with the goal of building machines that can see like humans to infer general principles and current situations from imagery, but it has become much broader than that. Applications such as image search engines, autonomous driving, computational photography, vision for graphics, human machine interaction, were unanticipated and other applications keep arising as scene understanding technology develops. As a core problem of high level computer vision, while it has enjoyed some great success in the past 50 years, a lot more is required to reach a complete understanding of visual scenes. Because of larger, faster and cheaper computation power, significant amount of data, new discoveries in human vision, we have precious opportunities that were not available before, to solve the problem. While scene understanding is of great importance, it is also known to be notoriously difficult. To help make further progress in this field, we propose to invite everyone in the field to participate in discussions, and showcase their latest innovations/ideas. This is the fifth installment, continuing the theme of the well-attended SUN workshops that we organized for CVPR 2013 - 2016.