Conditional random fields for scene labeling

High-level, or holistic, scene understanding involves reasoning about objects, regions, the 3D relationships between them, etc. Scene labeling underlies many of these problems in computer vision. Reasoning about scene images requires the decomposition into semantically meaningful regions over which a graphical model can be imposed. Typically, representational models, learned from data, are defined in terms of a unified energy function over the appearance and structure of the scene-under-investigation. In this chapter, we explore energy functions defined within the context of conditional random fields (CRF) and examine in detail, one learning and inference technique that can be used to reason about the scene. We specifically review methods involving semantic categorization (such as grass, sky, foreground, etc.) and geometric categorization (typically the vertical plane, the ground plane, and the sky plane). CRFs are an effective tool for partitioning images into their constituent semantic or geometric level regions and assigning the appropriate class labels to each region. We also present specific algorithms from the literature that have successfully used CRFs for labeling scene images.