Planar content selection in images and videos using frontalness

This paper addresses the problem of selecting instances of a planar object in a video or from a set of images based on an evaluation of its “frontalness”. We introduce the idea of “evaluating the frontalness” by computing how close the object's surface normal aligns with the optical axis of a camera. Unlike previous planar object pose estimation methods, our method does not require the true frontal image as a reference. The intuition is that a true frontal image can produce other non-frontal images by perspective projection, while the non-frontal images have limited ability to produce other non-frontal images. We show that this intuition of comparing ‘frontal’ and ‘non-frontal’ can be extended to comparing ‘more frontal’ and ‘less frontal’ images. Based on this observation, our method estimates the relative frontalness of an image by exploiting the objective space error. We also propose the use of K-invariant space to evaluate the frontalness even when the camera intrinsic parameters are unknown (e.g., images/videos from the web). Our method outperforms the homography decomposition-based method and does not require reference images. A qualitative evaluation is also carried out to show that our method can be applied in selecting the most frontal characters from a set of images captured in various viewpoints.