Recognizing Scene Viewpoint using Panoramic Place Representation


Abstract

The pose of an object carries crucial semantic meaning for object manipulation and usage (e.g., grabbing a mug, watching a television). Just as pose estimation is part of object recognition, viewpoint recognition is a necessary and unavoidable component of scene recognition. For instance, as shown in Figure 1, a theater has a clear distinct distributions of objects – a stage on one side and seats on the other – that defines unique views in different orientations. Just as observers will choose a view of a television that allows them to see the screen, observers in a theater will sit facing the stage when watching a show. The goal of this paper is to study the viewpoint recognition problem in scenes. We aim to design a model which, given a photo, can classify the place category to which it belongs (e.g. a theater), and predict the direction in which the observer is facing within that place (e.g. towards the stage). Our model learns the typical arrangement of visual features in a 360-degree panoramic representation of a place, and learns to map individual views of a place to that representation. Now, given an input photo, we will be able to place that photo within a larger panoramic image. This allows us to extrapolate the layout beyond the available view, as if we were to rotate the camera all around the observer.

Paper

Video

Short Fast-forward Introduction (Download) Full Length Video (Download)

Poster

More Example Results

Algorithm Analysis

Geometry of Panorama

Performance

More materials

Slides

Download full SUN360 dataset

Browse SUN360 dataset

Source code

Acknowledgments

We thank Tomasz Malisiewicz, Andrew Owens, Aditya Khosla, Dahua Lin and reviewers for helpful discussions. This work is funded by NSF grant (1016862) to A.O, Google research awards to A.O and A.T., ONR MURI N000141010933 and NSF Career Award No. 0747120 to A.T., and a NSF Graduate Research fellowship to K.A.E. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation and other funding agencies. All materials in this website, including images, data, and visualization, can be used for academic research purpose ONLY.