Ph.D. Students

• Genevieve Patterson
• Libin "Geoffrey" Sun

Visiting Students

• Mathias Eitz

Master's Students

• Vazheh Moussavi
• Paul Sastrasinh
• Yun Zhang
• David Dufresne (graduated)
• Sirion Vittayakorn (graduated)
• Arcady Goldmints-Orlov (graduated)

Undergraduate Students

• Sam Birch
• Eli Bosworth
• Travis Webb (graduated)

Quality Assessment for Crowdsourced Object Annotations.
Sirion Vittayakorn and James Hays.
British Machine Vision Conference (BMVC) 2011.

Project page, Paper, Bibtex

Scene categorization and detection: the power of global features
James Hays, Jianxiong Xiao, Krista Ehinger, Aude Oliva, and Antonio Torralba.
Vision Sciences Society annual meeting (VSS) 2010.

SUN Database: Large-scale Scene Recognition from Abbey to Zoo
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2010.

Project page, Paper, Browse database

we present the extensive Scene UNderstanding (SUN) database containing 899 categories and 130,519 images. We use 397 well-sampled categories to benchmark numerous state-of-the-art algorithms for scene recognition. We measure human scene classification performance on the SUN database and compare this with computational methods.

Our visual experience is extraordinarily varied and complex. The diversity of the visual world makes it difficult for computer vision to understand images and for computer graphics to synthesize visual content. But for all its richness, it turns out that the space of "scenes" might not be astronomically large. With access to imagery on an Internet scale, regularities start to emerge - for most images, there exist numerous examples of semantically and structurally similar scenes. Is it possible to sample the space of scenes so densely that one can use similar scenes to "brute force" otherwise difficult image understanding and manipulation tasks? This thesis is focused on exploiting and refining large scale scene matching to short circuit the typical computer vision and graphics pipelines for image understanding and manipulation.

IM2GPS: estimating geographic information from a single image
James Hays and Alexei Efros.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2008.

Project Page, Paper, Bibtex

Google Tech Talk.

Abstract: Estimating geographic information from an image is an excellent, difficult high-level computer vision problem whose time has come. The emergence of vast amounts of geographically-calibrated image data is a great reason for computer vision to start looking globally - on the scale of the entire planet! In this paper, we propose a simple algorithm for estimating a distribution over geographic locations from a single image using a purely data-driven scene matching approach. For this task, we will leverage a dataset of over 6 million GPS-tagged images from the Internet. We represent the estimated image location as a probability distribution over the Earth's surface. We quantitatively evaluate our approach in several geolocation tasks and demonstrate encouraging performance (up to 30 times better than chance). We show that geolocation estimates can provide the basis for numerous other image understanding tasks such as population density estimation, land cover estimation or urban/rural classification.

Scene Completion Using Millions of Photographs
James Hays and Alexei Efros.
Transactions on Graphics (SIGGRAPH 2007). August 2007, vol. 26, No. 3.

Abstract: What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of image completions and we allow users to select among them. We demonstrate the superiority of our algorithm over existing image completion approaches.

Interactive Tensor Field Design and Visualization on Surfaces
Eugene Zhang, James Hays, and Greg Turk.
IEEE Transaction on Visualization and Computer Graphics, 2007, Vol 13(1), pp 94-107.

This research project was primarily Eugene's work and I played only a small role.

Image De-fencing
Yanxi Liu, Tamara Belkina, James Hays, and Roberto Lublinerman.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2008.

We introduce a novel image segmentation algorithm that uses translational symmetry as the primary foreground/background separation cue. We use texture-based inpainting to recover an un-occluded background.

Discovering Texture Regularity as a Higher-Order Correspondence Problem
James Hays, Marius Leordeanu, Alexei Efros, and Yanxi Liu.
European Conference on Computer Vision (ECCV) 2006.

We find arbitrarily distorted regular patterns in real images by treating lattice-finding as a higher-order assignment problem. We leverage previous work from Marius Leordeanu and Martial Hebert to approximate the optimal assignment under second-order constraints.

Source code available upon request, although this code by Minwoo Park et al. is likely more accurate and faster.

Quantitative Evaluation of Near Regular Texture Synthesis Algorithms
Steve Lin, James Hays, Chenyu Wu, Vivek Kwatra, and Yanxi Liu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2006

Quantitative evaluation is difficult for texture synthesis. Ground truth is not well defined. But for certain textures you can objectively decide whether an algorithm has failed or not. Regular and near-regular textures imply a definite structure that should be preserved. We tested several popular algorithms on a large group of structured textures. In addition to the CVPR 2006 paper, a more detailed technical report is available.

Digital Papercutting
Yanxi Liu, James Hays, Ying-Qing Xu, and Harry Shum SIGGRAPH 2005 Sketch

Papercutting is a widespread and ancient artform which, as far as we could tell, had no previous computational treatment. We developed algorithms to analyze the symmetry of papercut patterns and produce efficient folding and cutting plans.

Near-Regular Texture Analysis and Manipulation
Yanxi Liu, Steve Lin, and James Hays. SIGGRAPH 2004

Abstract: A near-regular texture deviates geometrically and photometrically from a regular congruent tiling. Although near-regular textures are ubiquitous in the man-made and natural world, they present computational challenges for state of the art texture analysis and synthesis algorithms. Using regular tiling as our anchor point, and with user-assisted lattice extraction, we can explicitly model the deformation of a near-regular texture with respect to geometry, lighting and color. We treat a deformation field both as a function that acts on a texture and as a texture that is acted upon, and develop a multi-modal framework where each deformation field is subject to analysis, synthesis and manipulation. Using this formalization, we are able to construct simple parametric models to faithfully synthesize the appearance of a near-regular texture and purposefully control its regularity.

Image and Video Based Painterly Animation
James Hays and Irfan Essa. NPAR 2004.

We extend previous non-photorealistic rendering work to handle video significantly better by temporally constraining brush stroke properties in addition to other improvements.