Bridging the Gap between Computational Photography and Visual Recognition

Rosaura G. Vidalmata, Sreya Banerjee, Brandon Richardwebster, Michael Albright, Pedro Davalos, Scott McCloskey, Ben Miller, Asong Tambo, Sushobhan Ghosh, Sudarshan Nagesh, Ye Yuan, Yueyu Hu, Junru Wu, Wenhan Yang, Xiaoshuai Zhang, Jiaying Liu, Zhangyang Wang, Hwann Tzong Chen, Tzu Wei Huang, Wen Chi ChinYi Chun Li, Mahmoud Lababidi, Charles Otto, Walter J. Scheirer

Research output: Contribution to journalArticlepeer-review

Abstract

What is the current state-of-the-art for image restoration and enhancement applied to degraded images acquired under less than ideal circumstances? Can the application of such algorithms as a pre-processing step improve image interpretability for manual analysis or automatic visual recognition to classify scene content? While there have been important advances in the area of computational photography to restore or enhance the visual quality of an image, the capabilities of such techniques have not always translated in a useful way to visual recognition tasks. Consequently, there is a pressing need for the development of algorithms that are designed for the joint problem of improving visual appearance and recognition, which will be an enabling factor for the deployment of visual recognition tools in many real-world scenarios. To address this, we introduce the UG$^2$2 dataset as a large-scale benchmark composed of video imagery captured under challenging conditions, and two enhancement tasks designed to test algorithmic impact on visual quality and automatic object recognition. Furthermore, we propose a set of metrics to evaluate the joint improvement of such tasks as well as individual algorithmic advances, including a novel psychophysics-based evaluation regime for human assessment and a realistic set of quantitative measures for object recognition performance. We introduce six new algorithms for image restoration or enhancement, which were created as part of the IARPA sponsored UG$^2$2 Challenge workshop held at CVPR 2018. Under the proposed evaluation regime, we present an in-depth analysis of these algorithms and a host of deep learning-based and classic baseline approaches. From the observed results, it is evident that we are in the early days of building a bridge between computational photography and visual recognition, leaving many opportunities for innovation in this area.

Original languageEnglish
Pages (from-to)4272-4290
Number of pages19
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume43
Issue number12
DOIs
StatePublished - 1 Dec 2021
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 1979-2012 IEEE.

Keywords

  • Computational photography
  • deconvolution
  • deep learning
  • evaluation
  • object recognition
  • super-resolution

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'Bridging the Gap between Computational Photography and Visual Recognition'. Together they form a unique fingerprint.

Cite this