ICCV 2023 Tutorial on

Understanding the In-Camera Rendering Pipeline and the role of AI/Deep Learning

Oct 3, 2023 (Tuesday, Half Day Tutorial - PM)
Instructor
Michael S. Brown
Professor and Canada Research Chair
York University, Canada

Senior Director, Samsung AI Center
Toronto, Canada

Tutorial Description

This tutorial aims to provide attendees with a thorough overview of how modern digital cameras work. The tutorial is organized into two parts. The first part provides a background on color theory and color representations, namely the CIE 1931 XYZ color space and its derivatives commonly found in computer vision (sRGB, L*ab, Yuv, etc.). The first part of the tutorial will also discuss the routines applied onboard cameras' conventional image signal processing (ISP) hardware to convert the low-level sensor raw RGB responses to their final standard RGB (sRGB) colors. These routines include computational color constancy (auto white balance), colorimetric conversion, image demosaicing, image denoising, tone-mapping, super-resolution, and general color manipulation. The second part of this tutorial discusses recent AI-based methods that target improving individual ISP components. This is followed by current AI methods that aim to fully replace conventional ISPs with AI-based ISPs.

Tutorial Schedule

Part 1 (1.30PM-3.30PM) : Overview of Basic Color Manipulation and the Camera Imaging Pipeline (2 hours)
Level (Novice)

• Background on Color
• Human sensitivity
• CIE XYZ color matching functions and relationship to colorimetry
• Color constancy and its relationship to illuminations and color temperatures (CCT)
• Review of Color Models and Color Spaces (CIE XYZ and derivatives---CIE L*ab, sRGB,AdobeRGB, ProPhoto)
• Camera Pipeline Overview
• Flat field correction
• White balance
• Demosiacing
• Denoising
• Sensor characterization/Colorimetric conversion
• Global and Local Tone Mapping
• Color Manipulation and Color Preference
• Resizing and Digital Zoom

Coffee Break (3.30PM - 4.00PM)

Part 2 (4.00PM - 6.00PM) : AI for Improving ISPs (1.5 hours + 30 minutes Q/Q)
Level (Intermediate to Advanced)

• AI modules for individual ISP steps
• Auto-White-Balance
• Demosaicing
• Noise Reduction
• Super-resolution (digital zoom)
• Multi-frame processing (low-light/HDr)
• AI-based ISPs
• Single (monolithic) DNN-based ISPs
• Two-stage DNN-based ISPs
• Multi-stage DNN-based ISPs
• Misc topics
• Conclusion

Tutorial Slides

Link to PDF ( Slides )

Additional Materials

Books
R.W.G. Hunt, The Reproduction of Colour, Wiley , 2004
G. Sharma, Digital Color Imaging Handbook, CRC Press , 2003
M. Fairchild, Color Appearance Models, Wiley , 2005
D. Forsyth and J. Ponce, Computer Vision: A modern approach, Prentice Hall, 2011
P. Green and L. MacDonlad, Colour Engineering: Achieving Device Independent Colour, Wiley , 2002

Articles/Conference Papers
Jeong W. and Jung S.W. "RAWToBit: A Fully End-to-end Camera ISP Network", ECCV'22
Umn K.H. et al. "Image Compression-Aware Deep Camera ISP Network", IEEE Access'21
Liu et al. "Deep-FlexISP: A Three-stage Framework for Night Photography Rendering", CVPRW'22 (NTIRE)
Ershov et al. "NTIRE 2022 Challenge on Night Photography Rendering", CVPRW'22 (NTIRE)
Liang Z. et al. "CameraNet: A Two-Stage Frameworkfor Effective Camera ISP Learning", TIP'21
Souza M. and Heidrich W. "CRISPnet: Color Rendition ISP Net", arxiv'21
Chen C. et al. "Learning to See In the Dark", CVPR'18
Ignatov A. et al. "Replacing Mobile Camera ISP with a Single Deep Learning Model", CVPRW'19 (NTIRE)
Nam S. and Kim S.J. "Modelling the Scene Dependent Imaging in Cameras with a Deep Neural Network", CVPR'17
Kalantari N. and Ramanoorthi "Deep High Dynamic Range Imaging of Dynamic Scenes", SIGGRAPH'17
Mertens T. et al. "Exposure Fusion", Pacific Graphics'07
Senhar A. S. et al. "Burst Photography or Learning to Enhance Dark Images", TIP'21
Wronski B. et al. "Handheld Multi-Frame Photography", SIGGRAPH'19
Hassinoff S. W. et al. "Burst Photography for High-Dynamic Range and Low-light Imaging on Mobile Cameras", SIGGRAPH'16
Moon Y.S. et al. "A Fast Low-Light Multi-Image Fusion with Online Image Restoration", ICCE'13
Zhang K. et al. "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising", TIP'17
Dabov K. et al. "Image Denoising by Sparse 3D transform-domain collaborative filtering", TIP'07
Liu L. et al. "Joint Demosiacing and Denoising with Self Guidance", CVPR'21
Syu N. et al. "Learning Deep Convolutional Networks for Demosiacing", arvix'16
Gharbi M. et al. "Deep Joint Demosiacking and Denoising", SIGGRAPH Asia'16
Abdelhamed A. et al. "Leveraging the Availability of Two Cameras for Illuminat Estimation", CVPR'21
Hu Y. et al. "FC4: Fully Convolutional Color Constancy with Confidence-weighted Pooling", CVPR'17
Cheng D. et al. "Effective Learning-Based Illuminant Estimation Using Simple Features", CVPR'15
Ledig C. et al. "Photo-Realistic Single Image Super-Resolution Using a Generative Adversial Network", CVPR'17
Kim J. et al. "Accurate Image Super-Resolution Using Very Deep Convolutional Networks", CVPR'16
Dong C. et al. "Image Super-Resolution Using Deep Convolutional Networks", ECCV'14
Abdelhamed A. et al. "A High-Quality Denoising Dataset for Smartphone Cameras", CVPR'18
Chakrabarti A. et al. "Modeling Radiometric Uncertainty for Vision with Tone-mapped Color Images", TPAMI 2014
Cheng D. et al. "Beyond White: Ground Truth Colors for Color Constancy Correction", ICCV'15
Karaimer H. and Brown M.S. "A Software Platform for Manipulating the Camera Imaging Pipeline", ECCV'16

Preparation of this tutorial was supported by grants and gifts from NSERC, the NSERC Canada Reserach Chair program, the CFREF-VISTA program, Adobe Research, Google Research, Microsoft Research-Asia, and Samsung Research.