Efficient Unified Demosaicing for
Bayer and Non-Bayer Patterned Image Sensors

ICCV 2023

Haechang Lee1,3,*, Dongwon Park2,*, Wongi Jeong1,*,
Kijeong Kim3, Hyunwoo Je3, Dongil Ryu3, and Se Young Chun1,2
1Dept. of ECE, 2INMC & IPAI, Seoul National University, Republic of Korea,
3SK hynix, Republic of Korea
Paper arXiv Code

Abstract

As the physical size of recent CMOS image sensors (CIS) gets smaller, the latest mobile cameras adopt unique non-Bayer color filter array (CFA) patterns (e.g., Quad, Nona, QxQ), which consist of homogeneous color units with adjacent pixels. These non-Bayer CFAs are superior to conventional Bayer CFA thanks to their changeable pixel-bin sizes for different light conditions, but may introduce visual artifacts during demosaicing due to their inherent pixel pattern structures and sensor hardware characteristics. Previous demosaicing methods have primarily focused on Bayer CFA, necessitating distinct reconstruction methods for non-Bayer CIS with various CFA modes under different lighting conditions. In this work, we propose an efficient unified demosaicing method that can be applied to both conventional Bayer RAW and various non-Bayer CFAs' RAW data in different operation modes. Our Knowledge Learning-based demosaicing model for Adaptive Patterns, namely KLAP, utilizes CFA-adaptive filters for only 1% key filters in the network for each CFA, but still manages to effectively demosaic all the CFAs, yielding comparable performance to the large-scale models. Furthermore, by employing meta-learning during inference (KLAP-M), our model is able to eliminate unknown sensor-generic artifacts in real RAW data, effectively bridging the gap between synthetic images and real sensor RAW. Our KLAP and KLAP-M methods achieved state-of-the-art demosaicing performance in both synthetic and real RAW data of Bayer and non-Bayer CFAs.

Overview

{"overview"}

(a) Overview of our unified model (UM) for demosaicing all the Bayer and non-Bayer CFAs, called the Knowledge Learning-based demosaicing model for Adaptive Patterns using Meta-test learning (KLAP-M), even when ground truth is unavailable and unknown artifacts are present. (b) Comparing CIS RAW demosaicing results of KLAP (KLAP-M without meta-test learning) and KLAP-M (KLAP with meta-test learning).


Data Synthesis

{"Data_Synthesis"}

Overview of our pipeline for synthesizing realistic RAW images, specifically for QxQ patterns.


KLAP-M Architecture

{"Architecture"}

Overview of our proposed unified DM model, Knowledge Learning-based demosaicing model for Adaptive Pattern (KLAP) and KLAP with Meta-test learning (KLAP-M). KLAP consists of 2 steps: (a) two-stage knowledge learning (TKL) for training baselines, (b) fine-tuning using Adaptive Discriminant filters for each specific CFA Pattern (ADP). (c) KLAP-M employs meta-learning to reduce unknown artifacts in real RAW images during inference


Experiments

{"table1"}

Ablation studies and quantitative performance comparison (PynetQxQ, Kim, Uformer, Chen and Li) for KLAP (TKL+ADP) on DF2K-CIS test dataset. PyNetQxQ, Kim and Uformer are independent models, while Chen and Li are unified models. The methods highlighted in purple are based on NAFNet. Baseline-UM is a simple unified model. TKL refers to the TKL-applied Baseline UM, and ADP refers to the ADP-applied Baseline UM independently. TKL-to-IM refers to the fine-tuned IM using TKL. Avg. denotes mean of all CFA's PSNR (dB), and Par. denotes the required number of parameters (Million).


{"exp1"}

Comparison of demosaiced results top and their difference maps bottom on the synthetic RAW (DF2K-CIS) test set produced by different methods. The PSNR (dB) values displayed in the top-left corner are for the entire image. As shown, our proposed KLAP demonstrates the best performance. Note that CM is applied to the DM outputs for visualization.


{"exp2"}

Comparisons among different methods of robustness on DF2K-CIS with strong noise test dataset. The noise parameters used in the test are four times larger than the noise parameters used in the training. The number of meta-learning iterations in KLAP-M is set to 10, based on empirical determination through visualization of outputs in our experiments.


{"exp3"}

Qualitative DM results on the real CIS RAW. Note that KLAP with meta-test learning (KLAP-M) shows robust performance in real CIS RAW, despite of existence of sensor-generic unknown artifacts. Note that CM is applied to the DM outputs for visualization.


{"exp4"}

Additional inference images of CIS RAW, MIPI 2022 Quad remosaic challenge data using KLAP (trained by our synthetic dataset, DF2K-CIS) and KLAP-M. (a) demosaiced output images obtained using KLAP-M inference, and (b) the same images as in (b) after applying CM (Color-related Mapping function). As shown in the figures above, KLAP with meta-test learning (KLAP-M) shows robust performance in another real CIS RAW, MIPI 2022 Quad data, despite of existence of sensor-generic unknown artifacts.

BibTeX

@inproceedings{lee2023efficient,
  title={Efficient Unified Demosaicing for Bayer and Non-Bayer Patterned Image Sensors},
  author={Lee, Haechang and Park, Dongwon and Jeong, Wongi and Kim, Kijeong and Je, Hyunwoo and Ryu, Dongil and Chun, Se Young},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={12750--12759},
  year={2023}
}