Smartphone Image Denoising Dataset

Abdelrahman Abdelhamed1             Stephen Lin2             Michael S. Brown1

1York University             2Microsoft Research

SIDD Benchmark

The SIDD Benchmark has been hosted as a challenge at the New Trends in Image Restoration and Enhancement (NTIRE 2019) workshop in conjunction with CVPR 2019.
The participating solutions and results will be published in the challenge report in the CVPR Workshop proceedings.

Challenge preliminary results can be viewed at the following Codalab competitions:
NTIRE 2019 Real Image Denoising Challenge - Track 1: Raw-RGB
NTIRE 2019 Real Image Denoising Challenge - Track 2: sRGB

The benchmark is currently hosted here on this website.

Download

[Download the SIDD Benchmark Data (1.84 GB)]   MD5: decd113eaf99a8dbd1dbb7f7c9dafedd   SHA1: b8092d990139f41b6da97b4afa679a2876de53bd

[Download the SIDD Benchmark Code v1.2 (9 KB)]

Benchmark data as single Matlab arrays of dimensoins [#images, #blocks, height, width, #channels]:
Noisy raw-RGB data: [Download]
Noisy sRGB data: [Download]

Validation data and ground truth as single Matlab arrays of dimensoins [#images, #blocks, height, width, #channels]:
Noisy raw-RGB data: [Download]
Noisy sRGB data: [Download]
Ground-truth raw-RGB data: [Download]
Ground-truth sRGB data: [Download]

Description

The SIDD Benchmark consists of 40 images representing 40 scene instances. These images can be used to benchmark denoising methods.

For each image, the following is provided in one directory:

  1. Noisy Raw-RGB image (.MAT). Black Level subtracted, normalized to [0, 1].
  2. Noisy sRGB image (.PNG). Gamma corrected, without any tone mapping.
  3. Metadata extracted from the DNG file (.MAT). For example, black and saturation levels, as-shot neutral, noise level function, etc.

The PSNR and SSIM values are calculated only on 32 blocks of size 256 by 256 pixels. The block positions are provided in a file named "BenchmarkBlocks32.mat".

Follow the instructions in the Code_v/_ReadMe.txt file to

Upload your results

Click here to submit your results

Benchmark Results

Results of the NTIRE 2019 Challenge on Real Image Denoising can be found in this paper.

The following tables show the benchmark results published in the paper.

Results of denoising in raw-RGB space

Method PSNR SSIM Time
(Applied on/Evaluated on) (Applied on/Evaluated on) (seconds)
Raw/Raw Raw/sRGB Raw/Raw Raw/sRGB Raw
BM3D 45.52 30.95 0.980 0.863 34.3
NLM 44.06 29.39 0.971 0.846 210.7
KSVD 43.26 27.41 0.969 0.832 2243.9
KSVD-DCT 42.70 28.21 0.970 0.784 133.3
KSVD-G 42.50 28.13 0.969 0.781 153.6
LPG-PCA 42.79 30.01 0.974 0.854 438.1
FoE 43.13 27.18 0.969 0.812 6097.2
MLP 43.17 27.52 0.965 0.788 131.2
WNNM 44.85 29.54 0.975 0.888 1975.8
GLIDE 41.87 25.98 0.949 0.816 12440.5
TNRD 42.77 26.99 0.945 0.744 15.2
EPLL 40.73 25.19 0.935 0.842 653.1
DnCNN 43.30 28.24 0.965 0.829 51.7

Results of denoising in sRGB space

Method PSNR SSIM Time
(Applied on/Evaluated on) (Applied on/Evaluated on) (seconds)
sRGB/sRGB sRGB/sRGB sRGB
BM3D 25.65 0.685 27.4
NLM 26.75 0.699 621.9
KSVD 26.88 0.842 9881.0
KSVD-DCT 27.51 0.780 96.3
KSVD-G 27.19 0.771 92.2
LPG-PCA 24.49 0.681 2004.3
FoE 25.58 0.792 12166.8
MLP 24.71 0.641 564.8
WNNM 25.78 0.809 8882.2
GLIDE 24.71 0.774 36091.6
TNRD 24.73 0.643 45.1
EPLL 27.11 0.870 1996.4
DnCNN 23.66 0.583 158.9
CBDNet 33.28 0.868 4.48
DeepProxies BM3D 34.34 0.911 6.69
mwresnet 38.52 0.949 47.73
mwresnet 39.31 0.956 77.34
mwresnet 39.64 0.958 25.14
Path-Restore 38.21 0.946 0.89
HT-MWResnet 39.80 0.959 63.45
test 39.78 0.958 -1.00
test2 37.97 0.942 -1.00
test3 37.97 0.942 -1.00