THE PROBLEM OF IMAGE SUPER-RESOLUTION, DENOISING AND IMAGE RESTORATION METHODS IN DEEP LEARNING MODELS
Article Sidebar
Published:
Sep 6, 2024
Keywords:
denoising, ED-Unet, image restoration, pretrained weights, resolution
Main Article Content
Abstract
This article addresses the challenges of image super-resolution and noise reduction, which are crucial for enhancing the quality of images derived from low-resolution or noisy data. Several approaches were compared and assessed for upgrading low-resolution images to higher resolutions and eliminating unwanted noise while maintaining the essential characteristics of the original images and recovering images from poor quality or damaged data using deep learning models. It is indicated by research analysis and the experimental outcomes on image quality metrics that the ED-Unet neural network model, enhanced with pretrained weights, significantly outperforms other methods, achieving a Train PSNR of 30.791, a Valid PSNR of 30.699, and a Test PSNR of 31.0172.
Downloads
Download data is not yet available.
Article Details
How to Cite
1.
Pham N-G, Tong Le T-H, Duong V-H, Tran H-N, Vo P-H. THE PROBLEM OF IMAGE SUPER-RESOLUTION, DENOISING AND IMAGE RESTORATION METHODS IN DEEP LEARNING MODELS. journal [Internet]. 6Sep.2024 [cited 18Oct.2024];14(8). Available from: https://journal.tvu.edu.vn/index.php/journal/article/view/4360
Section
Articles
References
[1] Janani P. Image quality enhancement
techniques for digital media. Indian Journal
of Science and Technology. 2015;8(22): 1–12.
https://doi.org/10.17485/ijst/2015/v8i22/79318.
[2] He K, Zhang X, Ren S, Sun S. Deep residual learning
for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 27th–
30th June 2016; Las Vegas, NV, USA. IEEE; 2016.
p.770–778. https://doi.org/10.1109/CVPR.2016.90.
[3] Wang Z, Chen J, Hoi S. Deep learning
for image super-resolution: A survey. IEEE
Transactions on Pattern Analysis and
Machine Intelligence. 2021;43(10): 3365–3387.
https://doi.org/10.1109/TPAMI.2020.2982166.
[4] Liang X, Chen X, Ren K. Low-light image enhancement via adaptive frequency decomposition
network. Scientific Reports. 2023;13(1): 14107.
https://doi.org/10.1038/s41598-023-40899-8.
[5] Husem H, Orman Z. A survey on image ¨
super-resolution with generative adversarial
networks. Acta Infologica. 2020:4(2): 139–154.
https://doi.org/10.26650/acin.765320.
[6] Moghimi MK, Mohanna F. Real-time underwater
image resolution enhancement using super-resolution
with deep convolutional neural networks. Journal of
Real-Time Image Processing. 2021;18(5): 1653–1667.
https://doi.org/10.1007/s11554-020-01024-4.
[7] Dong C, Loy CC, He K, Tang X. Learning a deep
convolutional network for image super-resolution. In:
Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds.).
Computer Vision – ECCV 2014. Cham: Springer;
2014. https://doi.org/10.1007/978-3-319-10593-2_13.
[8] Kim J, Lee JK, Lee KM. Deeply-recursive convolutional network for image super-resolution. In:
IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 27th–30th June 2016;
Las Vegas, NV, USA. IEEE; 2016. p.1637–1645.
https://doi.org/10.1109/CVPR.2016.181.
[9] Dong C, Loy CC, Tang X. Accelerating the superresolution convolutional neural network. In: Leibe
B, Matas J, Sebe N, Welling M (eds.). Computer Vision – ECCV 2016. Cham: Springer; 2016.
https://doi.org/10.1007/978-3-319-46475-6_25.
[10] Shi W, Caballero J, Huszár F, Totz J, Aitken
AP, Bishop R, et al. Real-time single image and
video super-resolution using an efficient subpixel convolutional neural network. In: IEEE
Conference on Computer Vision and Pattern
Recognition (CVPR). 27th–30th June 2016; Las
Vegas, NV, USA. IEEE; 2016. p.1874–1883.
https://doi.org/10.1109/CVPR.2016.207.
[11] Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, et al. Photo-realistic single
image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 21st–26th July
2017; Honolulu, HI, USA,. IEEE; 2017. p.105–114.
https://doi.org/10.1109/CVPR.2017.19.
[12] Lim B, Son S, Kim H, Nah S, Lee KM. Enhanced deep residual networks for single image
super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops
(CVPRW). 21st–26th July 2017; Honolulu, HI,
USA. IEEE; 2017. p.1132–1140. https://doi.org
/10.1109/CVPRW.2017.151.
[13] Sajjadi MS, Scholkopf B, Hirsch M. EnhanceNet:
Single image super-resolution through automated
texture synthesis. In: IEEE International Conference on Computer Vision (ICCV). 22nd–29th October 2017; Venice, Italy. IEEE; 2017. p.4501–4510.
https://doi.org/10.1109/ICCV .2017.481.
[14] Tai Y, Yang J, Liu X, Xu C. MemNet:
A persistent memory network for image
restoration. In: IEEE International Conference
on Computer Vision (ICCV). 22nd–29th October
2017; Venice, Italy. IEEE; 2017. p.4549–4557.
https://doi.org/10.1109/ICCV.2017.486.
[15] Tong T, Li G, Liu X, Gao Q. Image super-resolution
using dense skip connections. IEEE International
Conference on Computer Vision (ICCV). 22nd–29th
October 2017; Venice, Italy. IEEE; 2017. p.4809–
4817. https://doi.org/10.1109/ICCV.2017.514.
[16] Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y. Residual dense network for image super-resolution. In:
IEEE/CVF Conference on Computer Vision and
Pattern Recognition. 18th–23rd June 2018; Salt
Lake City, UT, USA. IEEE; 2018. p.2472–2481.
https://doi.org/10.1109/CVPR.2018.00262.
[17] Ahn N, Kang B, Sohn KA. Fast, accurate, and
lightweight super-resolution with cascading residual
network. In: Ferrari V, Hebert M, Sminchisescu C,
Weiss Y (eds.). Computer Vision – ECCV 2018.
Cham: Springer; 2018. https://doi.org/10.1007/978-3-
030-01249-6_16.
[18] Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y. Image super-resolution using very deep residual channel attention networks. In: Ferrari V,
Hebert M, Sminchisescu C, Weiss Y (eds.) Computer Vision – ECCV 2018. Cham: Springer; 2018.
https://doi.org/10.1007/978-3-030-01234-2_18.
[19] Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C,
et al. ESRGAN: Enhanced super-resolution generative adversarial networks. In: Leal-Taixé L, Roth S
(eds.). Computer Vision – ECCV 2018 Workshops.
Cham: Springer; 2018. https://doi.org/10.1007/978-3-
030-11021-5_5.
[20] Zhang Y, Li K, Li K, Zhong B, Fu Y.
Residual non-local attention networks for
image restoration. arXiv [Preprint] 2019.
https://doi.org/10.48550/arXiv.1903.10082.
[21] Dai T, Cai J, Zhang Y, Xia ST, Zhang L. Second-order
attention network for single image super-resolution.
In: IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). 15th–20th June 2019;
Long Beach, CA, USA. IEEE; 2019. p.11057–11066.
https://doi.org/10.1109/CVPR .2019.01132.
[22] Niu B, Wen W, Ren W, Zhang X, Yang L, Wang
S, et al. Single image super-resolution via a ternary
attention network. Applied Intelligence. 2023;53(11):
13067–13081. https://doi.org /10.1007/s10489-022-
04129-4.
[23] Chen H, Wang Y, Guo T, Xu C, Deng Y, Liu
Z, et al. Pre-trained image processing transformer.
In: IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). 20th–25th June 2021;
Nashville, TN, USA. IEEE; 2021. p.12294–12305.
https://doi.org/10.1109/CVPR46437 .2021.01212.
[24] Liang J, Cao J, Sun G, Zhang K, Van Gool L, Timofte
R. SwinIR: Image restoration using swin transformer.
In: IEEE/CVF International Conference on Computer
Vision Workshops (ICCVW). 11–17 October 2021;
Montreal, BC, Canada. IEEE; 2021. p.1833–1844.
https://doi.org/10.1109 /ICCVW54120.2021.00210.
[25] Xia B, Zhang Y, Wang S, Wang Y, Wu X, Tian
Y, et al. DiffIR: efficient diffusion model for image restoration. In: IEEE/CVF International Conference on Computer Vision (ICCV). 01st–06th October
2023; Paris, France. IEEE; 2023. p.13049–13059.
https://doi.org/10.1109/ICCV 51070.2023.01204.
[26] Zhang K, Zuo W, Chen Y, Meng D, Zhang L.
Beyond a Gaussian Denoiser: Residual learning of
deep CNN for image denoising. IEEE Transactions on Image Processing. 2017;26(7): 3142–3155.
https://doi.org/10.1109/TIP.2017.2662206.
[27] Wang Z, Jian C, Steve CHH. Deep learning
for image super-resolution: A survey. IEEE
Transactions on Pattern Analysis and
Machine Intelligence. 2021;43(10): 3365–3387.
https://doi.org/10.1109/TPAMI.2020.2982166.
[28] Bashir SMA, Wang Y, Khan M, Niu Y. A comprehensive review of deep learning-based single image
super-resolution. PeerJ Computer Science. 2021;7:
e621. https://doi.org/10.7717/peerj-cs.621.
[29] Odena A, Dumoulin V, Olah C. Deconvolution
and checkerboard artifacts. Distill. 2016;1(10): e3.
http://doi.org/10.23915/distill.00003.
[30] Krishnamoorthy DN, Vigneshraaja M. Enhancing the
image super-resolution via U-net architecture for improved visual quality. Turkish Journal of Physiotherapy and Rehabilitation. 2021;32(2): 1073–1080.
[31] Parkhi OM, Vedaldi A, Zisserman A,
Jawahar CV. The Oxford-IIIT pet dataset.
https://www.robots.ox.ac.uk/ vgg/data/pets/ [Accessed
3rd April 2024].
[32] Zhao H, Gallo O, Frosio I, Kautz J. Loss functions
for image restoration with neural networks. IEEE
Transactions on Computational Imaging. 2017;3(1):
47–57. https://doi.org/10.1109/TCI .2016.2644865.
[33] Zha L, Yang Y, Lai Z, Zhang Z, Wen J. A lightweight
dense connected approach with attention on single image super-resolution. Electronics. 2021;10(11):
1234. https://doi.org/10.3390/electronics10111234.
[34] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal
covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning. 6th–
11th July 2015; Lille, France. JMLR: W&CP; 2015. p.448–456.
techniques for digital media. Indian Journal
of Science and Technology. 2015;8(22): 1–12.
https://doi.org/10.17485/ijst/2015/v8i22/79318.
[2] He K, Zhang X, Ren S, Sun S. Deep residual learning
for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 27th–
30th June 2016; Las Vegas, NV, USA. IEEE; 2016.
p.770–778. https://doi.org/10.1109/CVPR.2016.90.
[3] Wang Z, Chen J, Hoi S. Deep learning
for image super-resolution: A survey. IEEE
Transactions on Pattern Analysis and
Machine Intelligence. 2021;43(10): 3365–3387.
https://doi.org/10.1109/TPAMI.2020.2982166.
[4] Liang X, Chen X, Ren K. Low-light image enhancement via adaptive frequency decomposition
network. Scientific Reports. 2023;13(1): 14107.
https://doi.org/10.1038/s41598-023-40899-8.
[5] Husem H, Orman Z. A survey on image ¨
super-resolution with generative adversarial
networks. Acta Infologica. 2020:4(2): 139–154.
https://doi.org/10.26650/acin.765320.
[6] Moghimi MK, Mohanna F. Real-time underwater
image resolution enhancement using super-resolution
with deep convolutional neural networks. Journal of
Real-Time Image Processing. 2021;18(5): 1653–1667.
https://doi.org/10.1007/s11554-020-01024-4.
[7] Dong C, Loy CC, He K, Tang X. Learning a deep
convolutional network for image super-resolution. In:
Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds.).
Computer Vision – ECCV 2014. Cham: Springer;
2014. https://doi.org/10.1007/978-3-319-10593-2_13.
[8] Kim J, Lee JK, Lee KM. Deeply-recursive convolutional network for image super-resolution. In:
IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 27th–30th June 2016;
Las Vegas, NV, USA. IEEE; 2016. p.1637–1645.
https://doi.org/10.1109/CVPR.2016.181.
[9] Dong C, Loy CC, Tang X. Accelerating the superresolution convolutional neural network. In: Leibe
B, Matas J, Sebe N, Welling M (eds.). Computer Vision – ECCV 2016. Cham: Springer; 2016.
https://doi.org/10.1007/978-3-319-46475-6_25.
[10] Shi W, Caballero J, Huszár F, Totz J, Aitken
AP, Bishop R, et al. Real-time single image and
video super-resolution using an efficient subpixel convolutional neural network. In: IEEE
Conference on Computer Vision and Pattern
Recognition (CVPR). 27th–30th June 2016; Las
Vegas, NV, USA. IEEE; 2016. p.1874–1883.
https://doi.org/10.1109/CVPR.2016.207.
[11] Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, et al. Photo-realistic single
image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 21st–26th July
2017; Honolulu, HI, USA,. IEEE; 2017. p.105–114.
https://doi.org/10.1109/CVPR.2017.19.
[12] Lim B, Son S, Kim H, Nah S, Lee KM. Enhanced deep residual networks for single image
super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops
(CVPRW). 21st–26th July 2017; Honolulu, HI,
USA. IEEE; 2017. p.1132–1140. https://doi.org
/10.1109/CVPRW.2017.151.
[13] Sajjadi MS, Scholkopf B, Hirsch M. EnhanceNet:
Single image super-resolution through automated
texture synthesis. In: IEEE International Conference on Computer Vision (ICCV). 22nd–29th October 2017; Venice, Italy. IEEE; 2017. p.4501–4510.
https://doi.org/10.1109/ICCV .2017.481.
[14] Tai Y, Yang J, Liu X, Xu C. MemNet:
A persistent memory network for image
restoration. In: IEEE International Conference
on Computer Vision (ICCV). 22nd–29th October
2017; Venice, Italy. IEEE; 2017. p.4549–4557.
https://doi.org/10.1109/ICCV.2017.486.
[15] Tong T, Li G, Liu X, Gao Q. Image super-resolution
using dense skip connections. IEEE International
Conference on Computer Vision (ICCV). 22nd–29th
October 2017; Venice, Italy. IEEE; 2017. p.4809–
4817. https://doi.org/10.1109/ICCV.2017.514.
[16] Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y. Residual dense network for image super-resolution. In:
IEEE/CVF Conference on Computer Vision and
Pattern Recognition. 18th–23rd June 2018; Salt
Lake City, UT, USA. IEEE; 2018. p.2472–2481.
https://doi.org/10.1109/CVPR.2018.00262.
[17] Ahn N, Kang B, Sohn KA. Fast, accurate, and
lightweight super-resolution with cascading residual
network. In: Ferrari V, Hebert M, Sminchisescu C,
Weiss Y (eds.). Computer Vision – ECCV 2018.
Cham: Springer; 2018. https://doi.org/10.1007/978-3-
030-01249-6_16.
[18] Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y. Image super-resolution using very deep residual channel attention networks. In: Ferrari V,
Hebert M, Sminchisescu C, Weiss Y (eds.) Computer Vision – ECCV 2018. Cham: Springer; 2018.
https://doi.org/10.1007/978-3-030-01234-2_18.
[19] Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C,
et al. ESRGAN: Enhanced super-resolution generative adversarial networks. In: Leal-Taixé L, Roth S
(eds.). Computer Vision – ECCV 2018 Workshops.
Cham: Springer; 2018. https://doi.org/10.1007/978-3-
030-11021-5_5.
[20] Zhang Y, Li K, Li K, Zhong B, Fu Y.
Residual non-local attention networks for
image restoration. arXiv [Preprint] 2019.
https://doi.org/10.48550/arXiv.1903.10082.
[21] Dai T, Cai J, Zhang Y, Xia ST, Zhang L. Second-order
attention network for single image super-resolution.
In: IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). 15th–20th June 2019;
Long Beach, CA, USA. IEEE; 2019. p.11057–11066.
https://doi.org/10.1109/CVPR .2019.01132.
[22] Niu B, Wen W, Ren W, Zhang X, Yang L, Wang
S, et al. Single image super-resolution via a ternary
attention network. Applied Intelligence. 2023;53(11):
13067–13081. https://doi.org /10.1007/s10489-022-
04129-4.
[23] Chen H, Wang Y, Guo T, Xu C, Deng Y, Liu
Z, et al. Pre-trained image processing transformer.
In: IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). 20th–25th June 2021;
Nashville, TN, USA. IEEE; 2021. p.12294–12305.
https://doi.org/10.1109/CVPR46437 .2021.01212.
[24] Liang J, Cao J, Sun G, Zhang K, Van Gool L, Timofte
R. SwinIR: Image restoration using swin transformer.
In: IEEE/CVF International Conference on Computer
Vision Workshops (ICCVW). 11–17 October 2021;
Montreal, BC, Canada. IEEE; 2021. p.1833–1844.
https://doi.org/10.1109 /ICCVW54120.2021.00210.
[25] Xia B, Zhang Y, Wang S, Wang Y, Wu X, Tian
Y, et al. DiffIR: efficient diffusion model for image restoration. In: IEEE/CVF International Conference on Computer Vision (ICCV). 01st–06th October
2023; Paris, France. IEEE; 2023. p.13049–13059.
https://doi.org/10.1109/ICCV 51070.2023.01204.
[26] Zhang K, Zuo W, Chen Y, Meng D, Zhang L.
Beyond a Gaussian Denoiser: Residual learning of
deep CNN for image denoising. IEEE Transactions on Image Processing. 2017;26(7): 3142–3155.
https://doi.org/10.1109/TIP.2017.2662206.
[27] Wang Z, Jian C, Steve CHH. Deep learning
for image super-resolution: A survey. IEEE
Transactions on Pattern Analysis and
Machine Intelligence. 2021;43(10): 3365–3387.
https://doi.org/10.1109/TPAMI.2020.2982166.
[28] Bashir SMA, Wang Y, Khan M, Niu Y. A comprehensive review of deep learning-based single image
super-resolution. PeerJ Computer Science. 2021;7:
e621. https://doi.org/10.7717/peerj-cs.621.
[29] Odena A, Dumoulin V, Olah C. Deconvolution
and checkerboard artifacts. Distill. 2016;1(10): e3.
http://doi.org/10.23915/distill.00003.
[30] Krishnamoorthy DN, Vigneshraaja M. Enhancing the
image super-resolution via U-net architecture for improved visual quality. Turkish Journal of Physiotherapy and Rehabilitation. 2021;32(2): 1073–1080.
[31] Parkhi OM, Vedaldi A, Zisserman A,
Jawahar CV. The Oxford-IIIT pet dataset.
https://www.robots.ox.ac.uk/ vgg/data/pets/ [Accessed
3rd April 2024].
[32] Zhao H, Gallo O, Frosio I, Kautz J. Loss functions
for image restoration with neural networks. IEEE
Transactions on Computational Imaging. 2017;3(1):
47–57. https://doi.org/10.1109/TCI .2016.2644865.
[33] Zha L, Yang Y, Lai Z, Zhang Z, Wen J. A lightweight
dense connected approach with attention on single image super-resolution. Electronics. 2021;10(11):
1234. https://doi.org/10.3390/electronics10111234.
[34] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal
covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning. 6th–
11th July 2015; Lille, France. JMLR: W&CP; 2015. p.448–456.