Enhanced Deep Residual Networks for single-image super-resolution

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/vision/edsr/" />  <meta property="og:title" content="Keras documentation: Enhanced Deep Residual Networks for single-image super-resolution"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Enhanced Deep Residual Networks for single-image super-resolution"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Enhanced Deep Residual Networks for single-image super-resolution</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink active" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink2" href="/examples/vision/image_classification_from_scratch/">Image classification from scratch</a> <a class="nav-sublink2" href="/examples/vision/mnist_convnet/">Simple MNIST convnet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_efficientnet_fine_tuning/">Image classification via fine-tuning with EfficientNet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_with_vision_transformer/">Image classification with Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/attention_mil_classification/">Classification using Attention-based Deep Multiple Instance Learning</a> <a class="nav-sublink2" href="/examples/vision/mlp_image_classification/">Image classification with modern MLP models</a> <a class="nav-sublink2" href="/examples/vision/mobilevit/">A mobile-friendly Transformer-based model for image classification</a> <a class="nav-sublink2" href="/examples/vision/xray_classification_with_tpus/">Pneumonia Classification on TPU</a> <a class="nav-sublink2" href="/examples/vision/cct/">Compact Convolutional Transformers</a> <a class="nav-sublink2" href="/examples/vision/convmixer/">Image classification with ConvMixer</a> <a class="nav-sublink2" href="/examples/vision/eanet/">Image classification with EANet (External Attention Transformer)</a> <a class="nav-sublink2" href="/examples/vision/involution/">Involutional neural networks</a> <a class="nav-sublink2" href="/examples/vision/perceiver_image_classification/">Image classification with Perceiver</a> <a class="nav-sublink2" href="/examples/vision/reptile/">Few-Shot learning with Reptile</a> <a class="nav-sublink2" href="/examples/vision/semisupervised_simclr/">Semi-supervised image classification using contrastive pretraining with SimCLR</a> <a class="nav-sublink2" href="/examples/vision/swin_transformers/">Image classification with Swin Transformers</a> <a class="nav-sublink2" href="/examples/vision/vit_small_ds/">Train a Vision Transformer on small datasets</a> <a class="nav-sublink2" href="/examples/vision/shiftvit/">A Vision Transformer without Attention</a> <a class="nav-sublink2" href="/examples/vision/image_classification_using_global_context_vision_transformer/">Image Classification using Global Context Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/oxford_pets_image_segmentation/">Image segmentation with a U-Net-like architecture</a> <a class="nav-sublink2" href="/examples/vision/deeplabv3_plus/">Multiclass semantic segmentation using DeepLabV3+</a> <a class="nav-sublink2" href="/examples/vision/basnet_segmentation/">Highly accurate boundaries segmentation using BASNet</a> <a class="nav-sublink2" href="/examples/vision/fully_convolutional_network/">Image Segmentation using Composable Fully-Convolutional Networks</a> <a class="nav-sublink2" href="/examples/vision/retinanet/">Object Detection with RetinaNet</a> <a class="nav-sublink2" href="/examples/vision/keypoint_detection/">Keypoint Detection with Transfer Learning</a> <a class="nav-sublink2" href="/examples/vision/object_detection_using_vision_transformer/">Object detection with Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/3D_image_classification/">3D image classification from CT scans</a> <a class="nav-sublink2" href="/examples/vision/depth_estimation/">Monocular depth estimation</a> <a class="nav-sublink2" href="/examples/vision/nerf/">3D volumetric rendering with NeRF</a> <a class="nav-sublink2" href="/examples/vision/pointnet_segmentation/">Point cloud segmentation with PointNet</a> <a class="nav-sublink2" href="/examples/vision/pointnet/">Point cloud classification</a> <a class="nav-sublink2" href="/examples/vision/captcha_ocr/">OCR model for reading Captchas</a> <a class="nav-sublink2" href="/examples/vision/handwriting_recognition/">Handwriting recognition</a> <a class="nav-sublink2" href="/examples/vision/autoencoder/">Convolutional autoencoder for image denoising</a> <a class="nav-sublink2" href="/examples/vision/mirnet/">Low-light image enhancement using MIRNet</a> <a class="nav-sublink2" href="/examples/vision/super_resolution_sub_pixel/">Image Super-Resolution using an Efficient Sub-Pixel CNN</a> <a class="nav-sublink2 active" href="/examples/vision/edsr/">Enhanced Deep Residual Networks for single-image super-resolution</a> <a class="nav-sublink2" href="/examples/vision/zero_dce/">Zero-DCE for low-light image enhancement</a> <a class="nav-sublink2" href="/examples/vision/cutmix/">CutMix data augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/mixup/">MixUp augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/randaugment/">RandAugment for Image Classification for Improved Robustness</a> <a class="nav-sublink2" href="/examples/vision/image_captioning/">Image captioning</a> <a class="nav-sublink2" href="/examples/vision/nl_image_search/">Natural language image search with a Dual Encoder</a> <a class="nav-sublink2" href="/examples/vision/visualizing_what_convnets_learn/">Visualizing what convnets learn</a> <a class="nav-sublink2" href="/examples/vision/integrated_gradients/">Model interpretability with Integrated Gradients</a> <a class="nav-sublink2" href="/examples/vision/probing_vits/">Investigating Vision Transformer representations</a> <a class="nav-sublink2" href="/examples/vision/grad_cam/">Grad-CAM class activation visualization</a> <a class="nav-sublink2" href="/examples/vision/near_dup_search/">Near-duplicate image search</a> <a class="nav-sublink2" href="/examples/vision/semantic_image_clustering/">Semantic Image Clustering</a> <a class="nav-sublink2" href="/examples/vision/siamese_contrastive/">Image similarity estimation using a Siamese Network with a contrastive loss</a> <a class="nav-sublink2" href="/examples/vision/siamese_network/">Image similarity estimation using a Siamese Network with a triplet loss</a> <a class="nav-sublink2" href="/examples/vision/metric_learning/">Metric learning for image similarity search</a> <a class="nav-sublink2" href="/examples/vision/metric_learning_tf_similarity/">Metric learning for image similarity search using TensorFlow Similarity</a> <a class="nav-sublink2" href="/examples/vision/nnclr/">Self-supervised contrastive learning with NNCLR</a> <a class="nav-sublink2" href="/examples/vision/video_classification/">Video Classification with a CNN-RNN Architecture</a> <a class="nav-sublink2" href="/examples/vision/conv_lstm/">Next-Frame Video Prediction with Convolutional LSTMs</a> <a class="nav-sublink2" href="/examples/vision/video_transformers/">Video Classification with Transformers</a> <a class="nav-sublink2" href="/examples/vision/vivit/">Video Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/bit/">Image Classification using BigTransfer (BiT)</a> <a class="nav-sublink2" href="/examples/vision/gradient_centralization/">Gradient Centralization for Better Training Performance</a> <a class="nav-sublink2" href="/examples/vision/token_learner/">Learning to tokenize in Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/knowledge_distillation/">Knowledge Distillation</a> <a class="nav-sublink2" href="/examples/vision/fixres/">FixRes: Fixing train-test resolution discrepancy</a> <a class="nav-sublink2" href="/examples/vision/cait/">Class Attention Image Transformers with LayerScale</a> <a class="nav-sublink2" href="/examples/vision/patch_convnet/">Augmenting convnets with aggregated attention</a> <a class="nav-sublink2" href="/examples/vision/learnable_resizer/">Learning to Resize</a> <a class="nav-sublink2" href="/examples/vision/adamatch/">Semi-supervision and domain adaptation with AdaMatch</a> <a class="nav-sublink2" href="/examples/vision/barlow_twins/">Barlow Twins for Contrastive SSL</a> <a class="nav-sublink2" href="/examples/vision/consistency_training/">Consistency training with supervision</a> <a class="nav-sublink2" href="/examples/vision/deit/">Distilling Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/focal_modulation_network/">Focal Modulation: A replacement for Self-Attention</a> <a class="nav-sublink2" href="/examples/vision/forwardforward/">Using the Forward-Forward Algorithm for Image Classification</a> <a class="nav-sublink2" href="/examples/vision/masked_image_modeling/">Masked image modeling with Autoencoders</a> <a class="nav-sublink2" href="/examples/vision/sam/">Segment Anything Model with 🤗Transformers</a> <a class="nav-sublink2" href="/examples/vision/segformer/">Semantic segmentation with SegFormer and Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/vision/simsiam/">Self-supervised contrastive learning with SimSiam</a> <a class="nav-sublink2" href="/examples/vision/supervised-contrastive-learning/">Supervised Contrastive Learning</a> <a class="nav-sublink2" href="/examples/vision/temporal_latent_bottleneck/">When Recurrence meets Transformers</a> <a class="nav-sublink2" href="/examples/vision/yolov8/">Efficient Object Detection with YOLOV8 and KerasCV</a> <a class="nav-sublink" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparam Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/vision/'>Computer Vision</a> / Enhanced Deep Residual Networks for single-image super-resolution </div> <div class='k-content'> <h1 id="enhanced-deep-residual-networks-for-singleimage-superresolution">Enhanced Deep Residual Networks for single-image super-resolution</h1> Author: Gitesh Chawda Date created: 2022/04/07 Last modified: 2024/08/27 Description: Training an EDSR model on the DIV2K Dataset. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/edsr.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/vision/edsr.py">GitHub source</a> <hr /> <h2 id="introduction">Introduction</h2> In this example, we implement <a href="https://arxiv.org/abs/1707.02921">Enhanced Deep Residual Networks for Single Image Super-Resolution (EDSR)</a> by Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. The EDSR architecture is based on the SRResNet architecture and consists of multiple residual blocks. It uses constant scaling layers instead of batch normalization layers to produce consistent results (input and output have similar distributions, thus normalizing intermediate features may not be desirable). Instead of using a L2 loss (mean squared error), the authors employed an L1 loss (mean absolute error), which performs better empirically. Our implementation only includes 16 residual blocks with 64 channels. Alternatively, as shown in the Keras example <a href="https://keras.io/examples/vision/super_resolution_sub_pixel/#image-superresolution-using-an-efficient-subpixel-cnn">Image Super-Resolution using an Efficient Sub-Pixel CNN</a>, you can do super-resolution using an ESPCN Model. According to the survey paper, EDSR is one of the top-five best-performing super-resolution methods based on PSNR scores. However, it has more parameters and requires more computational power than other approaches. It has a PSNR value (≈34db) that is slightly higher than ESPCN (≈32db). As per the survey paper, EDSR performs better than ESPCN. Paper: <a href="https://arxiv.org/abs/2102.09351">A comprehensive review of deep learning based single image super-resolution</a> Comparison Graph: <img src="https://dfzljdn9uc3pi.cloudfront.net/2021/cs-621/1/fig-11-2x.jpg" width="500" /> <hr /> <h2 id="imports">Imports</h2> <div class="codehilite"><pre><code>import os os.environ["KERAS_BACKEND"] = "tensorflow" import numpy as np import tensorflow as tf import tensorflow_datasets as tfds import matplotlib.pyplot as plt import keras from keras import layers from keras import ops AUTOTUNE = tf.data.AUTOTUNE </code></pre></div> <hr /> <h2 id="download-the-training-dataset">Download the training dataset</h2> We use the DIV2K Dataset, a prominent single-image super-resolution dataset with 1,000 images of scenes with various sorts of degradations, divided into 800 images for training, 100 images for validation, and 100 images for testing. We use 4x bicubic downsampled images as our "low quality" reference. <div class="codehilite"><pre><code># Download DIV2K from TF Datasets # Using bicubic 4x degradation type div2k_data = tfds.image.Div2k(config="bicubic_x4") div2k_data.download_and_prepare() # Taking train data from div2k_data object train = div2k_data.as_dataset(split="train", as_supervised=True) train_cache = train.cache() # Validation data val = div2k_data.as_dataset(split="validation", as_supervised=True) val_cache = val.cache() </code></pre></div> <hr /> <h2 id="flip-crop-and-resize-images">Flip, crop and resize images</h2> <div class="codehilite"><pre><code>def flip_left_right(lowres_img, highres_img): """Flips Images to left and right.""" # Outputs random values from a uniform distribution in between 0 to 1 rn = keras.random.uniform(shape=(), maxval=1) # If rn is less than 0.5 it returns original lowres_img and highres_img # If rn is greater than 0.5 it returns flipped image return ops.cond( rn < 0.5, lambda: (lowres_img, highres_img), lambda: ( ops.flip(lowres_img), ops.flip(highres_img), ), ) def random_rotate(lowres_img, highres_img): """Rotates Images by 90 degrees.""" # Outputs random values from uniform distribution in between 0 to 4 rn = ops.cast( keras.random.uniform(shape=(), maxval=4, dtype="float32"), dtype="int32" ) # Here rn signifies number of times the image(s) are rotated by 90 degrees return tf.image.rot90(lowres_img, rn), tf.image.rot90(highres_img, rn) def random_crop(lowres_img, highres_img, hr_crop_size=96, scale=4): """Crop images. low resolution images: 24x24 high resolution images: 96x96 """ lowres_crop_size = hr_crop_size // scale # 96//4=24 lowres_img_shape = ops.shape(lowres_img)[:2] # (height,width) lowres_width = ops.cast( keras.random.uniform( shape=(), maxval=lowres_img_shape[1] - lowres_crop_size + 1, dtype="float32" ), dtype="int32", ) lowres_height = ops.cast( keras.random.uniform( shape=(), maxval=lowres_img_shape[0] - lowres_crop_size + 1, dtype="float32" ), dtype="int32", ) highres_width = lowres_width * scale highres_height = lowres_height * scale lowres_img_cropped = lowres_img[ lowres_height : lowres_height + lowres_crop_size, lowres_width : lowres_width + lowres_crop_size, ] # 24x24 highres_img_cropped = highres_img[ highres_height : highres_height + hr_crop_size, highres_width : highres_width + hr_crop_size, ] # 96x96 return lowres_img_cropped, highres_img_cropped </code></pre></div> <hr /> <h2 id="tfdatadataset">Prepare a <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset"><code>tf.data.Dataset</code></a> object</h2> We augment the training data with random horizontal flips and 90 rotations. As low resolution images, we use 24x24 RGB input patches. <div class="codehilite"><pre><code>def dataset_object(dataset_cache, training=True): ds = dataset_cache ds = ds.map( lambda lowres, highres: random_crop(lowres, highres, scale=4), num_parallel_calls=AUTOTUNE, ) if training: ds = ds.map(random_rotate, num_parallel_calls=AUTOTUNE) ds = ds.map(flip_left_right, num_parallel_calls=AUTOTUNE) # Batching Data ds = ds.batch(16) if training: # Repeating Data, so that cardinality if dataset becomes infinte ds = ds.repeat() # prefetching allows later images to be prepared while the current image is being processed ds = ds.prefetch(buffer_size=AUTOTUNE) return ds train_ds = dataset_object(train_cache, training=True) val_ds = dataset_object(val_cache, training=False) </code></pre></div> <hr /> <h2 id="visualize-the-data">Visualize the data</h2> Let's visualize a few sample images: <div class="codehilite"><pre><code>lowres, highres = next(iter(train_ds)) # High Resolution Images plt.figure(figsize=(10, 10)) for i in range(9): ax = plt.subplot(3, 3, i + 1) plt.imshow(highres[i].numpy().astype("uint8")) plt.title(highres[i].shape) plt.axis("off") # Low Resolution Images plt.figure(figsize=(10, 10)) for i in range(9): ax = plt.subplot(3, 3, i + 1) plt.imshow(lowres[i].numpy().astype("uint8")) plt.title(lowres[i].shape) plt.axis("off") def PSNR(super_resolution, high_resolution): """Compute the peak signal-to-noise ratio, measures quality of image.""" # Max value of pixel is 255 psnr_value = tf.image.psnr(high_resolution, super_resolution, max_val=255)[0] return psnr_value </code></pre></div> <img alt="png" src="/img/examples/vision/edsr/edsr_11_0.png" /> <img alt="png" src="/img/examples/vision/edsr/edsr_11_1.png" /> <hr /> <h2 id="build-the-model">Build the model</h2> In the paper, the authors train three models: EDSR, MDSR, and a baseline model. In this code example, we only train the baseline model. <h3 id="comparison-with-model-with-three-residual-blocks">Comparison with model with three residual blocks</h3> The residual block design of EDSR differs from that of ResNet. Batch normalization layers have been removed (together with the final ReLU activation): since batch normalization layers normalize the features, they hurt output value range flexibility. It is thus better to remove them. Further, it also helps reduce the amount of GPU RAM required by the model, since the batch normalization layers consume the same amount of memory as the preceding convolutional layers. <img src="https://miro.medium.com/max/1050/1*EPviXGqlGWotVtV2gqVvNg.png" width="500" /> <div class="codehilite"><pre><code>class EDSRModel(keras.Model): def train_step(self, data): # Unpack the data. Its structure depends on your model and # on what you pass to `fit()`. x, y = data with tf.GradientTape() as tape: y_pred = self(x, training=True) # Forward pass # Compute the loss value # (the loss function is configured in `compile()`) loss = self.compiled_loss(y, y_pred, regularization_losses=self.losses) # Compute gradients trainable_vars = self.trainable_variables gradients = tape.gradient(loss, trainable_vars) # Update weights self.optimizer.apply_gradients(zip(gradients, trainable_vars)) # Update metrics (includes the metric that tracks the loss) self.compiled_metrics.update_state(y, y_pred) # Return a dict mapping metric names to current value return {m.name: m.result() for m in self.metrics} def predict_step(self, x): # Adding dummy dimension using tf.expand_dims and converting to float32 using tf.cast x = ops.cast(tf.expand_dims(x, axis=0), dtype="float32") # Passing low resolution image to model super_resolution_img = self(x, training=False) # Clips the tensor from min(0) to max(255) super_resolution_img = ops.clip(super_resolution_img, 0, 255) # Rounds the values of a tensor to the nearest integer super_resolution_img = ops.round(super_resolution_img) # Removes dimensions of size 1 from the shape of a tensor and converting to uint8 super_resolution_img = ops.squeeze( ops.cast(super_resolution_img, dtype="uint8"), axis=0 ) return super_resolution_img # Residual Block def ResBlock(inputs): x = layers.Conv2D(64, 3, padding="same", activation="relu")(inputs) x = layers.Conv2D(64, 3, padding="same")(x) x = layers.Add()([inputs, x]) return x # Upsampling Block def Upsampling(inputs, factor=2, **kwargs): x = layers.Conv2D(64 * (factor**2), 3, padding="same", **kwargs)(inputs) x = layers.Lambda(lambda x: tf.nn.depth_to_space(x, block_size=factor))(x) x = layers.Conv2D(64 * (factor**2), 3, padding="same", **kwargs)(x) x = layers.Lambda(lambda x: tf.nn.depth_to_space(x, block_size=factor))(x) return x def make_model(num_filters, num_of_residual_blocks): # Flexible Inputs to input_layer input_layer = layers.Input(shape=(None, None, 3)) # Scaling Pixel Values x = layers.Rescaling(scale=1.0 / 255)(input_layer) x = x_new = layers.Conv2D(num_filters, 3, padding="same")(x) # 16 residual blocks for _ in range(num_of_residual_blocks): x_new = ResBlock(x_new) x_new = layers.Conv2D(num_filters, 3, padding="same")(x_new) x = layers.Add()([x, x_new]) x = Upsampling(x) x = layers.Conv2D(3, 3, padding="same")(x) output_layer = layers.Rescaling(scale=255)(x) return EDSRModel(input_layer, output_layer) model = make_model(num_filters=64, num_of_residual_blocks=16) </code></pre></div> <hr /> <h2 id="train-the-model">Train the model</h2> <div class="codehilite"><pre><code># Using adam optimizer with initial learning rate as 1e-4, changing learning rate after 5000 steps to 5e-5 optim_edsr = keras.optimizers.Adam( learning_rate=keras.optimizers.schedules.PiecewiseConstantDecay( boundaries=[5000], values=[1e-4, 5e-5] ) ) # Compiling model with loss as mean absolute error(L1 Loss) and metric as psnr model.compile(optimizer=optim_edsr, loss="mae", metrics=[PSNR]) # Training for more epochs will improve results model.fit(train_ds, epochs=100, steps_per_epoch=200, validation_data=val_ds) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Epoch 1/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 117s 472ms/step - psnr: 8.7874 - loss: 85.1546 - val_loss: 17.4624 - val_psnr: 8.7008 Epoch 10/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 58s 288ms/step - psnr: 8.9519 - loss: 94.4611 - val_loss: 8.6002 - val_psnr: 6.4303 Epoch 20/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 52s 261ms/step - psnr: 8.5120 - loss: 95.5767 - val_loss: 8.7330 - val_psnr: 6.3106 Epoch 30/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 53s 262ms/step - psnr: 8.6051 - loss: 96.1541 - val_loss: 7.5442 - val_psnr: 7.9715 Epoch 40/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 53s 263ms/step - psnr: 8.7405 - loss: 96.8159 - val_loss: 7.2734 - val_psnr: 7.6312 Epoch 50/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 52s 259ms/step - psnr: 8.7648 - loss: 95.7817 - val_loss: 8.1772 - val_psnr: 7.1330 Epoch 60/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 53s 264ms/step - psnr: 8.8651 - loss: 95.4793 - val_loss: 7.6550 - val_psnr: 7.2298 Epoch 70/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 53s 263ms/step - psnr: 8.8489 - loss: 94.5993 - val_loss: 7.4607 - val_psnr: 6.6841 Epoch 80/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 53s 263ms/step - psnr: 8.3046 - loss: 97.3796 - val_loss: 8.1050 - val_psnr: 8.0714 Epoch 90/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 53s 264ms/step - psnr: 7.9295 - loss: 96.0314 - val_loss: 7.1515 - val_psnr: 6.8712 Epoch 100/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 53s 263ms/step - psnr: 8.1666 - loss: 94.9792 - val_loss: 6.6524 - val_psnr: 6.5423 <keras.src.callbacks.history.History at 0x7fc1e8dd6890> </code></pre></div> </div> <hr /> <h2 id="run-inference-on-new-images-and-plot-the-results">Run inference on new images and plot the results</h2> <div class="codehilite"><pre><code>def plot_results(lowres, preds): """ Displays low resolution image and super resolution image """ plt.figure(figsize=(24, 14)) plt.subplot(132), plt.imshow(lowres), plt.title("Low resolution") plt.subplot(133), plt.imshow(preds), plt.title("Prediction") plt.show() for lowres, highres in val.take(10): lowres = tf.image.random_crop(lowres, (150, 150, 3)) preds = model.predict_step(lowres) plot_results(lowres, preds) </code></pre></div> <img alt="png" src="/img/examples/vision/edsr/edsr_17_0.png" /> <img alt="png" src="/img/examples/vision/edsr/edsr_17_1.png" /> <img alt="png" src="/img/examples/vision/edsr/edsr_17_2.png" /> <img alt="png" src="/img/examples/vision/edsr/edsr_17_3.png" /> <hr /> <h2 id="final-remarks">Final remarks</h2> In this example, we implemented the EDSR model (Enhanced Deep Residual Networks for Single Image Super-Resolution). You could improve the model accuracy by training the model for more epochs, as well as training the model with a wider variety of inputs with mixed downgrading factors, so as to be able to handle a greater range of real-world images. You could also improve on the given baseline EDSR model by implementing EDSR+, or MDSR( Multi-Scale super-resolution) and MDSR+, which were proposed in the same paper. </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#enhanced-deep-residual-networks-for-singleimage-superresolution'>Enhanced Deep Residual Networks for single-image super-resolution</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#imports'>Imports</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#download-the-training-dataset'>Download the training dataset</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#flip-crop-and-resize-images'>Flip, crop and resize images</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#prepare-a-tfdatadataset-object'>Prepare a <code>tf.data.Dataset</code> object</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#visualize-the-data'>Visualize the data</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#build-the-model'>Build the model</a> </div> <div class='k-outline-depth-3'> <a href='#comparison-with-model-with-three-residual-blocks'>Comparison with model with three residual blocks</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#train-the-model'>Train the model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#run-inference-on-new-images-and-plot-the-results'>Run inference on new images and plot the results</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#final-remarks'>Final remarks</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>

CINXE.COM

Enhanced Deep Residual Networks for single-image super-resolution