CutMix data augmentation for image classification

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/vision/cutmix/" />  <meta property="og:title" content="Keras documentation: CutMix data augmentation for image classification"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: CutMix data augmentation for image classification"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>CutMix data augmentation for image classification</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink active" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink2" href="/examples/vision/image_classification_from_scratch/">Image classification from scratch</a> <a class="nav-sublink2" href="/examples/vision/mnist_convnet/">Simple MNIST convnet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_efficientnet_fine_tuning/">Image classification via fine-tuning with EfficientNet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_with_vision_transformer/">Image classification with Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/attention_mil_classification/">Classification using Attention-based Deep Multiple Instance Learning</a> <a class="nav-sublink2" href="/examples/vision/mlp_image_classification/">Image classification with modern MLP models</a> <a class="nav-sublink2" href="/examples/vision/mobilevit/">A mobile-friendly Transformer-based model for image classification</a> <a class="nav-sublink2" href="/examples/vision/xray_classification_with_tpus/">Pneumonia Classification on TPU</a> <a class="nav-sublink2" href="/examples/vision/cct/">Compact Convolutional Transformers</a> <a class="nav-sublink2" href="/examples/vision/convmixer/">Image classification with ConvMixer</a> <a class="nav-sublink2" href="/examples/vision/eanet/">Image classification with EANet (External Attention Transformer)</a> <a class="nav-sublink2" href="/examples/vision/involution/">Involutional neural networks</a> <a class="nav-sublink2" href="/examples/vision/perceiver_image_classification/">Image classification with Perceiver</a> <a class="nav-sublink2" href="/examples/vision/reptile/">Few-Shot learning with Reptile</a> <a class="nav-sublink2" href="/examples/vision/semisupervised_simclr/">Semi-supervised image classification using contrastive pretraining with SimCLR</a> <a class="nav-sublink2" href="/examples/vision/swin_transformers/">Image classification with Swin Transformers</a> <a class="nav-sublink2" href="/examples/vision/vit_small_ds/">Train a Vision Transformer on small datasets</a> <a class="nav-sublink2" href="/examples/vision/shiftvit/">A Vision Transformer without Attention</a> <a class="nav-sublink2" href="/examples/vision/image_classification_using_global_context_vision_transformer/">Image Classification using Global Context Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/oxford_pets_image_segmentation/">Image segmentation with a U-Net-like architecture</a> <a class="nav-sublink2" href="/examples/vision/deeplabv3_plus/">Multiclass semantic segmentation using DeepLabV3+</a> <a class="nav-sublink2" href="/examples/vision/basnet_segmentation/">Highly accurate boundaries segmentation using BASNet</a> <a class="nav-sublink2" href="/examples/vision/fully_convolutional_network/">Image Segmentation using Composable Fully-Convolutional Networks</a> <a class="nav-sublink2" href="/examples/vision/retinanet/">Object Detection with RetinaNet</a> <a class="nav-sublink2" href="/examples/vision/keypoint_detection/">Keypoint Detection with Transfer Learning</a> <a class="nav-sublink2" href="/examples/vision/object_detection_using_vision_transformer/">Object detection with Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/3D_image_classification/">3D image classification from CT scans</a> <a class="nav-sublink2" href="/examples/vision/depth_estimation/">Monocular depth estimation</a> <a class="nav-sublink2" href="/examples/vision/nerf/">3D volumetric rendering with NeRF</a> <a class="nav-sublink2" href="/examples/vision/pointnet_segmentation/">Point cloud segmentation with PointNet</a> <a class="nav-sublink2" href="/examples/vision/pointnet/">Point cloud classification</a> <a class="nav-sublink2" href="/examples/vision/captcha_ocr/">OCR model for reading Captchas</a> <a class="nav-sublink2" href="/examples/vision/handwriting_recognition/">Handwriting recognition</a> <a class="nav-sublink2" href="/examples/vision/autoencoder/">Convolutional autoencoder for image denoising</a> <a class="nav-sublink2" href="/examples/vision/mirnet/">Low-light image enhancement using MIRNet</a> <a class="nav-sublink2" href="/examples/vision/super_resolution_sub_pixel/">Image Super-Resolution using an Efficient Sub-Pixel CNN</a> <a class="nav-sublink2" href="/examples/vision/edsr/">Enhanced Deep Residual Networks for single-image super-resolution</a> <a class="nav-sublink2" href="/examples/vision/zero_dce/">Zero-DCE for low-light image enhancement</a> <a class="nav-sublink2 active" href="/examples/vision/cutmix/">CutMix data augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/mixup/">MixUp augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/randaugment/">RandAugment for Image Classification for Improved Robustness</a> <a class="nav-sublink2" href="/examples/vision/image_captioning/">Image captioning</a> <a class="nav-sublink2" href="/examples/vision/nl_image_search/">Natural language image search with a Dual Encoder</a> <a class="nav-sublink2" href="/examples/vision/visualizing_what_convnets_learn/">Visualizing what convnets learn</a> <a class="nav-sublink2" href="/examples/vision/integrated_gradients/">Model interpretability with Integrated Gradients</a> <a class="nav-sublink2" href="/examples/vision/probing_vits/">Investigating Vision Transformer representations</a> <a class="nav-sublink2" href="/examples/vision/grad_cam/">Grad-CAM class activation visualization</a> <a class="nav-sublink2" href="/examples/vision/near_dup_search/">Near-duplicate image search</a> <a class="nav-sublink2" href="/examples/vision/semantic_image_clustering/">Semantic Image Clustering</a> <a class="nav-sublink2" href="/examples/vision/siamese_contrastive/">Image similarity estimation using a Siamese Network with a contrastive loss</a> <a class="nav-sublink2" href="/examples/vision/siamese_network/">Image similarity estimation using a Siamese Network with a triplet loss</a> <a class="nav-sublink2" href="/examples/vision/metric_learning/">Metric learning for image similarity search</a> <a class="nav-sublink2" href="/examples/vision/metric_learning_tf_similarity/">Metric learning for image similarity search using TensorFlow Similarity</a> <a class="nav-sublink2" href="/examples/vision/nnclr/">Self-supervised contrastive learning with NNCLR</a> <a class="nav-sublink2" href="/examples/vision/video_classification/">Video Classification with a CNN-RNN Architecture</a> <a class="nav-sublink2" href="/examples/vision/conv_lstm/">Next-Frame Video Prediction with Convolutional LSTMs</a> <a class="nav-sublink2" href="/examples/vision/video_transformers/">Video Classification with Transformers</a> <a class="nav-sublink2" href="/examples/vision/vivit/">Video Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/bit/">Image Classification using BigTransfer (BiT)</a> <a class="nav-sublink2" href="/examples/vision/gradient_centralization/">Gradient Centralization for Better Training Performance</a> <a class="nav-sublink2" href="/examples/vision/token_learner/">Learning to tokenize in Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/knowledge_distillation/">Knowledge Distillation</a> <a class="nav-sublink2" href="/examples/vision/fixres/">FixRes: Fixing train-test resolution discrepancy</a> <a class="nav-sublink2" href="/examples/vision/cait/">Class Attention Image Transformers with LayerScale</a> <a class="nav-sublink2" href="/examples/vision/patch_convnet/">Augmenting convnets with aggregated attention</a> <a class="nav-sublink2" href="/examples/vision/learnable_resizer/">Learning to Resize</a> <a class="nav-sublink2" href="/examples/vision/adamatch/">Semi-supervision and domain adaptation with AdaMatch</a> <a class="nav-sublink2" href="/examples/vision/barlow_twins/">Barlow Twins for Contrastive SSL</a> <a class="nav-sublink2" href="/examples/vision/consistency_training/">Consistency training with supervision</a> <a class="nav-sublink2" href="/examples/vision/deit/">Distilling Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/focal_modulation_network/">Focal Modulation: A replacement for Self-Attention</a> <a class="nav-sublink2" href="/examples/vision/forwardforward/">Using the Forward-Forward Algorithm for Image Classification</a> <a class="nav-sublink2" href="/examples/vision/masked_image_modeling/">Masked image modeling with Autoencoders</a> <a class="nav-sublink2" href="/examples/vision/sam/">Segment Anything Model with 🤗Transformers</a> <a class="nav-sublink2" href="/examples/vision/segformer/">Semantic segmentation with SegFormer and Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/vision/simsiam/">Self-supervised contrastive learning with SimSiam</a> <a class="nav-sublink2" href="/examples/vision/supervised-contrastive-learning/">Supervised Contrastive Learning</a> <a class="nav-sublink2" href="/examples/vision/temporal_latent_bottleneck/">When Recurrence meets Transformers</a> <a class="nav-sublink2" href="/examples/vision/yolov8/">Efficient Object Detection with YOLOV8 and KerasCV</a> <a class="nav-sublink" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparameter Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> <a class="nav-link" href="/keras_cv/" role="tab" aria-selected="">KerasCV: Computer Vision Workflows</a> <a class="nav-link" href="/keras_nlp/" role="tab" aria-selected="">KerasNLP: Natural Language Workflows</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/vision/'>Computer Vision</a> / CutMix data augmentation for image classification </div> <div class='k-content'> <h1 id="cutmix-data-augmentation-for-image-classification">CutMix data augmentation for image classification</h1> Author: <a href="https://twitter.com/sayannath2350">Sayan Nath</a> Date created: 2021/06/08 Last modified: 2023/11/14 Description: Data augmentation with CutMix for image classification on CIFAR-10. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/cutmix.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/vision/cutmix.py">GitHub source</a> <hr /> <h2 id="introduction">Introduction</h2> CutMix is a data augmentation technique that addresses the issue of information loss and inefficiency present in regional dropout strategies. Instead of removing pixels and filling them with black or grey pixels or Gaussian noise, you replace the removed regions with a patch from another image, while the ground truth labels are mixed proportionally to the number of pixels of combined images. CutMix was proposed in <a href="https://arxiv.org/abs/1905.04899">CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features</a> (Yun et al., 2019) It's implemented via the following formulas: <img src="https://i.imgur.com/cGvd13V.png" width="200"/> where <code>M</code> is the binary mask which indicates the cutout and the fill-in regions from the two randomly drawn images and <code>λ</code> (in <code>[0, 1]</code>) is drawn from a <a href="https://en.wikipedia.org/wiki/Beta_distribution"><code>Beta(α, α)</code> distribution</a> The coordinates of bounding boxes are: <img src="https://i.imgur.com/eNisep4.png" width="150"/> which indicates the cutout and fill-in regions in case of the images. The bounding box sampling is represented by: <img src="https://i.imgur.com/Snph9aj.png" width="200"/> where <code>rx, ry</code> are randomly drawn from a uniform distribution with upper bound. <hr /> <h2 id="setup">Setup</h2> <div class="codehilite"><pre><code>import numpy as np import keras import matplotlib.pyplot as plt from keras import layers # TF imports related to tf.data preprocessing from tensorflow import clip_by_value from tensorflow import data as tf_data from tensorflow import image as tf_image from tensorflow import random as tf_random keras.utils.set_random_seed(42) </code></pre></div> <hr /> <h2 id="load-the-cifar10-dataset">Load the CIFAR-10 dataset</h2> In this example, we will use the <a href="https://www.cs.toronto.edu/~kriz/cifar.html">CIFAR-10 image classification dataset</a>. <div class="codehilite"><pre><code>(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data() y_train = keras.utils.to_categorical(y_train, num_classes=10) y_test = keras.utils.to_categorical(y_test, num_classes=10) print(x_train.shape) print(y_train.shape) print(x_test.shape) print(y_test.shape) class_names = [ "Airplane", "Automobile", "Bird", "Cat", "Deer", "Dog", "Frog", "Horse", "Ship", "Truck", ] </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>(50000, 32, 32, 3) (50000, 10) (10000, 32, 32, 3) (10000, 10) </code></pre></div> </div> <hr /> <h2 id="define-hyperparameters">Define hyperparameters</h2> <div class="codehilite"><pre><code>AUTO = tf_data.AUTOTUNE BATCH_SIZE = 32 IMG_SIZE = 32 </code></pre></div> <hr /> <h2 id="define-the-image-preprocessing-function">Define the image preprocessing function</h2> <div class="codehilite"><pre><code>def preprocess_image(image, label): image = tf_image.resize(image, (IMG_SIZE, IMG_SIZE)) image = tf_image.convert_image_dtype(image, "float32") / 255.0 label = keras.ops.cast(label, dtype="float32") return image, label </code></pre></div> <hr /> <h2 id="convert-the-data-into-tensorflow-dataset-objects">Convert the data into TensorFlow <code>Dataset</code> objects</h2> <div class="codehilite"><pre><code>train_ds_one = ( tf_data.Dataset.from_tensor_slices((x_train, y_train)) .shuffle(1024) .map(preprocess_image, num_parallel_calls=AUTO) ) train_ds_two = ( tf_data.Dataset.from_tensor_slices((x_train, y_train)) .shuffle(1024) .map(preprocess_image, num_parallel_calls=AUTO) ) train_ds_simple = tf_data.Dataset.from_tensor_slices((x_train, y_train)) test_ds = tf_data.Dataset.from_tensor_slices((x_test, y_test)) train_ds_simple = ( train_ds_simple.map(preprocess_image, num_parallel_calls=AUTO) .batch(BATCH_SIZE) .prefetch(AUTO) ) # Combine two shuffled datasets from the same training data. train_ds = tf_data.Dataset.zip((train_ds_one, train_ds_two)) test_ds = ( test_ds.map(preprocess_image, num_parallel_calls=AUTO) .batch(BATCH_SIZE) .prefetch(AUTO) ) </code></pre></div> <hr /> <h2 id="define-the-cutmix-data-augmentation-function">Define the CutMix data augmentation function</h2> The CutMix function takes two <code>image</code> and <code>label</code> pairs to perform the augmentation. It samples <code>λ(l)</code> from the <a href="https://en.wikipedia.org/wiki/Beta_distribution">Beta distribution</a> and returns a bounding box from <code>get_box</code> function. We then crop the second image (<code>image2</code>) and pad this image in the final padded image at the same location. <div class="codehilite"><pre><code>def sample_beta_distribution(size, concentration_0=0.2, concentration_1=0.2): gamma_1_sample = tf_random.gamma(shape=[size], alpha=concentration_1) gamma_2_sample = tf_random.gamma(shape=[size], alpha=concentration_0) return gamma_1_sample / (gamma_1_sample + gamma_2_sample) def get_box(lambda_value): cut_rat = keras.ops.sqrt(1.0 - lambda_value) cut_w = IMG_SIZE * cut_rat # rw cut_w = keras.ops.cast(cut_w, "int32") cut_h = IMG_SIZE * cut_rat # rh cut_h = keras.ops.cast(cut_h, "int32") cut_x = keras.random.uniform((1,), minval=0, maxval=IMG_SIZE) # rx cut_x = keras.ops.cast(cut_x, "int32") cut_y = keras.random.uniform((1,), minval=0, maxval=IMG_SIZE) # ry cut_y = keras.ops.cast(cut_y, "int32") boundaryx1 = clip_by_value(cut_x[0] - cut_w // 2, 0, IMG_SIZE) boundaryy1 = clip_by_value(cut_y[0] - cut_h // 2, 0, IMG_SIZE) bbx2 = clip_by_value(cut_x[0] + cut_w // 2, 0, IMG_SIZE) bby2 = clip_by_value(cut_y[0] + cut_h // 2, 0, IMG_SIZE) target_h = bby2 - boundaryy1 if target_h == 0: target_h += 1 target_w = bbx2 - boundaryx1 if target_w == 0: target_w += 1 return boundaryx1, boundaryy1, target_h, target_w def cutmix(train_ds_one, train_ds_two): (image1, label1), (image2, label2) = train_ds_one, train_ds_two alpha = [0.25] beta = [0.25] # Get a sample from the Beta distribution lambda_value = sample_beta_distribution(1, alpha, beta) # Define Lambda lambda_value = lambda_value[0][0] # Get the bounding box offsets, heights and widths boundaryx1, boundaryy1, target_h, target_w = get_box(lambda_value) # Get a patch from the second image (`image2`) crop2 = tf_image.crop_to_bounding_box( image2, boundaryy1, boundaryx1, target_h, target_w ) # Pad the `image2` patch (`crop2`) with the same offset image2 = tf_image.pad_to_bounding_box( crop2, boundaryy1, boundaryx1, IMG_SIZE, IMG_SIZE ) # Get a patch from the first image (`image1`) crop1 = tf_image.crop_to_bounding_box( image1, boundaryy1, boundaryx1, target_h, target_w ) # Pad the `image1` patch (`crop1`) with the same offset img1 = tf_image.pad_to_bounding_box( crop1, boundaryy1, boundaryx1, IMG_SIZE, IMG_SIZE ) # Modify the first image by subtracting the patch from `image1` # (before applying the `image2` patch) image1 = image1 - img1 # Add the modified `image1` and `image2` together to get the CutMix image image = image1 + image2 # Adjust Lambda in accordance to the pixel ration lambda_value = 1 - (target_w * target_h) / (IMG_SIZE * IMG_SIZE) lambda_value = keras.ops.cast(lambda_value, "float32") # Combine the labels of both images label = lambda_value * label1 + (1 - lambda_value) * label2 return image, label </code></pre></div> Note: we are combining two images to create a single one. <hr /> <h2 id="visualize-the-new-dataset-after-applying-the-cutmix-augmentation">Visualize the new dataset after applying the CutMix augmentation</h2> <div class="codehilite"><pre><code># Create the new dataset using our `cutmix` utility train_ds_cmu = ( train_ds.shuffle(1024) .map(cutmix, num_parallel_calls=AUTO) .batch(BATCH_SIZE) .prefetch(AUTO) ) # Let's preview 9 samples from the dataset image_batch, label_batch = next(iter(train_ds_cmu)) plt.figure(figsize=(10, 10)) for i in range(9): ax = plt.subplot(3, 3, i + 1) plt.title(class_names[np.argmax(label_batch[i])]) plt.imshow(image_batch[i]) plt.axis("off") </code></pre></div> <img alt="png" src="/img/examples/vision/cutmix/cutmix_16_0.png" /> <hr /> <h2 id="define-a-resnet20-model">Define a ResNet-20 model</h2> <div class="codehilite"><pre><code>def resnet_layer( inputs, num_filters=16, kernel_size=3, strides=1, activation="relu", batch_normalization=True, conv_first=True, ): conv = layers.Conv2D( num_filters, kernel_size=kernel_size, strides=strides, padding="same", kernel_initializer="he_normal", kernel_regularizer=keras.regularizers.L2(1e-4), ) x = inputs if conv_first: x = conv(x) if batch_normalization: x = layers.BatchNormalization()(x) if activation is not None: x = layers.Activation(activation)(x) else: if batch_normalization: x = layers.BatchNormalization()(x) if activation is not None: x = layers.Activation(activation)(x) x = conv(x) return x def resnet_v20(input_shape, depth, num_classes=10): if (depth - 2) % 6 != 0: raise ValueError("depth should be 6n+2 (eg 20, 32, 44 in [a])") # Start model definition. num_filters = 16 num_res_blocks = int((depth - 2) / 6) inputs = layers.Input(shape=input_shape) x = resnet_layer(inputs=inputs) # Instantiate the stack of residual units for stack in range(3): for res_block in range(num_res_blocks): strides = 1 if stack > 0 and res_block == 0: # first layer but not first stack strides = 2 # downsample y = resnet_layer(inputs=x, num_filters=num_filters, strides=strides) y = resnet_layer(inputs=y, num_filters=num_filters, activation=None) if stack > 0 and res_block == 0: # first layer but not first stack # linear projection residual shortcut connection to match # changed dims x = resnet_layer( inputs=x, num_filters=num_filters, kernel_size=1, strides=strides, activation=None, batch_normalization=False, ) x = layers.add([x, y]) x = layers.Activation("relu")(x) num_filters *= 2 # Add classifier on top. # v1 does not use BN after last shortcut connection-ReLU x = layers.AveragePooling2D(pool_size=8)(x) y = layers.Flatten()(x) outputs = layers.Dense( num_classes, activation="softmax", kernel_initializer="he_normal" )(y) # Instantiate model. model = keras.Model(inputs=inputs, outputs=outputs) return model def training_model(): return resnet_v20((32, 32, 3), 20) initial_model = training_model() initial_model.save_weights("initial_weights.weights.h5") </code></pre></div> <hr /> <h2 id="train-the-model-with-the-dataset-augmented-by-cutmix">Train the model with the dataset augmented by CutMix</h2> <div class="codehilite"><pre><code>model = training_model() model.load_weights("initial_weights.weights.h5") model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"]) model.fit(train_ds_cmu, validation_data=test_ds, epochs=15) test_loss, test_accuracy = model.evaluate(test_ds) print("Test accuracy: {:.2f}%".format(test_accuracy * 100)) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Epoch 1/15 10/1563 [37m━━━━━━━━━━━━━━━━━━━━ 19s 13ms/step - accuracy: 0.0795 - loss: 5.3035 WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1699988196.560261 362411 device_compiler.h:187] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process. 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 64s 27ms/step - accuracy: 0.3148 - loss: 2.1918 - val_accuracy: 0.4067 - val_loss: 1.8339 Epoch 2/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 27s 17ms/step - accuracy: 0.4295 - loss: 1.9021 - val_accuracy: 0.5516 - val_loss: 1.4744 Epoch 3/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 28s 18ms/step - accuracy: 0.4883 - loss: 1.8076 - val_accuracy: 0.5305 - val_loss: 1.5067 Epoch 4/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 27s 17ms/step - accuracy: 0.5243 - loss: 1.7342 - val_accuracy: 0.6303 - val_loss: 1.2822 Epoch 5/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 27s 17ms/step - accuracy: 0.5574 - loss: 1.6614 - val_accuracy: 0.5370 - val_loss: 1.5912 Epoch 6/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 27s 17ms/step - accuracy: 0.5832 - loss: 1.6167 - val_accuracy: 0.6254 - val_loss: 1.3116 Epoch 7/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 26s 17ms/step - accuracy: 0.6045 - loss: 1.5738 - val_accuracy: 0.6101 - val_loss: 1.3408 Epoch 8/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 28s 18ms/step - accuracy: 0.6170 - loss: 1.5493 - val_accuracy: 0.6209 - val_loss: 1.2923 Epoch 9/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 29s 18ms/step - accuracy: 0.6292 - loss: 1.5299 - val_accuracy: 0.6290 - val_loss: 1.2813 Epoch 10/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 28s 18ms/step - accuracy: 0.6394 - loss: 1.5110 - val_accuracy: 0.7234 - val_loss: 1.0608 Epoch 11/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 26s 17ms/step - accuracy: 0.6467 - loss: 1.4915 - val_accuracy: 0.7498 - val_loss: 0.9854 Epoch 12/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 28s 18ms/step - accuracy: 0.6559 - loss: 1.4785 - val_accuracy: 0.6481 - val_loss: 1.2410 Epoch 13/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 26s 17ms/step - accuracy: 0.6596 - loss: 1.4656 - val_accuracy: 0.7551 - val_loss: 0.9784 Epoch 14/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 27s 17ms/step - accuracy: 0.6577 - loss: 1.4637 - val_accuracy: 0.6822 - val_loss: 1.1703 Epoch 15/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 26s 17ms/step - accuracy: 0.6702 - loss: 1.4445 - val_accuracy: 0.7108 - val_loss: 1.0805 313/313 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7140 - loss: 1.0766 Test accuracy: 71.08% </code></pre></div> </div> <hr /> <h2 id="train-the-model-using-the-original-nonaugmented-dataset">Train the model using the original non-augmented dataset</h2> <div class="codehilite"><pre><code>model = training_model() model.load_weights("initial_weights.weights.h5") model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"]) model.fit(train_ds_simple, validation_data=test_ds, epochs=15) test_loss, test_accuracy = model.evaluate(test_ds) print("Test accuracy: {:.2f}%".format(test_accuracy * 100)) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Epoch 1/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 41s 15ms/step - accuracy: 0.3943 - loss: 1.8736 - val_accuracy: 0.5359 - val_loss: 1.4376 Epoch 2/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 11s 7ms/step - accuracy: 0.6160 - loss: 1.2407 - val_accuracy: 0.5887 - val_loss: 1.4254 Epoch 3/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 11s 7ms/step - accuracy: 0.6927 - loss: 1.0448 - val_accuracy: 0.6102 - val_loss: 1.4850 Epoch 4/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.7411 - loss: 0.9222 - val_accuracy: 0.6262 - val_loss: 1.3898 Epoch 5/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.7711 - loss: 0.8439 - val_accuracy: 0.6283 - val_loss: 1.3425 Epoch 6/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.7983 - loss: 0.7886 - val_accuracy: 0.2460 - val_loss: 5.6869 Epoch 7/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 11s 7ms/step - accuracy: 0.8168 - loss: 0.7490 - val_accuracy: 0.1954 - val_loss: 21.7670 Epoch 8/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 11s 7ms/step - accuracy: 0.8113 - loss: 0.7779 - val_accuracy: 0.1027 - val_loss: 36.3144 Epoch 9/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 11s 7ms/step - accuracy: 0.6592 - loss: 1.4179 - val_accuracy: 0.1025 - val_loss: 40.0770 Epoch 10/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 8ms/step - accuracy: 0.5611 - loss: 1.9856 - val_accuracy: 0.1699 - val_loss: 40.6308 Epoch 11/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.6076 - loss: 1.7795 - val_accuracy: 0.1003 - val_loss: 63.4775 Epoch 12/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.6175 - loss: 1.8077 - val_accuracy: 0.1099 - val_loss: 21.9148 Epoch 13/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.6468 - loss: 1.6702 - val_accuracy: 0.1576 - val_loss: 72.7290 Epoch 14/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 12s 7ms/step - accuracy: 0.6437 - loss: 1.7858 - val_accuracy: 0.1000 - val_loss: 64.9249 Epoch 15/15 1563/1563 ━━━━━━━━━━━━━━━━━━━━ 13s 8ms/step - accuracy: 0.6587 - loss: 1.7587 - val_accuracy: 0.1000 - val_loss: 138.8463 313/313 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.0988 - loss: 139.3117 Test accuracy: 10.00% </code></pre></div> </div> <hr /> <h2 id="notes">Notes</h2> In this example, we trained our model for 15 epochs. In our experiment, the model with CutMix achieves a better accuracy on the CIFAR-10 dataset (77.34% in our experiment) compared to the model that doesn't use the augmentation (66.90%). You may notice it takes less time to train the model with the CutMix augmentation. You can experiment further with the CutMix technique by following the <a href="https://arxiv.org/abs/1905.04899">original paper</a>. </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#cutmix-data-augmentation-for-image-classification'>CutMix data augmentation for image classification</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#setup'>Setup</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#load-the-cifar10-dataset'>Load the CIFAR-10 dataset</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#define-hyperparameters'>Define hyperparameters</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#define-the-image-preprocessing-function'>Define the image preprocessing function</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#convert-the-data-into-tensorflow-dataset-objects'>Convert the data into TensorFlow <code>Dataset</code> objects</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#define-the-cutmix-data-augmentation-function'>Define the CutMix data augmentation function</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#visualize-the-new-dataset-after-applying-the-cutmix-augmentation'>Visualize the new dataset after applying the CutMix augmentation</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#define-a-resnet20-model'>Define a ResNet-20 model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#train-the-model-with-the-dataset-augmented-by-cutmix'>Train the model with the dataset augmented by CutMix</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#train-the-model-using-the-original-nonaugmented-dataset'>Train the model using the original non-augmented dataset</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#notes'>Notes</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>

CINXE.COM

CutMix data augmentation for image classification