CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/vision/object_detection_using_vision_transformer/" />  <meta property="og:title" content="Keras documentation: Object detection with Vision Transformers"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Object detection with Vision Transformers"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Object detection with Vision Transformers</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink active" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink2" href="/examples/vision/image_classification_from_scratch/">Image classification from scratch</a> <a class="nav-sublink2" href="/examples/vision/mnist_convnet/">Simple MNIST convnet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_efficientnet_fine_tuning/">Image classification via fine-tuning with EfficientNet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_with_vision_transformer/">Image classification with Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/attention_mil_classification/">Classification using Attention-based Deep Multiple Instance Learning</a> <a class="nav-sublink2" href="/examples/vision/mlp_image_classification/">Image classification with modern MLP models</a> <a class="nav-sublink2" href="/examples/vision/mobilevit/">A mobile-friendly Transformer-based model for image classification</a> <a class="nav-sublink2" href="/examples/vision/xray_classification_with_tpus/">Pneumonia Classification on TPU</a> <a class="nav-sublink2" href="/examples/vision/cct/">Compact Convolutional Transformers</a> <a class="nav-sublink2" href="/examples/vision/convmixer/">Image classification with ConvMixer</a> <a class="nav-sublink2" href="/examples/vision/eanet/">Image classification with EANet (External Attention Transformer)</a> <a class="nav-sublink2" href="/examples/vision/involution/">Involutional neural networks</a> <a class="nav-sublink2" href="/examples/vision/perceiver_image_classification/">Image classification with Perceiver</a> <a class="nav-sublink2" href="/examples/vision/reptile/">Few-Shot learning with Reptile</a> <a class="nav-sublink2" href="/examples/vision/semisupervised_simclr/">Semi-supervised image classification using contrastive pretraining with SimCLR</a> <a class="nav-sublink2" href="/examples/vision/swin_transformers/">Image classification with Swin Transformers</a> <a class="nav-sublink2" href="/examples/vision/vit_small_ds/">Train a Vision Transformer on small datasets</a> <a class="nav-sublink2" href="/examples/vision/shiftvit/">A Vision Transformer without Attention</a> <a class="nav-sublink2" href="/examples/vision/image_classification_using_global_context_vision_transformer/">Image Classification using Global Context Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/oxford_pets_image_segmentation/">Image segmentation with a U-Net-like architecture</a> <a class="nav-sublink2" href="/examples/vision/deeplabv3_plus/">Multiclass semantic segmentation using DeepLabV3+</a> <a class="nav-sublink2" href="/examples/vision/basnet_segmentation/">Highly accurate boundaries segmentation using BASNet</a> <a class="nav-sublink2" href="/examples/vision/fully_convolutional_network/">Image Segmentation using Composable Fully-Convolutional Networks</a> <a class="nav-sublink2" href="/examples/vision/retinanet/">Object Detection with RetinaNet</a> <a class="nav-sublink2" href="/examples/vision/keypoint_detection/">Keypoint Detection with Transfer Learning</a> <a class="nav-sublink2 active" href="/examples/vision/object_detection_using_vision_transformer/">Object detection with Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/3D_image_classification/">3D image classification from CT scans</a> <a class="nav-sublink2" href="/examples/vision/depth_estimation/">Monocular depth estimation</a> <a class="nav-sublink2" href="/examples/vision/nerf/">3D volumetric rendering with NeRF</a> <a class="nav-sublink2" href="/examples/vision/pointnet_segmentation/">Point cloud segmentation with PointNet</a> <a class="nav-sublink2" href="/examples/vision/pointnet/">Point cloud classification</a> <a class="nav-sublink2" href="/examples/vision/captcha_ocr/">OCR model for reading Captchas</a> <a class="nav-sublink2" href="/examples/vision/handwriting_recognition/">Handwriting recognition</a> <a class="nav-sublink2" href="/examples/vision/autoencoder/">Convolutional autoencoder for image denoising</a> <a class="nav-sublink2" href="/examples/vision/mirnet/">Low-light image enhancement using MIRNet</a> <a class="nav-sublink2" href="/examples/vision/super_resolution_sub_pixel/">Image Super-Resolution using an Efficient Sub-Pixel CNN</a> <a class="nav-sublink2" href="/examples/vision/edsr/">Enhanced Deep Residual Networks for single-image super-resolution</a> <a class="nav-sublink2" href="/examples/vision/zero_dce/">Zero-DCE for low-light image enhancement</a> <a class="nav-sublink2" href="/examples/vision/cutmix/">CutMix data augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/mixup/">MixUp augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/randaugment/">RandAugment for Image Classification for Improved Robustness</a> <a class="nav-sublink2" href="/examples/vision/image_captioning/">Image captioning</a> <a class="nav-sublink2" href="/examples/vision/nl_image_search/">Natural language image search with a Dual Encoder</a> <a class="nav-sublink2" href="/examples/vision/visualizing_what_convnets_learn/">Visualizing what convnets learn</a> <a class="nav-sublink2" href="/examples/vision/integrated_gradients/">Model interpretability with Integrated Gradients</a> <a class="nav-sublink2" href="/examples/vision/probing_vits/">Investigating Vision Transformer representations</a> <a class="nav-sublink2" href="/examples/vision/grad_cam/">Grad-CAM class activation visualization</a> <a class="nav-sublink2" href="/examples/vision/near_dup_search/">Near-duplicate image search</a> <a class="nav-sublink2" href="/examples/vision/semantic_image_clustering/">Semantic Image Clustering</a> <a class="nav-sublink2" href="/examples/vision/siamese_contrastive/">Image similarity estimation using a Siamese Network with a contrastive loss</a> <a class="nav-sublink2" href="/examples/vision/siamese_network/">Image similarity estimation using a Siamese Network with a triplet loss</a> <a class="nav-sublink2" href="/examples/vision/metric_learning/">Metric learning for image similarity search</a> <a class="nav-sublink2" href="/examples/vision/metric_learning_tf_similarity/">Metric learning for image similarity search using TensorFlow Similarity</a> <a class="nav-sublink2" href="/examples/vision/nnclr/">Self-supervised contrastive learning with NNCLR</a> <a class="nav-sublink2" href="/examples/vision/video_classification/">Video Classification with a CNN-RNN Architecture</a> <a class="nav-sublink2" href="/examples/vision/conv_lstm/">Next-Frame Video Prediction with Convolutional LSTMs</a> <a class="nav-sublink2" href="/examples/vision/video_transformers/">Video Classification with Transformers</a> <a class="nav-sublink2" href="/examples/vision/vivit/">Video Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/bit/">Image Classification using BigTransfer (BiT)</a> <a class="nav-sublink2" href="/examples/vision/gradient_centralization/">Gradient Centralization for Better Training Performance</a> <a class="nav-sublink2" href="/examples/vision/token_learner/">Learning to tokenize in Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/knowledge_distillation/">Knowledge Distillation</a> <a class="nav-sublink2" href="/examples/vision/fixres/">FixRes: Fixing train-test resolution discrepancy</a> <a class="nav-sublink2" href="/examples/vision/cait/">Class Attention Image Transformers with LayerScale</a> <a class="nav-sublink2" href="/examples/vision/patch_convnet/">Augmenting convnets with aggregated attention</a> <a class="nav-sublink2" href="/examples/vision/learnable_resizer/">Learning to Resize</a> <a class="nav-sublink2" href="/examples/vision/adamatch/">Semi-supervision and domain adaptation with AdaMatch</a> <a class="nav-sublink2" href="/examples/vision/barlow_twins/">Barlow Twins for Contrastive SSL</a> <a class="nav-sublink2" href="/examples/vision/consistency_training/">Consistency training with supervision</a> <a class="nav-sublink2" href="/examples/vision/deit/">Distilling Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/focal_modulation_network/">Focal Modulation: A replacement for Self-Attention</a> <a class="nav-sublink2" href="/examples/vision/forwardforward/">Using the Forward-Forward Algorithm for Image Classification</a> <a class="nav-sublink2" href="/examples/vision/masked_image_modeling/">Masked image modeling with Autoencoders</a> <a class="nav-sublink2" href="/examples/vision/sam/">Segment Anything Model with 🤗Transformers</a> <a class="nav-sublink2" href="/examples/vision/segformer/">Semantic segmentation with SegFormer and Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/vision/simsiam/">Self-supervised contrastive learning with SimSiam</a> <a class="nav-sublink2" href="/examples/vision/supervised-contrastive-learning/">Supervised Contrastive Learning</a> <a class="nav-sublink2" href="/examples/vision/temporal_latent_bottleneck/">When Recurrence meets Transformers</a> <a class="nav-sublink2" href="/examples/vision/yolov8/">Efficient Object Detection with YOLOV8 and KerasCV</a> <a class="nav-sublink" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparam Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/vision/'>Computer Vision</a> / Object detection with Vision Transformers </div> <div class='k-content'> <h1 id="object-detection-with-vision-transformers">Object detection with Vision Transformers</h1> Author: <a href="https://www.linkedin.com/in/karan-dave-811413164/">Karan V. Dave</a> Date created: 2022/03/27 Last modified: 2023/11/20 Description: A simple Keras implementation of object detection using Vision Transformers. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/object_detection_using_vision_transformer.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/vision/object_detection_using_vision_transformer.py">GitHub source</a> <hr /> <h2 id="introduction">Introduction</h2> The article <a href="https://arxiv.org/abs/2010.11929">Vision Transformer (ViT)</a> architecture by Alexey Dosovitskiy et al. demonstrates that a pure transformer applied directly to sequences of image patches can perform well on object detection tasks. In this Keras example, we implement an object detection ViT and we train it on the <a href="http://www.vision.caltech.edu/datasets/">Caltech 101 dataset</a> to detect an airplane in the given image. <hr /> <h2 id="imports-and-setup">Imports and setup</h2> <div class="codehilite"><pre><code>import os os.environ["KERAS_BACKEND"] = "jax" # @param ["tensorflow", "jax", "torch"] import numpy as np import keras from keras import layers from keras import ops import matplotlib.pyplot as plt import numpy as np import cv2 import os import scipy.io import shutil </code></pre></div> <hr /> <h2 id="prepare-dataset">Prepare dataset</h2> We use the <a href="https://data.caltech.edu/records/mzrjq-6wc02">Caltech 101 Dataset</a>. <div class="codehilite"><pre><code># Path to images and annotations path_images = "./101_ObjectCategories/airplanes/" path_annot = "./Annotations/Airplanes_Side_2/" path_to_downloaded_file = keras.utils.get_file( fname="caltech_101_zipped", origin="https://data.caltech.edu/records/mzrjq-6wc02/files/caltech-101.zip", extract=True, archive_format="zip", # downloaded file format cache_dir="/", # cache and extract in current directory ) download_base_dir = os.path.dirname(path_to_downloaded_file) # Extracting tar files found inside main zip file shutil.unpack_archive( os.path.join(download_base_dir, "caltech-101", "101_ObjectCategories.tar.gz"), "." ) shutil.unpack_archive( os.path.join(download_base_dir, "caltech-101", "Annotations.tar"), "." ) # list of paths to images and annotations image_paths = [ f for f in os.listdir(path_images) if os.path.isfile(os.path.join(path_images, f)) ] annot_paths = [ f for f in os.listdir(path_annot) if os.path.isfile(os.path.join(path_annot, f)) ] image_paths.sort() annot_paths.sort() image_size = 224 # resize input images to this size images, targets = [], [] # loop over the annotations and images, preprocess them and store in lists for i in range(0, len(annot_paths)): # Access bounding box coordinates annot = scipy.io.loadmat(path_annot + annot_paths[i])["box_coord"][0] top_left_x, top_left_y = annot[2], annot[0] bottom_right_x, bottom_right_y = annot[3], annot[1] image = keras.utils.load_img( path_images + image_paths[i], ) (w, h) = image.size[:2] # resize images image = image.resize((image_size, image_size)) # convert image to array and append to list images.append(keras.utils.img_to_array(image)) # apply relative scaling to bounding boxes as per given image and append to list targets.append( ( float(top_left_x) / w, float(top_left_y) / h, float(bottom_right_x) / w, float(bottom_right_y) / h, ) ) # Convert the list to numpy array, split to train and test dataset (x_train), (y_train) = ( np.asarray(images[: int(len(images) * 0.8)]), np.asarray(targets[: int(len(targets) * 0.8)]), ) (x_test), (y_test) = ( np.asarray(images[int(len(images) * 0.8) :]), np.asarray(targets[int(len(targets) * 0.8) :]), ) </code></pre></div> <hr /> <h2 id="implement-multilayerperceptron-mlp">Implement multilayer-perceptron (MLP)</h2> We use the code from the Keras example <a href="https://keras.io/examples/vision/image_classification_with_vision_transformer/">Image classification with Vision Transformer</a> as a reference. <div class="codehilite"><pre><code>def mlp(x, hidden_units, dropout_rate): for units in hidden_units: x = layers.Dense(units, activation=keras.activations.gelu)(x) x = layers.Dropout(dropout_rate)(x) return x </code></pre></div> <hr /> <h2 id="implement-the-patch-creation-layer">Implement the patch creation layer</h2> <div class="codehilite"><pre><code>class Patches(layers.Layer): def __init__(self, patch_size): super().__init__() self.patch_size = patch_size def call(self, images): input_shape = ops.shape(images) batch_size = input_shape[0] height = input_shape[1] width = input_shape[2] channels = input_shape[3] num_patches_h = height // self.patch_size num_patches_w = width // self.patch_size patches = keras.ops.image.extract_patches(images, size=self.patch_size) patches = ops.reshape( patches, ( batch_size, num_patches_h * num_patches_w, self.patch_size * self.patch_size * channels, ), ) return patches def get_config(self): config = super().get_config() config.update({"patch_size": self.patch_size}) return config </code></pre></div> <hr /> <h2 id="display-patches-for-an-input-image">Display patches for an input image</h2> <div class="codehilite"><pre><code>patch_size = 32 # Size of the patches to be extracted from the input images plt.figure(figsize=(4, 4)) plt.imshow(x_train[0].astype("uint8")) plt.axis("off") patches = Patches(patch_size)(np.expand_dims(x_train[0], axis=0)) print(f"Image size: {image_size} X {image_size}") print(f"Patch size: {patch_size} X {patch_size}") print(f"{patches.shape[1]} patches per image \n{patches.shape[-1]} elements per patch") n = int(np.sqrt(patches.shape[1])) plt.figure(figsize=(4, 4)) for i, patch in enumerate(patches[0]): ax = plt.subplot(n, n, i + 1) patch_img = ops.reshape(patch, (patch_size, patch_size, 3)) plt.imshow(ops.convert_to_numpy(patch_img).astype("uint8")) plt.axis("off") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Image size: 224 X 224 Patch size: 32 X 32 49 patches per image 3072 elements per patch </code></pre></div> </div> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_11_1.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_11_2.png" /> <hr /> <h2 id="implement-the-patch-encoding-layer">Implement the patch encoding layer</h2> The <code>PatchEncoder</code> layer linearly transforms a patch by projecting it into a vector of size <code>projection_dim</code>. It also adds a learnable position embedding to the projected vector. <div class="codehilite"><pre><code>class PatchEncoder(layers.Layer): def __init__(self, num_patches, projection_dim): super().__init__() self.num_patches = num_patches self.projection = layers.Dense(units=projection_dim) self.position_embedding = layers.Embedding( input_dim=num_patches, output_dim=projection_dim ) # Override function to avoid error while saving model def get_config(self): config = super().get_config().copy() config.update( { "input_shape": input_shape, "patch_size": patch_size, "num_patches": num_patches, "projection_dim": projection_dim, "num_heads": num_heads, "transformer_units": transformer_units, "transformer_layers": transformer_layers, "mlp_head_units": mlp_head_units, } ) return config def call(self, patch): positions = ops.expand_dims( ops.arange(start=0, stop=self.num_patches, step=1), axis=0 ) projected_patches = self.projection(patch) encoded = projected_patches + self.position_embedding(positions) return encoded </code></pre></div> <hr /> <h2 id="build-the-vit-model">Build the ViT model</h2> The ViT model has multiple Transformer blocks. The <code>MultiHeadAttention</code> layer is used for self-attention, applied to the sequence of image patches. The encoded patches (skip connection) and self-attention layer outputs are normalized and fed into a multilayer perceptron (MLP). The model outputs four dimensions representing the bounding box coordinates of an object. <div class="codehilite"><pre><code>def create_vit_object_detector( input_shape, patch_size, num_patches, projection_dim, num_heads, transformer_units, transformer_layers, mlp_head_units, ): inputs = keras.Input(shape=input_shape) # Create patches patches = Patches(patch_size)(inputs) # Encode patches encoded_patches = PatchEncoder(num_patches, projection_dim)(patches) # Create multiple layers of the Transformer block. for _ in range(transformer_layers): # Layer normalization 1. x1 = layers.LayerNormalization(epsilon=1e-6)(encoded_patches) # Create a multi-head attention layer. attention_output = layers.MultiHeadAttention( num_heads=num_heads, key_dim=projection_dim, dropout=0.1 )(x1, x1) # Skip connection 1. x2 = layers.Add()([attention_output, encoded_patches]) # Layer normalization 2. x3 = layers.LayerNormalization(epsilon=1e-6)(x2) # MLP x3 = mlp(x3, hidden_units=transformer_units, dropout_rate=0.1) # Skip connection 2. encoded_patches = layers.Add()([x3, x2]) # Create a [batch_size, projection_dim] tensor. representation = layers.LayerNormalization(epsilon=1e-6)(encoded_patches) representation = layers.Flatten()(representation) representation = layers.Dropout(0.3)(representation) # Add MLP. features = mlp(representation, hidden_units=mlp_head_units, dropout_rate=0.3) bounding_box = layers.Dense(4)( features ) # Final four neurons that output bounding box # return Keras model. return keras.Model(inputs=inputs, outputs=bounding_box) </code></pre></div> <hr /> <h2 id="run-the-experiment">Run the experiment</h2> <div class="codehilite"><pre><code>def run_experiment(model, learning_rate, weight_decay, batch_size, num_epochs): optimizer = keras.optimizers.AdamW( learning_rate=learning_rate, weight_decay=weight_decay ) # Compile model. model.compile(optimizer=optimizer, loss=keras.losses.MeanSquaredError()) checkpoint_filepath = "vit_object_detector.weights.h5" checkpoint_callback = keras.callbacks.ModelCheckpoint( checkpoint_filepath, monitor="val_loss", save_best_only=True, save_weights_only=True, ) history = model.fit( x=x_train, y=y_train, batch_size=batch_size, epochs=num_epochs, validation_split=0.1, callbacks=[ checkpoint_callback, keras.callbacks.EarlyStopping(monitor="val_loss", patience=10), ], ) return history input_shape = (image_size, image_size, 3) # input image shape learning_rate = 0.001 weight_decay = 0.0001 batch_size = 32 num_epochs = 100 num_patches = (image_size // patch_size) ** 2 projection_dim = 64 num_heads = 4 # Size of the transformer layers transformer_units = [ projection_dim * 2, projection_dim, ] transformer_layers = 4 mlp_head_units = [2048, 1024, 512, 64, 32] # Size of the dense layers history = [] num_patches = (image_size // patch_size) ** 2 vit_object_detector = create_vit_object_detector( input_shape, patch_size, num_patches, projection_dim, num_heads, transformer_units, transformer_layers, mlp_head_units, ) # Train model history = run_experiment( vit_object_detector, learning_rate, weight_decay, batch_size, num_epochs ) def plot_history(item): plt.plot(history.history[item], label=item) plt.plot(history.history["val_" + item], label="val_" + item) plt.xlabel("Epochs") plt.ylabel(item) plt.title("Train and Validation {} Over Epochs".format(item), fontsize=14) plt.legend() plt.grid() plt.show() plot_history("loss") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Epoch 1/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 9s 109ms/step - loss: 1.2097 - val_loss: 0.3468 Epoch 2/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.4260 - val_loss: 0.3102 Epoch 3/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.3268 - val_loss: 0.2727 Epoch 4/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.2815 - val_loss: 0.2391 Epoch 5/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.2290 - val_loss: 0.1735 Epoch 6/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step - loss: 0.1870 - val_loss: 0.1055 Epoch 7/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.1401 - val_loss: 0.0610 Epoch 8/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.1122 - val_loss: 0.0274 Epoch 9/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0924 - val_loss: 0.0296 Epoch 10/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step - loss: 0.0765 - val_loss: 0.0139 Epoch 11/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.0597 - val_loss: 0.0111 Epoch 12/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.0540 - val_loss: 0.0101 Epoch 13/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step - loss: 0.0432 - val_loss: 0.0053 Epoch 14/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step - loss: 0.0380 - val_loss: 0.0052 Epoch 15/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.0334 - val_loss: 0.0030 Epoch 16/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step - loss: 0.0283 - val_loss: 0.0021 Epoch 17/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step - loss: 0.0228 - val_loss: 0.0012 Epoch 18/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0244 - val_loss: 0.0017 Epoch 19/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0195 - val_loss: 0.0016 Epoch 20/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0189 - val_loss: 0.0020 Epoch 21/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0191 - val_loss: 0.0019 Epoch 22/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0174 - val_loss: 0.0016 Epoch 23/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0157 - val_loss: 0.0020 Epoch 24/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0157 - val_loss: 0.0015 Epoch 25/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0139 - val_loss: 0.0023 Epoch 26/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0130 - val_loss: 0.0017 Epoch 27/100 18/18 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step - loss: 0.0157 - val_loss: 0.0014 </code></pre></div> </div> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_17_1.png" /> <hr /> <h2 id="evaluate-the-model">Evaluate the model</h2> <div class="codehilite"><pre><code>import matplotlib.patches as patches # Saves the model in current path vit_object_detector.save("vit_object_detector.keras") # To calculate IoU (intersection over union, given two bounding boxes) def bounding_box_intersection_over_union(box_predicted, box_truth): # get (x, y) coordinates of intersection of bounding boxes top_x_intersect = max(box_predicted[0], box_truth[0]) top_y_intersect = max(box_predicted[1], box_truth[1]) bottom_x_intersect = min(box_predicted[2], box_truth[2]) bottom_y_intersect = min(box_predicted[3], box_truth[3]) # calculate area of the intersection bb (bounding box) intersection_area = max(0, bottom_x_intersect - top_x_intersect + 1) * max( 0, bottom_y_intersect - top_y_intersect + 1 ) # calculate area of the prediction bb and ground-truth bb box_predicted_area = (box_predicted[2] - box_predicted[0] + 1) * ( box_predicted[3] - box_predicted[1] + 1 ) box_truth_area = (box_truth[2] - box_truth[0] + 1) * ( box_truth[3] - box_truth[1] + 1 ) # calculate intersection over union by taking intersection # area and dividing it by the sum of predicted bb and ground truth # bb areas subtracted by the interesection area # return ioU return intersection_area / float( box_predicted_area + box_truth_area - intersection_area ) i, mean_iou = 0, 0 # Compare results for 10 images in the test set for input_image in x_test[:10]: fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 15)) im = input_image # Display the image ax1.imshow(im.astype("uint8")) ax2.imshow(im.astype("uint8")) input_image = cv2.resize( input_image, (image_size, image_size), interpolation=cv2.INTER_AREA ) input_image = np.expand_dims(input_image, axis=0) preds = vit_object_detector.predict(input_image)[0] (h, w) = (im).shape[0:2] top_left_x, top_left_y = int(preds[0] * w), int(preds[1] * h) bottom_right_x, bottom_right_y = int(preds[2] * w), int(preds[3] * h) box_predicted = [top_left_x, top_left_y, bottom_right_x, bottom_right_y] # Create the bounding box rect = patches.Rectangle( (top_left_x, top_left_y), bottom_right_x - top_left_x, bottom_right_y - top_left_y, facecolor="none", edgecolor="red", linewidth=1, ) # Add the bounding box to the image ax1.add_patch(rect) ax1.set_xlabel( "Predicted: " + str(top_left_x) + ", " + str(top_left_y) + ", " + str(bottom_right_x) + ", " + str(bottom_right_y) ) top_left_x, top_left_y = int(y_test[i][0] * w), int(y_test[i][1] * h) bottom_right_x, bottom_right_y = int(y_test[i][2] * w), int(y_test[i][3] * h) box_truth = top_left_x, top_left_y, bottom_right_x, bottom_right_y mean_iou += bounding_box_intersection_over_union(box_predicted, box_truth) # Create the bounding box rect = patches.Rectangle( (top_left_x, top_left_y), bottom_right_x - top_left_x, bottom_right_y - top_left_y, facecolor="none", edgecolor="red", linewidth=1, ) # Add the bounding box to the image ax2.add_patch(rect) ax2.set_xlabel( "Target: " + str(top_left_x) + ", " + str(top_left_y) + ", " + str(bottom_right_x) + ", " + str(bottom_right_y) + "\n" + "IoU" + str(bounding_box_intersection_over_union(box_predicted, box_truth)) ) i = i + 1 print("mean_iou: " + str(mean_iou / len(x_test[:10]))) plt.show() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code> 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 1s/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step mean_iou: 0.9092338486331416 </code></pre></div> </div> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_1.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_2.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_3.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_4.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_5.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_6.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_7.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_8.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_9.png" /> <img alt="png" src="/img/examples/vision/object_detection_using_vision_transformer/object_detection_using_vision_transformer_19_10.png" /> This example demonstrates that a pure Transformer can be trained to predict the bounding boxes of an object in a given image, thus extending the use of Transformers to object detection tasks. The model can be improved further by tuning hyper-parameters and pre-training. </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#object-detection-with-vision-transformers'>Object detection with Vision Transformers</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#imports-and-setup'>Imports and setup</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#prepare-dataset'>Prepare dataset</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#implement-multilayerperceptron-mlp'>Implement multilayer-perceptron (MLP)</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#implement-the-patch-creation-layer'>Implement the patch creation layer</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#display-patches-for-an-input-image'>Display patches for an input image</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#implement-the-patch-encoding-layer'>Implement the patch encoding layer</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#build-the-vit-model'>Build the ViT model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#run-the-experiment'>Run the experiment</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#evaluate-the-model'>Evaluate the model</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>