Keypoint Detection with Transfer Learning

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/vision/keypoint_detection/" />  <meta property="og:title" content="Keras documentation: Keypoint Detection with Transfer Learning"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Keypoint Detection with Transfer Learning"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Keypoint Detection with Transfer Learning</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink active" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink2" href="/examples/vision/image_classification_from_scratch/">Image classification from scratch</a> <a class="nav-sublink2" href="/examples/vision/mnist_convnet/">Simple MNIST convnet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_efficientnet_fine_tuning/">Image classification via fine-tuning with EfficientNet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_with_vision_transformer/">Image classification with Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/attention_mil_classification/">Classification using Attention-based Deep Multiple Instance Learning</a> <a class="nav-sublink2" href="/examples/vision/mlp_image_classification/">Image classification with modern MLP models</a> <a class="nav-sublink2" href="/examples/vision/mobilevit/">A mobile-friendly Transformer-based model for image classification</a> <a class="nav-sublink2" href="/examples/vision/xray_classification_with_tpus/">Pneumonia Classification on TPU</a> <a class="nav-sublink2" href="/examples/vision/cct/">Compact Convolutional Transformers</a> <a class="nav-sublink2" href="/examples/vision/convmixer/">Image classification with ConvMixer</a> <a class="nav-sublink2" href="/examples/vision/eanet/">Image classification with EANet (External Attention Transformer)</a> <a class="nav-sublink2" href="/examples/vision/involution/">Involutional neural networks</a> <a class="nav-sublink2" href="/examples/vision/perceiver_image_classification/">Image classification with Perceiver</a> <a class="nav-sublink2" href="/examples/vision/reptile/">Few-Shot learning with Reptile</a> <a class="nav-sublink2" href="/examples/vision/semisupervised_simclr/">Semi-supervised image classification using contrastive pretraining with SimCLR</a> <a class="nav-sublink2" href="/examples/vision/swin_transformers/">Image classification with Swin Transformers</a> <a class="nav-sublink2" href="/examples/vision/vit_small_ds/">Train a Vision Transformer on small datasets</a> <a class="nav-sublink2" href="/examples/vision/shiftvit/">A Vision Transformer without Attention</a> <a class="nav-sublink2" href="/examples/vision/image_classification_using_global_context_vision_transformer/">Image Classification using Global Context Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/temporal_latent_bottleneck/">When Recurrence meets Transformers</a> <a class="nav-sublink2" href="/examples/vision/oxford_pets_image_segmentation/">Image segmentation with a U-Net-like architecture</a> <a class="nav-sublink2" href="/examples/vision/deeplabv3_plus/">Multiclass semantic segmentation using DeepLabV3+</a> <a class="nav-sublink2" href="/examples/vision/basnet_segmentation/">Highly accurate boundaries segmentation using BASNet</a> <a class="nav-sublink2" href="/examples/vision/fully_convolutional_network/">Image Segmentation using Composable Fully-Convolutional Networks</a> <a class="nav-sublink2" href="/examples/vision/retinanet/">Object Detection with RetinaNet</a> <a class="nav-sublink2 active" href="/examples/vision/keypoint_detection/">Keypoint Detection with Transfer Learning</a> <a class="nav-sublink2" href="/examples/vision/object_detection_using_vision_transformer/">Object detection with Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/3D_image_classification/">3D image classification from CT scans</a> <a class="nav-sublink2" href="/examples/vision/depth_estimation/">Monocular depth estimation</a> <a class="nav-sublink2" href="/examples/vision/nerf/">3D volumetric rendering with NeRF</a> <a class="nav-sublink2" href="/examples/vision/pointnet_segmentation/">Point cloud segmentation with PointNet</a> <a class="nav-sublink2" href="/examples/vision/pointnet/">Point cloud classification</a> <a class="nav-sublink2" href="/examples/vision/captcha_ocr/">OCR model for reading Captchas</a> <a class="nav-sublink2" href="/examples/vision/handwriting_recognition/">Handwriting recognition</a> <a class="nav-sublink2" href="/examples/vision/autoencoder/">Convolutional autoencoder for image denoising</a> <a class="nav-sublink2" href="/examples/vision/mirnet/">Low-light image enhancement using MIRNet</a> <a class="nav-sublink2" href="/examples/vision/super_resolution_sub_pixel/">Image Super-Resolution using an Efficient Sub-Pixel CNN</a> <a class="nav-sublink2" href="/examples/vision/edsr/">Enhanced Deep Residual Networks for single-image super-resolution</a> <a class="nav-sublink2" href="/examples/vision/zero_dce/">Zero-DCE for low-light image enhancement</a> <a class="nav-sublink2" href="/examples/vision/cutmix/">CutMix data augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/mixup/">MixUp augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/randaugment/">RandAugment for Image Classification for Improved Robustness</a> <a class="nav-sublink2" href="/examples/vision/image_captioning/">Image captioning</a> <a class="nav-sublink2" href="/examples/vision/nl_image_search/">Natural language image search with a Dual Encoder</a> <a class="nav-sublink2" href="/examples/vision/visualizing_what_convnets_learn/">Visualizing what convnets learn</a> <a class="nav-sublink2" href="/examples/vision/integrated_gradients/">Model interpretability with Integrated Gradients</a> <a class="nav-sublink2" href="/examples/vision/probing_vits/">Investigating Vision Transformer representations</a> <a class="nav-sublink2" href="/examples/vision/grad_cam/">Grad-CAM class activation visualization</a> <a class="nav-sublink2" href="/examples/vision/near_dup_search/">Near-duplicate image search</a> <a class="nav-sublink2" href="/examples/vision/semantic_image_clustering/">Semantic Image Clustering</a> <a class="nav-sublink2" href="/examples/vision/siamese_contrastive/">Image similarity estimation using a Siamese Network with a contrastive loss</a> <a class="nav-sublink2" href="/examples/vision/siamese_network/">Image similarity estimation using a Siamese Network with a triplet loss</a> <a class="nav-sublink2" href="/examples/vision/metric_learning/">Metric learning for image similarity search</a> <a class="nav-sublink2" href="/examples/vision/metric_learning_tf_similarity/">Metric learning for image similarity search using TensorFlow Similarity</a> <a class="nav-sublink2" href="/examples/vision/nnclr/">Self-supervised contrastive learning with NNCLR</a> <a class="nav-sublink2" href="/examples/vision/video_classification/">Video Classification with a CNN-RNN Architecture</a> <a class="nav-sublink2" href="/examples/vision/conv_lstm/">Next-Frame Video Prediction with Convolutional LSTMs</a> <a class="nav-sublink2" href="/examples/vision/video_transformers/">Video Classification with Transformers</a> <a class="nav-sublink2" href="/examples/vision/vivit/">Video Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/bit/">Image Classification using BigTransfer (BiT)</a> <a class="nav-sublink2" href="/examples/vision/gradient_centralization/">Gradient Centralization for Better Training Performance</a> <a class="nav-sublink2" href="/examples/vision/token_learner/">Learning to tokenize in Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/knowledge_distillation/">Knowledge Distillation</a> <a class="nav-sublink2" href="/examples/vision/fixres/">FixRes: Fixing train-test resolution discrepancy</a> <a class="nav-sublink2" href="/examples/vision/cait/">Class Attention Image Transformers with LayerScale</a> <a class="nav-sublink2" href="/examples/vision/patch_convnet/">Augmenting convnets with aggregated attention</a> <a class="nav-sublink2" href="/examples/vision/learnable_resizer/">Learning to Resize</a> <a class="nav-sublink2" href="/examples/vision/adamatch/">Semi-supervision and domain adaptation with AdaMatch</a> <a class="nav-sublink2" href="/examples/vision/barlow_twins/">Barlow Twins for Contrastive SSL</a> <a class="nav-sublink2" href="/examples/vision/consistency_training/">Consistency training with supervision</a> <a class="nav-sublink2" href="/examples/vision/deit/">Distilling Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/focal_modulation_network/">Focal Modulation: A replacement for Self-Attention</a> <a class="nav-sublink2" href="/examples/vision/forwardforward/">Using the Forward-Forward Algorithm for Image Classification</a> <a class="nav-sublink2" href="/examples/vision/masked_image_modeling/">Masked image modeling with Autoencoders</a> <a class="nav-sublink2" href="/examples/vision/sam/">Segment Anything Model with 🤗Transformers</a> <a class="nav-sublink2" href="/examples/vision/segformer/">Semantic segmentation with SegFormer and Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/vision/simsiam/">Self-supervised contrastive learning with SimSiam</a> <a class="nav-sublink2" href="/examples/vision/supervised-contrastive-learning/">Supervised Contrastive Learning</a> <a class="nav-sublink2" href="/examples/vision/yolov8/">Efficient Object Detection with YOLOV8 and KerasCV</a> <a class="nav-sublink" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparam Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/vision/'>Computer Vision</a> / Keypoint Detection with Transfer Learning </div> <div class='k-content'> <h1 id="keypoint-detection-with-transfer-learning">Keypoint Detection with Transfer Learning</h1> Author: <a href="https://twitter.com/RisingSayak">Sayak Paul</a>, converted to Keras 3 by <a href="https://anasrz.com">Muhammad Anas Raza</a> Date created: 2021/05/02 Last modified: 2023/07/19 Description: Training a keypoint detector with data augmentation and transfer learning. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/keypoint_detection.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/vision/keypoint_detection.py">GitHub source</a> Keypoint detection consists of locating key object parts. For example, the key parts of our faces include nose tips, eyebrows, eye corners, and so on. These parts help to represent the underlying object in a feature-rich manner. Keypoint detection has applications that include pose estimation, face detection, etc. In this example, we will build a keypoint detector using the <a href="https://github.com/benjiebob/StanfordExtra">StanfordExtra dataset</a>, using transfer learning. This example requires TensorFlow 2.4 or higher, as well as <a href="https://imgaug.readthedocs.io/"><code>imgaug</code></a> library, which can be installed using the following command: <div class="codehilite"><pre><code>!pip install -q -U imgaug </code></pre></div> <hr /> <h2 id="data-collection">Data collection</h2> The StanfordExtra dataset contains 12,000 images of dogs together with keypoints and segmentation maps. It is developed from the <a href="http://vision.stanford.edu/aditya86/ImageNetDogs/">Stanford dogs dataset</a>. It can be downloaded with the command below: <div class="codehilite"><pre><code>!wget -q http://vision.stanford.edu/aditya86/ImageNetDogs/images.tar </code></pre></div> Annotations are provided as a single JSON file in the StanfordExtra dataset and one needs to fill <a href="https://forms.gle/sRtbicgxsWvRtRmUA">this form</a> to get access to it. The authors explicitly instruct users not to share the JSON file, and this example respects this wish: you should obtain the JSON file yourself. The JSON file is expected to be locally available as <code>stanfordextra_v12.zip</code>. After the files are downloaded, we can extract the archives. <div class="codehilite"><pre><code>!tar xf images.tar !unzip -qq ~/stanfordextra_v12.zip </code></pre></div> <hr /> <h2 id="imports">Imports</h2> <div class="codehilite"><pre><code>from keras import layers import keras from imgaug.augmentables.kps import KeypointsOnImage from imgaug.augmentables.kps import Keypoint import imgaug.augmenters as iaa from PIL import Image from sklearn.model_selection import train_test_split from matplotlib import pyplot as plt import pandas as pd import numpy as np import json import os </code></pre></div> <hr /> <h2 id="define-hyperparameters">Define hyperparameters</h2> <div class="codehilite"><pre><code>IMG_SIZE = 224 BATCH_SIZE = 64 EPOCHS = 5 NUM_KEYPOINTS = 24 * 2 # 24 pairs each having x and y coordinates </code></pre></div> <hr /> <h2 id="load-data">Load data</h2> The authors also provide a metadata file that specifies additional information about the keypoints, like color information, animal pose name, etc. We will load this file in a <code>pandas</code> dataframe to extract information for visualization purposes. <div class="codehilite"><pre><code>IMG_DIR = "Images" JSON = "StanfordExtra_V12/StanfordExtra_v12.json" KEYPOINT_DEF = ( "https://github.com/benjiebob/StanfordExtra/raw/master/keypoint_definitions.csv" ) # Load the ground-truth annotations. with open(JSON) as infile: json_data = json.load(infile) # Set up a dictionary, mapping all the ground-truth information # with respect to the path of the image. json_dict = {i["img_path"]: i for i in json_data} </code></pre></div> A single entry of <code>json_dict</code> looks like the following: <div class="codehilite"><pre><code>'n02085782-Japanese_spaniel/n02085782_2886.jpg': {'img_bbox': [205, 20, 116, 201], 'img_height': 272, 'img_path': 'n02085782-Japanese_spaniel/n02085782_2886.jpg', 'img_width': 350, 'is_multiple_dogs': False, 'joints': [[108.66666666666667, 252.0, 1], [147.66666666666666, 229.0, 1], [163.5, 208.5, 1], [0, 0, 0], [0, 0, 0], [0, 0, 0], [54.0, 244.0, 1], [77.33333333333333, 225.33333333333334, 1], [79.0, 196.5, 1], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [150.66666666666666, 86.66666666666667, 1], [88.66666666666667, 73.0, 1], [116.0, 106.33333333333333, 1], [109.0, 123.33333333333333, 1], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], 'seg': ...} </code></pre></div> In this example, the keys we are interested in are: <ul> <li><code>img_path</code></li> <li><code>joints</code></li> </ul> There are a total of 24 entries present inside <code>joints</code>. Each entry has 3 values: <ul> <li>x-coordinate</li> <li>y-coordinate</li> <li>visibility flag of the keypoints (1 indicates visibility and 0 indicates non-visibility)</li> </ul> As we can see <code>joints</code> contain multiple <code>[0, 0, 0]</code> entries which denote that those keypoints were not labeled. In this example, we will consider both non-visible as well as unlabeled keypoints in order to allow mini-batch learning. <div class="codehilite"><pre><code># Load the metdata definition file and preview it. keypoint_def = pd.read_csv(KEYPOINT_DEF) keypoint_def.head() # Extract the colours and labels. colours = keypoint_def["Hex colour"].values.tolist() colours = ["#" + colour for colour in colours] labels = keypoint_def["Name"].values.tolist() # Utility for reading an image and for getting its annotations. def get_dog(name): data = json_dict[name] img_data = plt.imread(os.path.join(IMG_DIR, data["img_path"])) # If the image is RGBA convert it to RGB. if img_data.shape[-1] == 4: img_data = img_data.astype(np.uint8) img_data = Image.fromarray(img_data) img_data = np.array(img_data.convert("RGB")) data["img_data"] = img_data return data </code></pre></div> <hr /> <h2 id="visualize-data">Visualize data</h2> Now, we write a utility function to visualize the images and their keypoints. <div class="codehilite"><pre><code># Parts of this code come from here: # https://github.com/benjiebob/StanfordExtra/blob/master/demo.ipynb def visualize_keypoints(images, keypoints): fig, axes = plt.subplots(nrows=len(images), ncols=2, figsize=(16, 12)) [ax.axis("off") for ax in np.ravel(axes)] for (ax_orig, ax_all), image, current_keypoint in zip(axes, images, keypoints): ax_orig.imshow(image) ax_all.imshow(image) # If the keypoints were formed by `imgaug` then the coordinates need # to be iterated differently. if isinstance(current_keypoint, KeypointsOnImage): for idx, kp in enumerate(current_keypoint.keypoints): ax_all.scatter( [kp.x], [kp.y], c=colours[idx], marker="x", s=50, linewidths=5, ) else: current_keypoint = np.array(current_keypoint) # Since the last entry is the visibility flag, we discard it. current_keypoint = current_keypoint[:, :2] for idx, (x, y) in enumerate(current_keypoint): ax_all.scatter([x], [y], c=colours[idx], marker="x", s=50, linewidths=5) plt.tight_layout(pad=2.0) plt.show() # Select four samples randomly for visualization. samples = list(json_dict.keys()) num_samples = 4 selected_samples = np.random.choice(samples, num_samples, replace=False) images, keypoints = [], [] for sample in selected_samples: data = get_dog(sample) image = data["img_data"] keypoint = data["joints"] images.append(image) keypoints.append(keypoint) visualize_keypoints(images, keypoints) </code></pre></div> <img alt="png" src="/img/examples/vision/keypoint_detection/keypoint_detection_18_0.png" /> The plots show that we have images of non-uniform sizes, which is expected in most real-world scenarios. However, if we resize these images to have a uniform shape (for instance (224 x 224)) their ground-truth annotations will also be affected. The same applies if we apply any geometric transformation (horizontal flip, for e.g.) to an image. Fortunately, <code>imgaug</code> provides utilities that can handle this issue. In the next section, we will write a data generator inheriting the <a href="https://keras.io/api/utils/python_utils/#sequence-class"><code>keras.utils.Sequence</code></a> class that applies data augmentation on batches of data using <code>imgaug</code>. <hr /> <h2 id="prepare-data-generator">Prepare data generator</h2> <div class="codehilite"><pre><code>class KeyPointsDataset(keras.utils.PyDataset): def __init__(self, image_keys, aug, batch_size=BATCH_SIZE, train=True, **kwargs): super().__init__(**kwargs) self.image_keys = image_keys self.aug = aug self.batch_size = batch_size self.train = train self.on_epoch_end() def __len__(self): return len(self.image_keys) // self.batch_size def on_epoch_end(self): self.indexes = np.arange(len(self.image_keys)) if self.train: np.random.shuffle(self.indexes) def __getitem__(self, index): indexes = self.indexes[index * self.batch_size : (index + 1) * self.batch_size] image_keys_temp = [self.image_keys[k] for k in indexes] (images, keypoints) = self.__data_generation(image_keys_temp) return (images, keypoints) def __data_generation(self, image_keys_temp): batch_images = np.empty((self.batch_size, IMG_SIZE, IMG_SIZE, 3), dtype="int") batch_keypoints = np.empty( (self.batch_size, 1, 1, NUM_KEYPOINTS), dtype="float32" ) for i, key in enumerate(image_keys_temp): data = get_dog(key) current_keypoint = np.array(data["joints"])[:, :2] kps = [] # To apply our data augmentation pipeline, we first need to # form Keypoint objects with the original coordinates. for j in range(0, len(current_keypoint)): kps.append(Keypoint(x=current_keypoint[j][0], y=current_keypoint[j][1])) # We then project the original image and its keypoint coordinates. current_image = data["img_data"] kps_obj = KeypointsOnImage(kps, shape=current_image.shape) # Apply the augmentation pipeline. (new_image, new_kps_obj) = self.aug(image=current_image, keypoints=kps_obj) batch_images[i,] = new_image # Parse the coordinates from the new keypoint object. kp_temp = [] for keypoint in new_kps_obj: kp_temp.append(np.nan_to_num(keypoint.x)) kp_temp.append(np.nan_to_num(keypoint.y)) # More on why this reshaping later. batch_keypoints[i,] = np.array(kp_temp).reshape(1, 1, 24 * 2) # Scale the coordinates to [0, 1] range. batch_keypoints = batch_keypoints / IMG_SIZE return (batch_images, batch_keypoints) </code></pre></div> To know more about how to operate with keypoints in <code>imgaug</code> check out <a href="https://imgaug.readthedocs.io/en/latest/source/examples_keypoints.html">this document</a>. <hr /> <h2 id="define-augmentation-transforms">Define augmentation transforms</h2> <div class="codehilite"><pre><code>train_aug = iaa.Sequential( [ iaa.Resize(IMG_SIZE, interpolation="linear"), iaa.Fliplr(0.3), # `Sometimes()` applies a function randomly to the inputs with # a given probability (0.3, in this case). iaa.Sometimes(0.3, iaa.Affine(rotate=10, scale=(0.5, 0.7))), ] ) test_aug = iaa.Sequential([iaa.Resize(IMG_SIZE, interpolation="linear")]) </code></pre></div> <hr /> <h2 id="create-training-and-validation-splits">Create training and validation splits</h2> <div class="codehilite"><pre><code>np.random.shuffle(samples) train_keys, validation_keys = ( samples[int(len(samples) * 0.15) :], samples[: int(len(samples) * 0.15)], ) </code></pre></div> <hr /> <h2 id="data-generator-investigation">Data generator investigation</h2> <div class="codehilite"><pre><code>train_dataset = KeyPointsDataset( train_keys, train_aug, workers=2, use_multiprocessing=True ) validation_dataset = KeyPointsDataset( validation_keys, test_aug, train=False, workers=2, use_multiprocessing=True ) print(f"Total batches in training set: {len(train_dataset)}") print(f"Total batches in validation set: {len(validation_dataset)}") sample_images, sample_keypoints = next(iter(train_dataset)) assert sample_keypoints.max() == 1.0 assert sample_keypoints.min() == 0.0 sample_keypoints = sample_keypoints[:4].reshape(-1, 24, 2) * IMG_SIZE visualize_keypoints(sample_images[:4], sample_keypoints) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Total batches in training set: 166 Total batches in validation set: 29 </code></pre></div> </div> <img alt="png" src="/img/examples/vision/keypoint_detection/keypoint_detection_28_1.png" /> <hr /> <h2 id="model-building">Model building</h2> The <a href="http://vision.stanford.edu/aditya86/ImageNetDogs/">Stanford dogs dataset</a> (on which the StanfordExtra dataset is based) was built using the <a href="http://image-net.org/">ImageNet-1k dataset</a>. So, it is likely that the models pretrained on the ImageNet-1k dataset would be useful for this task. We will use a MobileNetV2 pre-trained on this dataset as a backbone to extract meaningful features from the images and then pass those to a custom regression head for predicting coordinates. <div class="codehilite"><pre><code>def get_model(): # Load the pre-trained weights of MobileNetV2 and freeze the weights backbone = keras.applications.MobileNetV2( weights="imagenet", include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3), ) backbone.trainable = False inputs = layers.Input((IMG_SIZE, IMG_SIZE, 3)) x = keras.applications.mobilenet_v2.preprocess_input(inputs) x = backbone(x) x = layers.Dropout(0.3)(x) x = layers.SeparableConv2D( NUM_KEYPOINTS, kernel_size=5, strides=1, activation="relu" )(x) outputs = layers.SeparableConv2D( NUM_KEYPOINTS, kernel_size=3, strides=1, activation="sigmoid" )(x) return keras.Model(inputs, outputs, name="keypoint_detector") </code></pre></div> Our custom network is fully-convolutional which makes it more parameter-friendly than the same version of the network having fully-connected dense layers. <div class="codehilite"><pre><code>get_model().summary() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_224_no_top.h5 9406464/9406464 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step </code></pre></div> </div> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">Model: "keypoint_detector" </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ input_layer_1 (InputLayer) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ true_divide (TrueDivide) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ subtract (Subtract) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ mobilenetv2_1.00_224 │ (None, 7, 7, 1280) │ 2,257,984 │ │ (Functional) │ │ │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ dropout (Dropout) │ (None, 7, 7, 1280) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ separable_conv2d │ (None, 3, 3, 48) │ 93,488 │ │ (SeparableConv2D) │ │ │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ separable_conv2d_1 │ (None, 1, 1, 48) │ 2,784 │ │ (SeparableConv2D) │ │ │ └─────────────────────────────────┴───────────────────────────┴────────────┘ </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Total params: 2,354,256 (8.98 MB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Trainable params: 96,272 (376.06 KB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Non-trainable params: 2,257,984 (8.61 MB) </pre> Notice the output shape of the network: <code>(None, 1, 1, 48)</code>. This is why we have reshaped the coordinates as: <code>batch_keypoints[i, :] = np.array(kp_temp).reshape(1, 1, 24 * 2)</code>. <hr /> <h2 id="model-compilation-and-training">Model compilation and training</h2> For this example, we will train the network only for five epochs. <div class="codehilite"><pre><code>model = get_model() model.compile(loss="mse", optimizer=keras.optimizers.Adam(1e-4)) model.fit(train_dataset, validation_data=validation_dataset, epochs=EPOCHS) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Epoch 1/5 166/166 ━━━━━━━━━━━━━━━━━━━━ 84s 415ms/step - loss: 0.1110 - val_loss: 0.0959 Epoch 2/5 166/166 ━━━━━━━━━━━━━━━━━━━━ 79s 472ms/step - loss: 0.0874 - val_loss: 0.0802 Epoch 3/5 166/166 ━━━━━━━━━━━━━━━━━━━━ 78s 463ms/step - loss: 0.0789 - val_loss: 0.0765 Epoch 4/5 166/166 ━━━━━━━━━━━━━━━━━━━━ 78s 467ms/step - loss: 0.0769 - val_loss: 0.0731 Epoch 5/5 166/166 ━━━━━━━━━━━━━━━━━━━━ 77s 464ms/step - loss: 0.0753 - val_loss: 0.0712 <keras.src.callbacks.history.History at 0x7fb5c4299ae0> </code></pre></div> </div> <hr /> <h2 id="make-predictions-and-visualize-them">Make predictions and visualize them</h2> <div class="codehilite"><pre><code>sample_val_images, sample_val_keypoints = next(iter(validation_dataset)) sample_val_images = sample_val_images[:4] sample_val_keypoints = sample_val_keypoints[:4].reshape(-1, 24, 2) * IMG_SIZE predictions = model.predict(sample_val_images).reshape(-1, 24, 2) * IMG_SIZE # Ground-truth visualize_keypoints(sample_val_images, sample_val_keypoints) # Predictions visualize_keypoints(sample_val_images, predictions) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code> 1/1 ━━━━━━━━━━━━━━━━━━━━ 7s 7s/step </code></pre></div> </div> <img alt="png" src="/img/examples/vision/keypoint_detection/keypoint_detection_37_1.png" /> <img alt="png" src="/img/examples/vision/keypoint_detection/keypoint_detection_37_2.png" /> Predictions will likely improve with more training. <hr /> <h2 id="going-further">Going further</h2> <ul> <li>Try using other augmentation transforms from <code>imgaug</code> to investigate how that changes the results.</li> <li>Here, we transferred the features from the pre-trained network linearly that is we did not <a href="https://keras.io/guides/transfer_learning/">fine-tune</a> it. You are encouraged to fine-tune it on this task and see if that improves the performance. You can also try different architectures and see how they affect the final performance.</li> </ul> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#keypoint-detection-with-transfer-learning'>Keypoint Detection with Transfer Learning</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#data-collection'>Data collection</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#imports'>Imports</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#define-hyperparameters'>Define hyperparameters</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#load-data'>Load data</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#visualize-data'>Visualize data</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#prepare-data-generator'>Prepare data generator</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#define-augmentation-transforms'>Define augmentation transforms</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#create-training-and-validation-splits'>Create training and validation splits</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#data-generator-investigation'>Data generator investigation</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#model-building'>Model building</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#model-compilation-and-training'>Model compilation and training</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#make-predictions-and-visualize-them'>Make predictions and visualize them</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#going-further'>Going further</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>

CINXE.COM

Keypoint Detection with Transfer Learning