Model interpretability with Integrated Gradients

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/vision/integrated_gradients/" />  <meta property="og:title" content="Keras documentation: Model interpretability with Integrated Gradients"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Model interpretability with Integrated Gradients"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Model interpretability with Integrated Gradients</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink active" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink2" href="/examples/vision/image_classification_from_scratch/">Image classification from scratch</a> <a class="nav-sublink2" href="/examples/vision/mnist_convnet/">Simple MNIST convnet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_efficientnet_fine_tuning/">Image classification via fine-tuning with EfficientNet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_with_vision_transformer/">Image classification with Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/attention_mil_classification/">Classification using Attention-based Deep Multiple Instance Learning</a> <a class="nav-sublink2" href="/examples/vision/mlp_image_classification/">Image classification with modern MLP models</a> <a class="nav-sublink2" href="/examples/vision/mobilevit/">A mobile-friendly Transformer-based model for image classification</a> <a class="nav-sublink2" href="/examples/vision/xray_classification_with_tpus/">Pneumonia Classification on TPU</a> <a class="nav-sublink2" href="/examples/vision/cct/">Compact Convolutional Transformers</a> <a class="nav-sublink2" href="/examples/vision/convmixer/">Image classification with ConvMixer</a> <a class="nav-sublink2" href="/examples/vision/eanet/">Image classification with EANet (External Attention Transformer)</a> <a class="nav-sublink2" href="/examples/vision/involution/">Involutional neural networks</a> <a class="nav-sublink2" href="/examples/vision/perceiver_image_classification/">Image classification with Perceiver</a> <a class="nav-sublink2" href="/examples/vision/reptile/">Few-Shot learning with Reptile</a> <a class="nav-sublink2" href="/examples/vision/semisupervised_simclr/">Semi-supervised image classification using contrastive pretraining with SimCLR</a> <a class="nav-sublink2" href="/examples/vision/swin_transformers/">Image classification with Swin Transformers</a> <a class="nav-sublink2" href="/examples/vision/vit_small_ds/">Train a Vision Transformer on small datasets</a> <a class="nav-sublink2" href="/examples/vision/shiftvit/">A Vision Transformer without Attention</a> <a class="nav-sublink2" href="/examples/vision/image_classification_using_global_context_vision_transformer/">Image Classification using Global Context Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/temporal_latent_bottleneck/">When Recurrence meets Transformers</a> <a class="nav-sublink2" href="/examples/vision/oxford_pets_image_segmentation/">Image segmentation with a U-Net-like architecture</a> <a class="nav-sublink2" href="/examples/vision/deeplabv3_plus/">Multiclass semantic segmentation using DeepLabV3+</a> <a class="nav-sublink2" href="/examples/vision/basnet_segmentation/">Highly accurate boundaries segmentation using BASNet</a> <a class="nav-sublink2" href="/examples/vision/fully_convolutional_network/">Image Segmentation using Composable Fully-Convolutional Networks</a> <a class="nav-sublink2" href="/examples/vision/retinanet/">Object Detection with RetinaNet</a> <a class="nav-sublink2" href="/examples/vision/keypoint_detection/">Keypoint Detection with Transfer Learning</a> <a class="nav-sublink2" href="/examples/vision/object_detection_using_vision_transformer/">Object detection with Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/3D_image_classification/">3D image classification from CT scans</a> <a class="nav-sublink2" href="/examples/vision/depth_estimation/">Monocular depth estimation</a> <a class="nav-sublink2" href="/examples/vision/nerf/">3D volumetric rendering with NeRF</a> <a class="nav-sublink2" href="/examples/vision/pointnet_segmentation/">Point cloud segmentation with PointNet</a> <a class="nav-sublink2" href="/examples/vision/pointnet/">Point cloud classification</a> <a class="nav-sublink2" href="/examples/vision/captcha_ocr/">OCR model for reading Captchas</a> <a class="nav-sublink2" href="/examples/vision/handwriting_recognition/">Handwriting recognition</a> <a class="nav-sublink2" href="/examples/vision/autoencoder/">Convolutional autoencoder for image denoising</a> <a class="nav-sublink2" href="/examples/vision/mirnet/">Low-light image enhancement using MIRNet</a> <a class="nav-sublink2" href="/examples/vision/super_resolution_sub_pixel/">Image Super-Resolution using an Efficient Sub-Pixel CNN</a> <a class="nav-sublink2" href="/examples/vision/edsr/">Enhanced Deep Residual Networks for single-image super-resolution</a> <a class="nav-sublink2" href="/examples/vision/zero_dce/">Zero-DCE for low-light image enhancement</a> <a class="nav-sublink2" href="/examples/vision/cutmix/">CutMix data augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/mixup/">MixUp augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/randaugment/">RandAugment for Image Classification for Improved Robustness</a> <a class="nav-sublink2" href="/examples/vision/image_captioning/">Image captioning</a> <a class="nav-sublink2" href="/examples/vision/nl_image_search/">Natural language image search with a Dual Encoder</a> <a class="nav-sublink2" href="/examples/vision/visualizing_what_convnets_learn/">Visualizing what convnets learn</a> <a class="nav-sublink2 active" href="/examples/vision/integrated_gradients/">Model interpretability with Integrated Gradients</a> <a class="nav-sublink2" href="/examples/vision/probing_vits/">Investigating Vision Transformer representations</a> <a class="nav-sublink2" href="/examples/vision/grad_cam/">Grad-CAM class activation visualization</a> <a class="nav-sublink2" href="/examples/vision/near_dup_search/">Near-duplicate image search</a> <a class="nav-sublink2" href="/examples/vision/semantic_image_clustering/">Semantic Image Clustering</a> <a class="nav-sublink2" href="/examples/vision/siamese_contrastive/">Image similarity estimation using a Siamese Network with a contrastive loss</a> <a class="nav-sublink2" href="/examples/vision/siamese_network/">Image similarity estimation using a Siamese Network with a triplet loss</a> <a class="nav-sublink2" href="/examples/vision/metric_learning/">Metric learning for image similarity search</a> <a class="nav-sublink2" href="/examples/vision/metric_learning_tf_similarity/">Metric learning for image similarity search using TensorFlow Similarity</a> <a class="nav-sublink2" href="/examples/vision/nnclr/">Self-supervised contrastive learning with NNCLR</a> <a class="nav-sublink2" href="/examples/vision/video_classification/">Video Classification with a CNN-RNN Architecture</a> <a class="nav-sublink2" href="/examples/vision/conv_lstm/">Next-Frame Video Prediction with Convolutional LSTMs</a> <a class="nav-sublink2" href="/examples/vision/video_transformers/">Video Classification with Transformers</a> <a class="nav-sublink2" href="/examples/vision/vivit/">Video Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/bit/">Image Classification using BigTransfer (BiT)</a> <a class="nav-sublink2" href="/examples/vision/gradient_centralization/">Gradient Centralization for Better Training Performance</a> <a class="nav-sublink2" href="/examples/vision/token_learner/">Learning to tokenize in Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/knowledge_distillation/">Knowledge Distillation</a> <a class="nav-sublink2" href="/examples/vision/fixres/">FixRes: Fixing train-test resolution discrepancy</a> <a class="nav-sublink2" href="/examples/vision/cait/">Class Attention Image Transformers with LayerScale</a> <a class="nav-sublink2" href="/examples/vision/patch_convnet/">Augmenting convnets with aggregated attention</a> <a class="nav-sublink2" href="/examples/vision/learnable_resizer/">Learning to Resize</a> <a class="nav-sublink2" href="/examples/vision/adamatch/">Semi-supervision and domain adaptation with AdaMatch</a> <a class="nav-sublink2" href="/examples/vision/barlow_twins/">Barlow Twins for Contrastive SSL</a> <a class="nav-sublink2" href="/examples/vision/consistency_training/">Consistency training with supervision</a> <a class="nav-sublink2" href="/examples/vision/deit/">Distilling Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/focal_modulation_network/">Focal Modulation: A replacement for Self-Attention</a> <a class="nav-sublink2" href="/examples/vision/forwardforward/">Using the Forward-Forward Algorithm for Image Classification</a> <a class="nav-sublink2" href="/examples/vision/masked_image_modeling/">Masked image modeling with Autoencoders</a> <a class="nav-sublink2" href="/examples/vision/sam/">Segment Anything Model with 🤗Transformers</a> <a class="nav-sublink2" href="/examples/vision/segformer/">Semantic segmentation with SegFormer and Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/vision/simsiam/">Self-supervised contrastive learning with SimSiam</a> <a class="nav-sublink2" href="/examples/vision/supervised-contrastive-learning/">Supervised Contrastive Learning</a> <a class="nav-sublink2" href="/examples/vision/yolov8/">Efficient Object Detection with YOLOV8 and KerasCV</a> <a class="nav-sublink" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparam Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/vision/'>Computer Vision</a> / Model interpretability with Integrated Gradients </div> <div class='k-content'> <h1 id="model-interpretability-with-integrated-gradients">Model interpretability with Integrated Gradients</h1> Author: <a href="https://twitter.com/A_K_Nain">A_K_Nain</a> Date created: 2020/06/02 Last modified: 2020/06/02 Description: How to obtain integrated gradients for a classification model. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/integrated_gradients.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/vision/integrated_gradients.py">GitHub source</a> <hr /> <h2 id="integrated-gradients">Integrated Gradients</h2> <a href="https://arxiv.org/abs/1703.01365">Integrated Gradients</a> is a technique for attributing a classification model's prediction to its input features. It is a model interpretability technique: you can use it to visualize the relationship between input features and model predictions. Integrated Gradients is a variation on computing the gradient of the prediction output with regard to features of the input. To compute integrated gradients, we need to perform the following steps: <ol> <li> Identify the input and the output. In our case, the input is an image and the output is the last layer of our model (dense layer with softmax activation). </li> <li> Compute which features are important to a neural network when making a prediction on a particular data point. To identify these features, we need to choose a baseline input. A baseline input can be a black image (all pixel values set to zero) or random noise. The shape of the baseline input needs to be the same as our input image, e.g. (299, 299, 3). </li> <li> Interpolate the baseline for a given number of steps. The number of steps represents the steps we need in the gradient approximation for a given input image. The number of steps is a hyperparameter. The authors recommend using anywhere between 20 and 1000 steps. </li> <li> Preprocess these interpolated images and do a forward pass. </li> <li>Get the gradients for these interpolated images.</li> <li>Approximate the gradients integral using the trapezoidal rule.</li> </ol> To read in-depth about integrated gradients and why this method works, consider reading this excellent <a href="https://distill.pub/2020/attribution-baselines/">article</a>. References: <ul> <li>Integrated Gradients original <a href="https://arxiv.org/abs/1703.01365">paper</a></li> <li><a href="https://github.com/ankurtaly/Integrated-Gradients">Original implementation</a></li> </ul> <hr /> <h2 id="setup">Setup</h2> <div class="codehilite"><pre><code>import numpy as np import matplotlib.pyplot as plt from scipy import ndimage from IPython.display import Image, display import tensorflow as tf import keras from keras import layers from keras.applications import xception # Size of the input image img_size = (299, 299, 3) # Load Xception model with imagenet weights model = xception.Xception(weights="imagenet") # The local path to our target image img_path = keras.utils.get_file("elephant.jpg", "https://i.imgur.com/Bvro0YD.png") display(Image(img_path)) </code></pre></div> <img alt="jpeg" src="/img/examples/vision/integrated_gradients/integrated_gradients_3_0.jpg" /> <hr /> <h2 id="integrated-gradients-algorithm">Integrated Gradients algorithm</h2> <div class="codehilite"><pre><code>def get_img_array(img_path, size=(299, 299)): # `img` is a PIL image of size 299x299 img = keras.utils.load_img(img_path, target_size=size) # `array` is a float32 Numpy array of shape (299, 299, 3) array = keras.utils.img_to_array(img) # We add a dimension to transform our array into a "batch" # of size (1, 299, 299, 3) array = np.expand_dims(array, axis=0) return array def get_gradients(img_input, top_pred_idx): """Computes the gradients of outputs w.r.t input image. Args: img_input: 4D image tensor top_pred_idx: Predicted label for the input image Returns: Gradients of the predictions w.r.t img_input """ images = tf.cast(img_input, tf.float32) with tf.GradientTape() as tape: tape.watch(images) preds = model(images) top_class = preds[:, top_pred_idx] grads = tape.gradient(top_class, images) return grads def get_integrated_gradients(img_input, top_pred_idx, baseline=None, num_steps=50): """Computes Integrated Gradients for a predicted label. Args: img_input (ndarray): Original image top_pred_idx: Predicted label for the input image baseline (ndarray): The baseline image to start with for interpolation num_steps: Number of interpolation steps between the baseline and the input used in the computation of integrated gradients. These steps along determine the integral approximation error. By default, num_steps is set to 50. Returns: Integrated gradients w.r.t input image """ # If baseline is not provided, start with a black image # having same size as the input image. if baseline is None: baseline = np.zeros(img_size).astype(np.float32) else: baseline = baseline.astype(np.float32) # 1. Do interpolation. img_input = img_input.astype(np.float32) interpolated_image = [ baseline + (step / num_steps) * (img_input - baseline) for step in range(num_steps + 1) ] interpolated_image = np.array(interpolated_image).astype(np.float32) # 2. Preprocess the interpolated images interpolated_image = xception.preprocess_input(interpolated_image) # 3. Get the gradients grads = [] for i, img in enumerate(interpolated_image): img = tf.expand_dims(img, axis=0) grad = get_gradients(img, top_pred_idx=top_pred_idx) grads.append(grad[0]) grads = tf.convert_to_tensor(grads, dtype=tf.float32) # 4. Approximate the integral using the trapezoidal rule grads = (grads[:-1] + grads[1:]) / 2.0 avg_grads = tf.reduce_mean(grads, axis=0) # 5. Calculate integrated gradients and return integrated_grads = (img_input - baseline) * avg_grads return integrated_grads def random_baseline_integrated_gradients( img_input, top_pred_idx, num_steps=50, num_runs=2 ): """Generates a number of random baseline images. Args: img_input (ndarray): 3D image top_pred_idx: Predicted label for the input image num_steps: Number of interpolation steps between the baseline and the input used in the computation of integrated gradients. These steps along determine the integral approximation error. By default, num_steps is set to 50. num_runs: number of baseline images to generate Returns: Averaged integrated gradients for `num_runs` baseline images """ # 1. List to keep track of Integrated Gradients (IG) for all the images integrated_grads = [] # 2. Get the integrated gradients for all the baselines for run in range(num_runs): baseline = np.random.random(img_size) * 255 igrads = get_integrated_gradients( img_input=img_input, top_pred_idx=top_pred_idx, baseline=baseline, num_steps=num_steps, ) integrated_grads.append(igrads) # 3. Return the average integrated gradients for the image integrated_grads = tf.convert_to_tensor(integrated_grads) return tf.reduce_mean(integrated_grads, axis=0) </code></pre></div> <hr /> <h2 id="helper-class-for-visualizing-gradients-and-integrated-gradients">Helper class for visualizing gradients and integrated gradients</h2> <div class="codehilite"><pre><code>class GradVisualizer: """Plot gradients of the outputs w.r.t an input image.""" def __init__(self, positive_channel=None, negative_channel=None): if positive_channel is None: self.positive_channel = [0, 255, 0] else: self.positive_channel = positive_channel if negative_channel is None: self.negative_channel = [255, 0, 0] else: self.negative_channel = negative_channel def apply_polarity(self, attributions, polarity): if polarity == "positive": return np.clip(attributions, 0, 1) else: return np.clip(attributions, -1, 0) def apply_linear_transformation( self, attributions, clip_above_percentile=99.9, clip_below_percentile=70.0, lower_end=0.2, ): # 1. Get the thresholds m = self.get_thresholded_attributions( attributions, percentage=100 - clip_above_percentile ) e = self.get_thresholded_attributions( attributions, percentage=100 - clip_below_percentile ) # 2. Transform the attributions by a linear function f(x) = a*x + b such that # f(m) = 1.0 and f(e) = lower_end transformed_attributions = (1 - lower_end) * (np.abs(attributions) - e) / ( m - e ) + lower_end # 3. Make sure that the sign of transformed attributions is the same as original attributions transformed_attributions *= np.sign(attributions) # 4. Only keep values that are bigger than the lower_end transformed_attributions *= transformed_attributions >= lower_end # 5. Clip values and return transformed_attributions = np.clip(transformed_attributions, 0.0, 1.0) return transformed_attributions def get_thresholded_attributions(self, attributions, percentage): if percentage == 100.0: return np.min(attributions) # 1. Flatten the attributions flatten_attr = attributions.flatten() # 2. Get the sum of the attributions total = np.sum(flatten_attr) # 3. Sort the attributions from largest to smallest. sorted_attributions = np.sort(np.abs(flatten_attr))[::-1] # 4. Calculate the percentage of the total sum that each attribution # and the values about it contribute. cum_sum = 100.0 * np.cumsum(sorted_attributions) / total # 5. Threshold the attributions by the percentage indices_to_consider = np.where(cum_sum >= percentage)[0][0] # 6. Select the desired attributions and return attributions = sorted_attributions[indices_to_consider] return attributions def binarize(self, attributions, threshold=0.001): return attributions > threshold def morphological_cleanup_fn(self, attributions, structure=np.ones((4, 4))): closed = ndimage.grey_closing(attributions, structure=structure) opened = ndimage.grey_opening(closed, structure=structure) return opened def draw_outlines( self, attributions, percentage=90, connected_component_structure=np.ones((3, 3)), ): # 1. Binarize the attributions. attributions = self.binarize(attributions) # 2. Fill the gaps attributions = ndimage.binary_fill_holes(attributions) # 3. Compute connected components connected_components, num_comp = ndimage.label( attributions, structure=connected_component_structure ) # 4. Sum up the attributions for each component total = np.sum(attributions[connected_components > 0]) component_sums = [] for comp in range(1, num_comp + 1): mask = connected_components == comp component_sum = np.sum(attributions[mask]) component_sums.append((component_sum, mask)) # 5. Compute the percentage of top components to keep sorted_sums_and_masks = sorted(component_sums, key=lambda x: x[0], reverse=True) sorted_sums = list(zip(*sorted_sums_and_masks))[0] cumulative_sorted_sums = np.cumsum(sorted_sums) cutoff_threshold = percentage * total / 100 cutoff_idx = np.where(cumulative_sorted_sums >= cutoff_threshold)[0][0] if cutoff_idx > 2: cutoff_idx = 2 # 6. Set the values for the kept components border_mask = np.zeros_like(attributions) for i in range(cutoff_idx + 1): border_mask[sorted_sums_and_masks[i][1]] = 1 # 7. Make the mask hollow and show only the border eroded_mask = ndimage.binary_erosion(border_mask, iterations=1) border_mask[eroded_mask] = 0 # 8. Return the outlined mask return border_mask def process_grads( self, image, attributions, polarity="positive", clip_above_percentile=99.9, clip_below_percentile=0, morphological_cleanup=False, structure=np.ones((3, 3)), outlines=False, outlines_component_percentage=90, overlay=True, ): if polarity not in ["positive", "negative"]: raise ValueError( f""" Allowed polarity values: 'positive' or 'negative' but provided {polarity}""" ) if clip_above_percentile < 0 or clip_above_percentile > 100: raise ValueError("clip_above_percentile must be in [0, 100]") if clip_below_percentile < 0 or clip_below_percentile > 100: raise ValueError("clip_below_percentile must be in [0, 100]") # 1. Apply polarity if polarity == "positive": attributions = self.apply_polarity(attributions, polarity=polarity) channel = self.positive_channel else: attributions = self.apply_polarity(attributions, polarity=polarity) attributions = np.abs(attributions) channel = self.negative_channel # 2. Take average over the channels attributions = np.average(attributions, axis=2) # 3. Apply linear transformation to the attributions attributions = self.apply_linear_transformation( attributions, clip_above_percentile=clip_above_percentile, clip_below_percentile=clip_below_percentile, lower_end=0.0, ) # 4. Cleanup if morphological_cleanup: attributions = self.morphological_cleanup_fn( attributions, structure=structure ) # 5. Draw the outlines if outlines: attributions = self.draw_outlines( attributions, percentage=outlines_component_percentage ) # 6. Expand the channel axis and convert to RGB attributions = np.expand_dims(attributions, 2) * channel # 7.Superimpose on the original image if overlay: attributions = np.clip((attributions * 0.8 + image), 0, 255) return attributions def visualize( self, image, gradients, integrated_gradients, polarity="positive", clip_above_percentile=99.9, clip_below_percentile=0, morphological_cleanup=False, structure=np.ones((3, 3)), outlines=False, outlines_component_percentage=90, overlay=True, figsize=(15, 8), ): # 1. Make two copies of the original image img1 = np.copy(image) img2 = np.copy(image) # 2. Process the normal gradients grads_attr = self.process_grads( image=img1, attributions=gradients, polarity=polarity, clip_above_percentile=clip_above_percentile, clip_below_percentile=clip_below_percentile, morphological_cleanup=morphological_cleanup, structure=structure, outlines=outlines, outlines_component_percentage=outlines_component_percentage, overlay=overlay, ) # 3. Process the integrated gradients igrads_attr = self.process_grads( image=img2, attributions=integrated_gradients, polarity=polarity, clip_above_percentile=clip_above_percentile, clip_below_percentile=clip_below_percentile, morphological_cleanup=morphological_cleanup, structure=structure, outlines=outlines, outlines_component_percentage=outlines_component_percentage, overlay=overlay, ) _, ax = plt.subplots(1, 3, figsize=figsize) ax[0].imshow(image) ax[1].imshow(grads_attr.astype(np.uint8)) ax[2].imshow(igrads_attr.astype(np.uint8)) ax[0].set_title("Input") ax[1].set_title("Normal gradients") ax[2].set_title("Integrated gradients") plt.show() </code></pre></div> <hr /> <h2 id="lets-testdrive-it">Let's test-drive it</h2> <div class="codehilite"><pre><code># 1. Convert the image to numpy array img = get_img_array(img_path) # 2. Keep a copy of the original image orig_img = np.copy(img[0]).astype(np.uint8) # 3. Preprocess the image img_processed = tf.cast(xception.preprocess_input(img), dtype=tf.float32) # 4. Get model predictions preds = model.predict(img_processed) top_pred_idx = tf.argmax(preds[0]) print("Predicted:", top_pred_idx, xception.decode_predictions(preds, top=1)[0]) # 5. Get the gradients of the last layer for the predicted label grads = get_gradients(img_processed, top_pred_idx=top_pred_idx) # 6. Get the integrated gradients igrads = random_baseline_integrated_gradients( np.copy(orig_img), top_pred_idx=top_pred_idx, num_steps=50, num_runs=2 ) # 7. Process the gradients and plot vis = GradVisualizer() vis.visualize( image=orig_img, gradients=grads[0].numpy(), integrated_gradients=igrads.numpy(), clip_above_percentile=99, clip_below_percentile=0, ) vis.visualize( image=orig_img, gradients=grads[0].numpy(), integrated_gradients=igrads.numpy(), clip_above_percentile=95, clip_below_percentile=28, morphological_cleanup=True, outlines=True, ) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code> 1/1 ━━━━━━━━━━━━━━━━━━━━ 5s 5s/step WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1699486705.534012 86541 device_compiler.h:187] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process. Predicted: tf.Tensor(386, shape=(), dtype=int64) [('n02504458', 'African_elephant', 0.8871446)] </code></pre></div> </div> <img alt="png" src="/img/examples/vision/integrated_gradients/integrated_gradients_9_3.png" /> <img alt="png" src="/img/examples/vision/integrated_gradients/integrated_gradients_9_4.png" /> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#model-interpretability-with-integrated-gradients'>Model interpretability with Integrated Gradients</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#integrated-gradients'>Integrated Gradients</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#setup'>Setup</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#integrated-gradients-algorithm'>Integrated Gradients algorithm</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#helper-class-for-visualizing-gradients-and-integrated-gradients'>Helper class for visualizing gradients and integrated gradients</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#lets-testdrive-it'>Let's test-drive it</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>

CINXE.COM

Model interpretability with Integrated Gradients