Next-Frame Video Prediction with Convolutional LSTMs

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/vision/conv_lstm/" />  <meta property="og:title" content="Keras documentation: Next-Frame Video Prediction with Convolutional LSTMs"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Next-Frame Video Prediction with Convolutional LSTMs"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Next-Frame Video Prediction with Convolutional LSTMs</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink active" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink2" href="/examples/vision/image_classification_from_scratch/">Image classification from scratch</a> <a class="nav-sublink2" href="/examples/vision/mnist_convnet/">Simple MNIST convnet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_efficientnet_fine_tuning/">Image classification via fine-tuning with EfficientNet</a> <a class="nav-sublink2" href="/examples/vision/image_classification_with_vision_transformer/">Image classification with Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/attention_mil_classification/">Classification using Attention-based Deep Multiple Instance Learning</a> <a class="nav-sublink2" href="/examples/vision/mlp_image_classification/">Image classification with modern MLP models</a> <a class="nav-sublink2" href="/examples/vision/mobilevit/">A mobile-friendly Transformer-based model for image classification</a> <a class="nav-sublink2" href="/examples/vision/xray_classification_with_tpus/">Pneumonia Classification on TPU</a> <a class="nav-sublink2" href="/examples/vision/cct/">Compact Convolutional Transformers</a> <a class="nav-sublink2" href="/examples/vision/convmixer/">Image classification with ConvMixer</a> <a class="nav-sublink2" href="/examples/vision/eanet/">Image classification with EANet (External Attention Transformer)</a> <a class="nav-sublink2" href="/examples/vision/involution/">Involutional neural networks</a> <a class="nav-sublink2" href="/examples/vision/perceiver_image_classification/">Image classification with Perceiver</a> <a class="nav-sublink2" href="/examples/vision/reptile/">Few-Shot learning with Reptile</a> <a class="nav-sublink2" href="/examples/vision/semisupervised_simclr/">Semi-supervised image classification using contrastive pretraining with SimCLR</a> <a class="nav-sublink2" href="/examples/vision/swin_transformers/">Image classification with Swin Transformers</a> <a class="nav-sublink2" href="/examples/vision/vit_small_ds/">Train a Vision Transformer on small datasets</a> <a class="nav-sublink2" href="/examples/vision/shiftvit/">A Vision Transformer without Attention</a> <a class="nav-sublink2" href="/examples/vision/image_classification_using_global_context_vision_transformer/">Image Classification using Global Context Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/oxford_pets_image_segmentation/">Image segmentation with a U-Net-like architecture</a> <a class="nav-sublink2" href="/examples/vision/deeplabv3_plus/">Multiclass semantic segmentation using DeepLabV3+</a> <a class="nav-sublink2" href="/examples/vision/basnet_segmentation/">Highly accurate boundaries segmentation using BASNet</a> <a class="nav-sublink2" href="/examples/vision/fully_convolutional_network/">Image Segmentation using Composable Fully-Convolutional Networks</a> <a class="nav-sublink2" href="/examples/vision/retinanet/">Object Detection with RetinaNet</a> <a class="nav-sublink2" href="/examples/vision/keypoint_detection/">Keypoint Detection with Transfer Learning</a> <a class="nav-sublink2" href="/examples/vision/object_detection_using_vision_transformer/">Object detection with Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/3D_image_classification/">3D image classification from CT scans</a> <a class="nav-sublink2" href="/examples/vision/depth_estimation/">Monocular depth estimation</a> <a class="nav-sublink2" href="/examples/vision/nerf/">3D volumetric rendering with NeRF</a> <a class="nav-sublink2" href="/examples/vision/pointnet_segmentation/">Point cloud segmentation with PointNet</a> <a class="nav-sublink2" href="/examples/vision/pointnet/">Point cloud classification</a> <a class="nav-sublink2" href="/examples/vision/captcha_ocr/">OCR model for reading Captchas</a> <a class="nav-sublink2" href="/examples/vision/handwriting_recognition/">Handwriting recognition</a> <a class="nav-sublink2" href="/examples/vision/autoencoder/">Convolutional autoencoder for image denoising</a> <a class="nav-sublink2" href="/examples/vision/mirnet/">Low-light image enhancement using MIRNet</a> <a class="nav-sublink2" href="/examples/vision/super_resolution_sub_pixel/">Image Super-Resolution using an Efficient Sub-Pixel CNN</a> <a class="nav-sublink2" href="/examples/vision/edsr/">Enhanced Deep Residual Networks for single-image super-resolution</a> <a class="nav-sublink2" href="/examples/vision/zero_dce/">Zero-DCE for low-light image enhancement</a> <a class="nav-sublink2" href="/examples/vision/cutmix/">CutMix data augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/mixup/">MixUp augmentation for image classification</a> <a class="nav-sublink2" href="/examples/vision/randaugment/">RandAugment for Image Classification for Improved Robustness</a> <a class="nav-sublink2" href="/examples/vision/image_captioning/">Image captioning</a> <a class="nav-sublink2" href="/examples/vision/nl_image_search/">Natural language image search with a Dual Encoder</a> <a class="nav-sublink2" href="/examples/vision/visualizing_what_convnets_learn/">Visualizing what convnets learn</a> <a class="nav-sublink2" href="/examples/vision/integrated_gradients/">Model interpretability with Integrated Gradients</a> <a class="nav-sublink2" href="/examples/vision/probing_vits/">Investigating Vision Transformer representations</a> <a class="nav-sublink2" href="/examples/vision/grad_cam/">Grad-CAM class activation visualization</a> <a class="nav-sublink2" href="/examples/vision/near_dup_search/">Near-duplicate image search</a> <a class="nav-sublink2" href="/examples/vision/semantic_image_clustering/">Semantic Image Clustering</a> <a class="nav-sublink2" href="/examples/vision/siamese_contrastive/">Image similarity estimation using a Siamese Network with a contrastive loss</a> <a class="nav-sublink2" href="/examples/vision/siamese_network/">Image similarity estimation using a Siamese Network with a triplet loss</a> <a class="nav-sublink2" href="/examples/vision/metric_learning/">Metric learning for image similarity search</a> <a class="nav-sublink2" href="/examples/vision/metric_learning_tf_similarity/">Metric learning for image similarity search using TensorFlow Similarity</a> <a class="nav-sublink2" href="/examples/vision/nnclr/">Self-supervised contrastive learning with NNCLR</a> <a class="nav-sublink2" href="/examples/vision/video_classification/">Video Classification with a CNN-RNN Architecture</a> <a class="nav-sublink2 active" href="/examples/vision/conv_lstm/">Next-Frame Video Prediction with Convolutional LSTMs</a> <a class="nav-sublink2" href="/examples/vision/video_transformers/">Video Classification with Transformers</a> <a class="nav-sublink2" href="/examples/vision/vivit/">Video Vision Transformer</a> <a class="nav-sublink2" href="/examples/vision/bit/">Image Classification using BigTransfer (BiT)</a> <a class="nav-sublink2" href="/examples/vision/gradient_centralization/">Gradient Centralization for Better Training Performance</a> <a class="nav-sublink2" href="/examples/vision/token_learner/">Learning to tokenize in Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/knowledge_distillation/">Knowledge Distillation</a> <a class="nav-sublink2" href="/examples/vision/fixres/">FixRes: Fixing train-test resolution discrepancy</a> <a class="nav-sublink2" href="/examples/vision/cait/">Class Attention Image Transformers with LayerScale</a> <a class="nav-sublink2" href="/examples/vision/patch_convnet/">Augmenting convnets with aggregated attention</a> <a class="nav-sublink2" href="/examples/vision/learnable_resizer/">Learning to Resize</a> <a class="nav-sublink2" href="/examples/vision/adamatch/">Semi-supervision and domain adaptation with AdaMatch</a> <a class="nav-sublink2" href="/examples/vision/barlow_twins/">Barlow Twins for Contrastive SSL</a> <a class="nav-sublink2" href="/examples/vision/consistency_training/">Consistency training with supervision</a> <a class="nav-sublink2" href="/examples/vision/deit/">Distilling Vision Transformers</a> <a class="nav-sublink2" href="/examples/vision/focal_modulation_network/">Focal Modulation: A replacement for Self-Attention</a> <a class="nav-sublink2" href="/examples/vision/forwardforward/">Using the Forward-Forward Algorithm for Image Classification</a> <a class="nav-sublink2" href="/examples/vision/masked_image_modeling/">Masked image modeling with Autoencoders</a> <a class="nav-sublink2" href="/examples/vision/sam/">Segment Anything Model with 🤗Transformers</a> <a class="nav-sublink2" href="/examples/vision/segformer/">Semantic segmentation with SegFormer and Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/vision/simsiam/">Self-supervised contrastive learning with SimSiam</a> <a class="nav-sublink2" href="/examples/vision/supervised-contrastive-learning/">Supervised Contrastive Learning</a> <a class="nav-sublink2" href="/examples/vision/temporal_latent_bottleneck/">When Recurrence meets Transformers</a> <a class="nav-sublink2" href="/examples/vision/yolov8/">Efficient Object Detection with YOLOV8 and KerasCV</a> <a class="nav-sublink" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparam Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/vision/'>Computer Vision</a> / Next-Frame Video Prediction with Convolutional LSTMs </div> <div class='k-content'> <h1 id="nextframe-video-prediction-with-convolutional-lstms">Next-Frame Video Prediction with Convolutional LSTMs</h1> Author: <a href="https://github.com/amogh7joshi">Amogh Joshi</a> Date created: 2021/06/02 Last modified: 2023/11/10 Description: How to build and train a convolutional LSTM model for next-frame video prediction. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/conv_lstm.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/vision/conv_lstm.py">GitHub source</a> <hr /> <h2 id="introduction">Introduction</h2> The <a href="https://papers.nips.cc/paper/2015/file/07563a3fe3bbe7e3ba84431ad9d055af-Paper.pdf">Convolutional LSTM</a> architectures bring together time series processing and computer vision by introducing a convolutional recurrent cell in a LSTM layer. In this example, we will explore the Convolutional LSTM model in an application to next-frame prediction, the process of predicting what video frames come next given a series of past frames. <hr /> <h2 id="setup">Setup</h2> <div class="codehilite"><pre><code>import numpy as np import matplotlib.pyplot as plt import keras from keras import layers import io import imageio from IPython.display import Image, display from ipywidgets import widgets, Layout, HBox </code></pre></div> <hr /> <h2 id="dataset-construction">Dataset Construction</h2> For this example, we will be using the <a href="http://www.cs.toronto.edu/~nitish/unsupervised_video/">Moving MNIST</a> dataset. We will download the dataset and then construct and preprocess training and validation sets. For next-frame prediction, our model will be using a previous frame, which we'll call <code>f_n</code>, to predict a new frame, called <code>f_(n + 1)</code>. To allow the model to create these predictions, we'll need to process the data such that we have "shifted" inputs and outputs, where the input data is frame <code>x_n</code>, being used to predict frame <code>y_(n + 1)</code>. <div class="codehilite"><pre><code># Download and load the dataset. fpath = keras.utils.get_file( "moving_mnist.npy", "http://www.cs.toronto.edu/~nitish/unsupervised_video/mnist_test_seq.npy", ) dataset = np.load(fpath) # Swap the axes representing the number of frames and number of data samples. dataset = np.swapaxes(dataset, 0, 1) # We'll pick out 1000 of the 10000 total examples and use those. dataset = dataset[:1000, ...] # Add a channel dimension since the images are grayscale. dataset = np.expand_dims(dataset, axis=-1) # Split into train and validation sets using indexing to optimize memory. indexes = np.arange(dataset.shape[0]) np.random.shuffle(indexes) train_index = indexes[: int(0.9 * dataset.shape[0])] val_index = indexes[int(0.9 * dataset.shape[0]) :] train_dataset = dataset[train_index] val_dataset = dataset[val_index] # Normalize the data to the 0-1 range. train_dataset = train_dataset / 255 val_dataset = val_dataset / 255 # We'll define a helper function to shift the frames, where # `x` is frames 0 to n - 1, and `y` is frames 1 to n. def create_shifted_frames(data): x = data[:, 0 : data.shape[1] - 1, :, :] y = data[:, 1 : data.shape[1], :, :] return x, y # Apply the processing function to the datasets. x_train, y_train = create_shifted_frames(train_dataset) x_val, y_val = create_shifted_frames(val_dataset) # Inspect the dataset. print("Training Dataset Shapes: " + str(x_train.shape) + ", " + str(y_train.shape)) print("Validation Dataset Shapes: " + str(x_val.shape) + ", " + str(y_val.shape)) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Downloading data from http://www.cs.toronto.edu/~nitish/unsupervised_video/mnist_test_seq.npy 819200096/819200096 ━━━━━━━━━━━━━━━━━━━━ 116s 0us/step Training Dataset Shapes: (900, 19, 64, 64, 1), (900, 19, 64, 64, 1) Validation Dataset Shapes: (100, 19, 64, 64, 1), (100, 19, 64, 64, 1) </code></pre></div> </div> <hr /> <h2 id="data-visualization">Data Visualization</h2> Our data consists of sequences of frames, each of which are used to predict the upcoming frame. Let's take a look at some of these sequential frames. <div class="codehilite"><pre><code># Construct a figure on which we will visualize the images. fig, axes = plt.subplots(4, 5, figsize=(10, 8)) # Plot each of the sequential images for one random data example. data_choice = np.random.choice(range(len(train_dataset)), size=1)[0] for idx, ax in enumerate(axes.flat): ax.imshow(np.squeeze(train_dataset[data_choice][idx]), cmap="gray") ax.set_title(f"Frame {idx + 1}") ax.axis("off") # Print information and display the figure. print(f"Displaying frames for example {data_choice}.") plt.show() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Displaying frames for example 95. </code></pre></div> </div> <img alt="png" src="/img/examples/vision/conv_lstm/conv_lstm_7_1.png" /> <hr /> <h2 id="model-construction">Model Construction</h2> To build a Convolutional LSTM model, we will use the <code>ConvLSTM2D</code> layer, which will accept inputs of shape <code>(batch_size, num_frames, width, height, channels)</code>, and return a prediction movie of the same shape. <div class="codehilite"><pre><code># Construct the input layer with no definite frame size. inp = layers.Input(shape=(None, *x_train.shape[2:])) # We will construct 3 `ConvLSTM2D` layers with batch normalization, # followed by a `Conv3D` layer for the spatiotemporal outputs. x = layers.ConvLSTM2D( filters=64, kernel_size=(5, 5), padding="same", return_sequences=True, activation="relu", )(inp) x = layers.BatchNormalization()(x) x = layers.ConvLSTM2D( filters=64, kernel_size=(3, 3), padding="same", return_sequences=True, activation="relu", )(x) x = layers.BatchNormalization()(x) x = layers.ConvLSTM2D( filters=64, kernel_size=(1, 1), padding="same", return_sequences=True, activation="relu", )(x) x = layers.Conv3D( filters=1, kernel_size=(3, 3, 3), activation="sigmoid", padding="same" )(x) # Next, we will build the complete model and compile it. model = keras.models.Model(inp, x) model.compile( loss=keras.losses.binary_crossentropy, optimizer=keras.optimizers.Adam(), ) </code></pre></div> <hr /> <h2 id="model-training">Model Training</h2> With our model and data constructed, we can now train the model. <div class="codehilite"><pre><code># Define some callbacks to improve training. early_stopping = keras.callbacks.EarlyStopping(monitor="val_loss", patience=10) reduce_lr = keras.callbacks.ReduceLROnPlateau(monitor="val_loss", patience=5) # Define modifiable training hyperparameters. epochs = 20 batch_size = 5 # Fit the model to the training data. model.fit( x_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(x_val, y_val), callbacks=[early_stopping, reduce_lr], ) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Epoch 1/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 50s 226ms/step - loss: 0.1510 - val_loss: 0.2966 - learning_rate: 0.0010 Epoch 2/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0287 - val_loss: 0.1766 - learning_rate: 0.0010 Epoch 3/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0269 - val_loss: 0.0661 - learning_rate: 0.0010 Epoch 4/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0264 - val_loss: 0.0279 - learning_rate: 0.0010 Epoch 5/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0258 - val_loss: 0.0254 - learning_rate: 0.0010 Epoch 6/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0256 - val_loss: 0.0253 - learning_rate: 0.0010 Epoch 7/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0251 - val_loss: 0.0248 - learning_rate: 0.0010 Epoch 8/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0251 - val_loss: 0.0251 - learning_rate: 0.0010 Epoch 9/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0247 - val_loss: 0.0243 - learning_rate: 0.0010 Epoch 10/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0246 - val_loss: 0.0246 - learning_rate: 0.0010 Epoch 11/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0245 - val_loss: 0.0247 - learning_rate: 0.0010 Epoch 12/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0241 - val_loss: 0.0243 - learning_rate: 0.0010 Epoch 13/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0244 - val_loss: 0.0245 - learning_rate: 0.0010 Epoch 14/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0241 - val_loss: 0.0241 - learning_rate: 0.0010 Epoch 15/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0243 - val_loss: 0.0241 - learning_rate: 0.0010 Epoch 16/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0242 - val_loss: 0.0242 - learning_rate: 0.0010 Epoch 17/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0240 - val_loss: 0.0240 - learning_rate: 0.0010 Epoch 18/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0240 - val_loss: 0.0243 - learning_rate: 0.0010 Epoch 19/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0240 - val_loss: 0.0244 - learning_rate: 0.0010 Epoch 20/20 180/180 ━━━━━━━━━━━━━━━━━━━━ 40s 219ms/step - loss: 0.0237 - val_loss: 0.0238 - learning_rate: 1.0000e-04 <keras.src.callbacks.history.History at 0x7ff294f9c340> </code></pre></div> </div> <hr /> <h2 id="frame-prediction-visualizations">Frame Prediction Visualizations</h2> With our model now constructed and trained, we can generate some example frame predictions based on a new video. We'll pick a random example from the validation set and then choose the first ten frames from them. From there, we can allow the model to predict 10 new frames, which we can compare to the ground truth frame predictions. <div class="codehilite"><pre><code># Select a random example from the validation dataset. example = val_dataset[np.random.choice(range(len(val_dataset)), size=1)[0]] # Pick the first/last ten frames from the example. frames = example[:10, ...] original_frames = example[10:, ...] # Predict a new set of 10 frames. for _ in range(10): # Extract the model's prediction and post-process it. new_prediction = model.predict(np.expand_dims(frames, axis=0)) new_prediction = np.squeeze(new_prediction, axis=0) predicted_frame = np.expand_dims(new_prediction[-1, ...], axis=0) # Extend the set of prediction frames. frames = np.concatenate((frames, predicted_frame), axis=0) # Construct a figure for the original and new frames. fig, axes = plt.subplots(2, 10, figsize=(20, 4)) # Plot the original frames. for idx, ax in enumerate(axes[0]): ax.imshow(np.squeeze(original_frames[idx]), cmap="gray") ax.set_title(f"Frame {idx + 11}") ax.axis("off") # Plot the new frames. new_frames = frames[10:, ...] for idx, ax in enumerate(axes[1]): ax.imshow(np.squeeze(new_frames[idx]), cmap="gray") ax.set_title(f"Frame {idx + 11}") ax.axis("off") # Display the figure. plt.show() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code> 1/1 ━━━━━━━━━━━━━━━━━━━━ 2s 2s/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 800ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 805ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 790ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 821ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 824ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 928ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 813ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 810ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 814ms/step </code></pre></div> </div> <img alt="png" src="/img/examples/vision/conv_lstm/conv_lstm_13_1.png" /> <hr /> <h2 id="predicted-videos">Predicted Videos</h2> Finally, we'll pick a few examples from the validation set and construct some GIFs with them to see the model's predicted videos. You can use the trained model hosted on <a href="https://huggingface.co/keras-io/conv-lstm">Hugging Face Hub</a> and try the demo on <a href="https://huggingface.co/spaces/keras-io/conv-lstm">Hugging Face Spaces</a>. <div class="codehilite"><pre><code># Select a few random examples from the dataset. examples = val_dataset[np.random.choice(range(len(val_dataset)), size=5)] # Iterate over the examples and predict the frames. predicted_videos = [] for example in examples: # Pick the first/last ten frames from the example. frames = example[:10, ...] original_frames = example[10:, ...] new_predictions = np.zeros(shape=(10, *frames[0].shape)) # Predict a new set of 10 frames. for i in range(10): # Extract the model's prediction and post-process it. frames = example[: 10 + i + 1, ...] new_prediction = model.predict(np.expand_dims(frames, axis=0)) new_prediction = np.squeeze(new_prediction, axis=0) predicted_frame = np.expand_dims(new_prediction[-1, ...], axis=0) # Extend the set of prediction frames. new_predictions[i] = predicted_frame # Create and save GIFs for each of the ground truth/prediction images. for frame_set in [original_frames, new_predictions]: # Construct a GIF from the selected video frames. current_frames = np.squeeze(frame_set) current_frames = current_frames[..., np.newaxis] * np.ones(3) current_frames = (current_frames * 255).astype(np.uint8) current_frames = list(current_frames) # Construct a GIF from the frames. with io.BytesIO() as gif: imageio.mimsave(gif, current_frames, "GIF", duration=200) predicted_videos.append(gif.getvalue()) # Display the videos. print(" Truth\tPrediction") for i in range(0, len(predicted_videos), 2): # Construct and display an `HBox` with the ground truth and prediction. box = HBox( [ widgets.Image(value=predicted_videos[i]), widgets.Image(value=predicted_videos[i + 1]), ] ) display(box) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code> 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 790ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step Truth Prediction HBox(children=(Image(value=b'GIF89a@\x00@\x00\x87\x00\x00\xff\xff\xff\xfe\xfe\xfe\xfd\xfd\xfd\xfc\xfc\xfc\xf8\… HBox(children=(Image(value=b'GIF89a@\x00@\x00\x86\x00\x00\xff\xff\xff\xfd\xfd\xfd\xfc\xfc\xfc\xfb\xfb\xfb\xf4\… HBox(children=(Image(value=b'GIF89a@\x00@\x00\x86\x00\x00\xff\xff\xff\xfe\xfe\xfe\xfd\xfd\xfd\xfc\xfc\xfc\xfb\… HBox(children=(Image(value=b'GIF89a@\x00@\x00\x86\x00\x00\xff\xff\xff\xfe\xfe\xfe\xfd\xfd\xfd\xfc\xfc\xfc\xfb\… HBox(children=(Image(value=b'GIF89a@\x00@\x00\x86\x00\x00\xff\xff\xff\xfd\xfd\xfd\xfc\xfc\xfc\xf9\xf9\xf9\xf7\… </code></pre></div> </div> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#nextframe-video-prediction-with-convolutional-lstms'>Next-Frame Video Prediction with Convolutional LSTMs</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#setup'>Setup</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#dataset-construction'>Dataset Construction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#data-visualization'>Data Visualization</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#model-construction'>Model Construction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#model-training'>Model Training</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#frame-prediction-visualizations'>Frame Prediction Visualizations</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#predicted-videos'>Predicted Videos</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>

CINXE.COM

Next-Frame Video Prediction with Convolutional LSTMs