CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/getting_started/faq/" />  <meta property="og:title" content="Keras documentation: Keras FAQ"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Keras FAQ"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Keras FAQ</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link active" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-sublink" href="/getting_started/intro_to_keras_for_engineers/">Introduction to Keras for engineers</a> <a class="nav-sublink" href="/getting_started/benchmarks/">Keras 3 benchmarks</a> <a class="nav-sublink" href="/getting_started/ecosystem/">The Keras ecosystem</a> <a class="nav-sublink active" href="/getting_started/faq/">Frequently Asked Questions</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparam Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/getting_started/'>Getting started</a> / Keras FAQ </div> <div class='k-content'> <h1 id="keras-faq">Keras FAQ</h1> A list of frequently Asked Keras Questions. <h2 id="general-questions">General questions</h2> <ul> <li><a href="#how-can-i-train-a-keras-model-on-multiple-gpus-on-a-single-machine">How can I train a Keras model on multiple GPUs (on a single machine)?</a></li> <li><a href="#how-can-i-train-a-keras-model-on-tpu">How can I train a Keras model on TPU?</a></li> <li><a href="#where-is-the-keras-configuration-file-stored">Where is the Keras configuration file stored?</a></li> <li><a href="#how-to-do-hyperparameter-tuning-with-keras">How to do hyperparameter tuning with Keras?</a></li> <li><a href="#how-can-i-obtain-reproducible-results-using-keras-during-development">How can I obtain reproducible results using Keras during development?</a></li> <li><a href="#what-are-my-options-for-saving-models">What are my options for saving models?</a></li> <li><a href="#how-can-i-install-hdf5-or-h5py-to-save-my-models">How can I install HDF5 or h5py to save my models?</a></li> <li><a href="#how-should-i-cite-keras">How should I cite Keras?</a></li> </ul> <h2 id="trainingrelated-questions">Training-related questions</h2> <ul> <li><a href="#what-do-sample-batch-and-epoch-mean">What do "sample", "batch", and "epoch" mean?</a></li> <li><a href="#why-is-my-training-loss-much-higher-than-my-testing-loss">Why is my training loss much higher than my testing loss?</a></li> <li><a href="#how-can-i-ensure-my-training-run-can-recover-from-program-interruptions">How can I ensure my training run can recover from program interruptions?</a></li> <li><a href="#how-can-i-interrupt-training-when-the-validation-loss-isnt-decreasing-anymore">How can I interrupt training when the validation loss isn't decreasing anymore?</a></li> <li><a href="#how-can-i-freeze-layers-and-do-finetuning">How can I freeze layers and do fine-tuning?</a></li> <li><a href="#whats-the-difference-between-the-training-argument-in-call-and-the-trainable-attribute">What's the difference between the <code>training</code> argument in <code>call()</code> and the <code>trainable</code> attribute?</a></li> <li><a href="#in-fit-how-is-the-validation-split-computed">In <code>fit()</code>, how is the validation split computed?</a></li> <li><a href="#in-fit-is-the-data-shuffled-during-training">In <code>fit()</code>, is the data shuffled during training?</a></li> <li><a href="#whats-the-recommended-way-to-monitor-my-metrics-when-training-with-fit">What's the recommended way to monitor my metrics when training with <code>fit()</code>?</a></li> <li><a href="#what-if-i-need-to-customize-what-fit-does">What if I need to customize what <code>fit()</code> does?</a></li> <li><a href="#whats-the-difference-between-model-methods-predict-and-call">What's the difference between <code>Model</code> methods <code>predict()</code> and <code>__call__()</code>?</a></li> </ul> <h2 id="modelingrelated-questions">Modeling-related questions</h2> <ul> <li><a href="#how-can-i-obtain-the-output-of-an-intermediate-layer-feature-extraction">How can I obtain the output of an intermediate layer (feature extraction)?</a></li> <li><a href="#how-can-i-use-pre-trained-models-in-keras">How can I use pre-trained models in Keras?</a></li> <li><a href="#how-can-i-use-stateful-rnns">How can I use stateful RNNs?</a></li> </ul> <hr /> <h2 id="general-questions">General questions</h2> <h3 id="how-can-i-train-a-keras-model-on-multiple-gpus-on-a-single-machine">How can I train a Keras model on multiple GPUs (on a single machine)?</h3> There are two ways to run a single model on multiple GPUs: data parallelism and device parallelism. Keras covers both. For data parallelism, Keras supports the built-in data parallel distribution APIs of JAX, TensorFlow, and PyTorch. See the following guides: <ul> <li><a href="/guides/distributed_training_with_jax/">Multi-GPU distributed training with JAX</a></li> <li><a href="/guides/distributed_training_with_tensorflow/">Multi-GPU distributed training with TensorFlow</a></li> <li><a href="/guides/distributed_training_with_torch/">Multi-GPU distributed training with PyTorch</a></li> </ul> For model parallelism, Keras has its own distribution API, which is currently only support by the JAX backend. See <a href="/api/distribution/">the documentation for the <code>LayoutMap</code> API</a>. <hr /> <h3 id="how-can-i-train-a-keras-model-on-tpu">How can I train a Keras model on TPU?</h3> TPUs are a fast & efficient hardware accelerator for deep learning that is publicly available on Google Cloud. You can use TPUs via Colab, Kaggle notebooks, and GCP Deep Learning VMs (provided the <code>TPU_NAME</code> environment variable is set on the VM). All Keras backends (JAX, TensorFlow, PyTorch) are supported on TPU, but we recommend JAX or TensorFlow in this case. Using JAX: When connected to a TPU runtime, just insert this code snippet before model construction: <div class="codehilite"><pre><code>import jax distribution = keras.distribution.DataParallel(devices=jax.devices()) keras.distribution.set_distribution(distribution) </code></pre></div> Using TensorFlow: When connected to a TPU runtime, use <code>TPUClusterResolver</code> to detect the TPU. Then, create <code>TPUStrategy</code> and construct your model in the strategy scope: <div class="codehilite"><pre><code>try: tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect() print("Device:", tpu.master()) strategy = tf.distribute.TPUStrategy(tpu) except: strategy = tf.distribute.get_strategy() print("Number of replicas:", strategy.num_replicas_in_sync) with strategy.scope(): # Create your model here. ... </code></pre></div> Importantly, you should: <ul> <li>Make sure you are able to read your data fast enough to keep the TPU utilized.</li> <li>Consider running multiple steps of gradient descent per graph execution in order to keep the TPU utilized. You can do this via the <code>experimental_steps_per_execution</code> argument <code>compile()</code>. It will yield a significant speed up for small models.</li> </ul> <hr /> <h3 id="where-is-the-keras-configuration-file-stored">Where is the Keras configuration file stored?</h3> The default directory where all Keras data is stored is: <code>$HOME/.keras/</code> For instance, for me, on a MacBook Pro, it's <code>/Users/fchollet/.keras/</code>. Note that Windows users should replace <code>$HOME</code> with <code>%USERPROFILE%</code>. In case Keras cannot create the above directory (e.g. due to permission issues), <code>/tmp/.keras/</code> is used as a backup. The Keras configuration file is a JSON file stored at <code>$HOME/.keras/keras.json</code>. The default configuration file looks like this: <div class="codehilite"><pre><code>{ "image_data_format": "channels_last", "epsilon": 1e-07, "floatx": "float32", "backend": "tensorflow" } </code></pre></div> It contains the following fields: <ul> <li>The image data format to be used as default by image processing layers and utilities (either <code>channels_last</code> or <code>channels_first</code>).</li> <li>The <code>epsilon</code> numerical fuzz factor to be used to prevent division by zero in some operations.</li> <li>The default float data type.</li> <li>The default backend. It can be one of <code>"jax"</code>, <code>"tensorflow"</code>, <code>"torch"</code>, or <code>"numpy"</code>.</li> </ul> Likewise, cached dataset files, such as those downloaded with <a href="/utils/#get_file"><code>get_file()</code></a>, are stored by default in <code>$HOME/.keras/datasets/</code>, and cached model weights files from Keras Applications are stored by default in <code>$HOME/.keras/models/</code>. <hr /> <h3 id="how-to-do-hyperparameter-tuning-with-keras">How to do hyperparameter tuning with Keras?</h3> We recommend using <a href="https://keras.io/keras_tuner/">KerasTuner</a>. <hr /> <h3 id="how-can-i-obtain-reproducible-results-using-keras-during-development">How can I obtain reproducible results using Keras during development?</h3> There are four sources of randomness to consider: <ol> <li>Keras itself (e.g. <code>keras.random</code> ops or random layers from <code>keras.layers</code>).</li> <li>The current Keras backend (e.g. JAX, TensorFlow, or PyTorch).</li> <li>The Python runtime.</li> <li>The CUDA runtime. When running on a GPU, some operations have non-deterministic outputs. This is due to the fact that GPUs run many operations in parallel, so the order of execution is not always guaranteed. Due to the limited precision of floats, even adding several numbers together may give slightly different results depending on the order in which you add them.</li> </ol> To make both Keras and the current backend framework deterministic, use this: <div class="codehilite"><pre><code>keras.utils.set_random_seed(1337) </code></pre></div> To make Python deterministic, you need to set the <code>PYTHONHASHSEED</code> environment variable to <code>0</code> before the program starts (not within the program itself). This is necessary in Python 3.2.3 onwards to have reproducible behavior for certain hash-based operations (e.g., the item order in a set or a dict, see <a href="https://docs.python.org/3.7/using/cmdline.html#envvar-PYTHONHASHSEED">Python's documentation</a>). To make the CUDA runtime deterministic: if using the TensorFlow backend, call <a href="https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism"><code>tf.config.experimental.enable_op_determinism</code></a>. Note that this will have a performance cost. What to do for other backends may vary – check the documentation of your backend framework directly. <hr /> <h3 id="what-are-my-options-for-saving-models">What are my options for saving models?</h3> Note: it is not recommended to use pickle or cPickle to save a Keras model. 1) Whole-model saving (configuration + weights) Whole-model saving means creating a file that will contain: <ul> <li>the architecture of the model, allowing you to re-create the model</li> <li>the weights of the model</li> <li>the training configuration (loss, optimizer)</li> <li>the state of the optimizer, allowing you to resume training exactly where you left off.</li> </ul> The default and recommended way to save a whole model is to just do: <code>model.save(your_file_path.keras)</code>. After saving a model in either format, you can reinstantiate it via <code>model = keras.models.load_model(your_file_path.keras)</code>. Example: <div class="codehilite"><pre><code>from keras.saving import load_model model.save('my_model.keras') del model # deletes the existing model # returns a compiled model # identical to the previous one model = load_model('my_model.keras') </code></pre></div> 2) Weights-only saving If you need to save the weights of a model, you can do so in HDF5 with the code below, using the file extension <code>.weights.h5</code>: <div class="codehilite"><pre><code>model.save_weights('my_model.weights.h5') </code></pre></div> Assuming you have code for instantiating your model, you can then load the weights you saved into a model with the same architecture: <div class="codehilite"><pre><code>model.load_weights('my_model.weights.h5') </code></pre></div> If you need to load the weights into a different architecture (with some layers in common), for instance for fine-tuning or transfer-learning, you can load them by layer name: <div class="codehilite"><pre><code>model.load_weights('my_model.weights.h5', by_name=True) </code></pre></div> Example: <div class="codehilite"><pre><code>""" Assuming the original model looks like this: model = Sequential() model.add(Dense(2, input_dim=3, name='dense_1')) model.add(Dense(3, name='dense_2')) ... model.save_weights(fname) """ # new model model = Sequential() model.add(Dense(2, input_dim=3, name='dense_1')) # will be loaded model.add(Dense(10, name='new_dense')) # will not be loaded # load weights from the first model; will only affect the first layer, dense_1. model.load_weights(fname, by_name=True) </code></pre></div> See also <a href="#how-can-i-install-hdf5-or-h5py-to-save-my-models">How can I install HDF5 or h5py to save my models?</a> for instructions on how to install <code>h5py</code>. 3) Configuration-only saving (serialization) If you only need to save the architecture of a model, and not its weights or its training configuration, you can do: <div class="codehilite"><pre><code># save as JSON json_string = model.to_json() </code></pre></div> The generated JSON file is human-readable and can be manually edited if needed. You can then build a fresh model from this data: <div class="codehilite"><pre><code># model reconstruction from JSON: from keras.models import model_from_json model = model_from_json(json_string) </code></pre></div> 4) Handling custom layers (or other custom objects) in saved models If the model you want to load includes custom layers or other custom classes or functions, you can pass them to the loading mechanism via the <code>custom_objects</code> argument: <div class="codehilite"><pre><code>from keras.models import load_model # Assuming your model includes instance of an "AttentionLayer" class model = load_model('my_model.h5', custom_objects={'AttentionLayer': AttentionLayer}) </code></pre></div> Alternatively, you can use a <a href="https://keras.io/utils/#customobjectscope">custom object scope</a>: <div class="codehilite"><pre><code>from keras.utils import CustomObjectScope with CustomObjectScope({'AttentionLayer': AttentionLayer}): model = load_model('my_model.h5') </code></pre></div> Custom objects handling works the same way for <code>load_model</code> & <code>model_from_json</code>: <div class="codehilite"><pre><code>from keras.models import model_from_json model = model_from_json(json_string, custom_objects={'AttentionLayer': AttentionLayer}) </code></pre></div> <hr /> <h3 id="how-can-i-install-hdf5-or-h5py-to-save-my-models">How can I install HDF5 or h5py to save my models?</h3> In order to save your Keras models as HDF5 files, Keras uses the h5py Python package. It is a dependency of Keras and should be installed by default. On Debian-based distributions, you will have to additionally install <code>libhdf5</code>: <div class="k-default-code-block"> <div class="codehilite"><pre><code>sudo apt-get install libhdf5-serial-dev </code></pre></div> </div> If you are unsure if h5py is installed you can open a Python shell and load the module via <div class="codehilite"><pre><code>import h5py </code></pre></div> If it imports without error it is installed, otherwise you can find <a href="http://docs.h5py.org/en/latest/build.html">detailed installation instructions here</a>. <hr /> <h3 id="how-should-i-cite-keras">How should I cite Keras?</h3> Please cite Keras in your publications if it helps your research. Here is an example BibTeX entry: <code style="color: gray;"> @misc{chollet2015keras,   title={Keras},   author={Chollet, Fran\c{c}ois and others},   year={2015},   howpublished={\url{https://keras.io}}, } </code> <hr /> <h2 id="trainingrelated-questions">Training-related questions</h2> <h3 id="what-do-sample-batch-and-epoch-mean">What do "sample", "batch", and "epoch" mean?</h3> Below are some common definitions that are necessary to know and understand to correctly utilize Keras <code>fit()</code>: <ul> <li>Sample: one element of a dataset. For instance, one image is a sample in a convolutional network. One audio snippet is a sample for a speech recognition model.</li> </ul> <ul> <li>Batch: a set of N samples. The samples in a batch are processed independently, in parallel. If training, a batch results in only one update to the model. A batch generally approximates the distribution of the input data better than a single input. The larger the batch, the better the approximation; however, it is also true that the batch will take longer to process and will still result in only one update. For inference (evaluate/predict), it is recommended to pick a batch size that is as large as you can afford without going out of memory (since larger batches will usually result in faster evaluation/prediction).</li> </ul> <ul> <li>Epoch: an arbitrary cutoff, generally defined as "one pass over the entire dataset", used to separate training into distinct phases, which is useful for logging and periodic evaluation. When using <code>validation_data</code> or <code>validation_split</code> with the <code>fit</code> method of Keras models, evaluation will be run at the end of every epoch. Within Keras, there is the ability to add <a href="/api/callbacks/">callbacks</a> specifically designed to be run at the end of an epoch. Examples of these are learning rate changes and model checkpointing (saving).</li> </ul> <hr /> <h3 id="why-is-my-training-loss-much-higher-than-my-testing-loss">Why is my training loss much higher than my testing loss?</h3> A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time. They are reflected in the training time loss but not in the test time loss. Besides, the training loss that Keras displays is the average of the losses for each batch of training data, over the current epoch. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. This can bring the epoch-wise average down. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss. <hr /> <h3 id="how-can-i-ensure-my-training-run-can-recover-from-program-interruptions">How can I ensure my training run can recover from program interruptions?</h3> To ensure the ability to recover from an interrupted training run at any time (fault tolerance), you should use a <a href="/api/callbacks/backup_and_restore#backupandrestore-class"><code>keras.callbacks.BackupAndRestore</code></a> callback that regularly saves your training progress, including the epoch number and weights, to disk, and loads it the next time you call <code>Model.fit()</code>. <div class="codehilite"><pre><code>import numpy as np import keras class InterruptingCallback(keras.callbacks.Callback): """A callback to intentionally introduce interruption to training.""" def on_epoch_end(self, epoch, log=None): if epoch == 15: raise RuntimeError('Interruption') model = keras.Sequential([keras.layers.Dense(10)]) optimizer = keras.optimizers.SGD() model.compile(optimizer, loss="mse") x = np.random.random((24, 10)) y = np.random.random((24,)) backup_callback = keras.callbacks.experimental.BackupAndRestore( backup_dir='/tmp/backup') try: model.fit(x, y, epochs=20, steps_per_epoch=5, callbacks=[backup_callback, InterruptingCallback()]) except RuntimeError: print('***Handling interruption***') # This continues at the epoch where it left off. model.fit(x, y, epochs=20, steps_per_epoch=5, callbacks=[backup_callback]) </code></pre></div> Find out more in the <a href="/api/callbacks/">callbacks documentation</a>. <hr /> <h3 id="how-can-i-interrupt-training-when-the-validation-loss-isnt-decreasing-anymore">How can I interrupt training when the validation loss isn't decreasing anymore?</h3> You can use an <code>EarlyStopping</code> callback: <div class="codehilite"><pre><code>from keras.callbacks import EarlyStopping early_stopping = EarlyStopping(monitor='val_loss', patience=2) model.fit(x, y, validation_split=0.2, callbacks=[early_stopping]) </code></pre></div> Find out more in the <a href="/api/callbacks/">callbacks documentation</a>. <hr /> <h3 id="how-can-i-freeze-layers-and-do-finetuning">How can I freeze layers and do fine-tuning?</h3> Setting the <code>trainable</code> attribute All layers & models have a <code>layer.trainable</code> boolean attribute: <div class="codehilite"><pre><code>>>> layer = Dense(3) >>> layer.trainable True </code></pre></div> On all layers & models, the <code>trainable</code> attribute can be set (to True or False). When set to <code>False</code>, the <code>layer.trainable_weights</code> attribute is empty: <div class="codehilite"><pre><code>>>> layer = Dense(3) >>> layer.build(input_shape=(None, 3)) # Create the weights of the layer >>> layer.trainable True >>> layer.trainable_weights [<KerasVariable shape=(3, 3), dtype=float32, path=dense/kernel>, <KerasVariable shape=(3,), dtype=float32, path=dense/bias>] >>> layer.trainable = False >>> layer.trainable_weights [] </code></pre></div> Setting the <code>trainable</code> attribute on a layer recursively sets it on all children layers (contents of <code>self.layers</code>). 1) When training with <code>fit()</code>: To do fine-tuning with <code>fit()</code>, you would: <ul> <li>Instantiate a base model and load pre-trained weights</li> <li>Freeze that base model</li> <li>Add trainable layers on top</li> <li>Call <code>compile()</code> and <code>fit()</code></li> </ul> Like this: <div class="codehilite"><pre><code>model = Sequential([ ResNet50Base(input_shape=(32, 32, 3), weights='pretrained'), Dense(10), ]) model.layers[0].trainable = False # Freeze ResNet50Base. assert model.layers[0].trainable_weights == [] # ResNet50Base has no trainable weights. assert len(model.trainable_weights) == 2 # Just the bias & kernel of the Dense layer. model.compile(...) model.fit(...) # Train Dense while excluding ResNet50Base. </code></pre></div> You can follow a similar workflow with the Functional API or the model subclassing API. Make sure to call <code>compile()</code> after changing the value of <code>trainable</code> in order for your changes to be taken into account. Calling <code>compile()</code> will freeze the state of the training step of the model. 2) When using a custom training loop: When writing a training loop, make sure to only update weights that are part of <code>model.trainable_weights</code> (and not all <code>model.weights</code>). Here's a simple TensorFlow example: <div class="codehilite"><pre><code>model = Sequential([ ResNet50Base(input_shape=(32, 32, 3), weights='pretrained'), Dense(10), ]) model.layers[0].trainable = False # Freeze ResNet50Base. # Iterate over the batches of a dataset. for inputs, targets in dataset: # Open a GradientTape. with tf.GradientTape() as tape: # Forward pass. predictions = model(inputs) # Compute the loss value for this batch. loss_value = loss_fn(targets, predictions) # Get gradients of loss wrt the *trainable* weights. gradients = tape.gradient(loss_value, model.trainable_weights) # Update the weights of the model. optimizer.apply_gradients(zip(gradients, model.trainable_weights)) </code></pre></div> Interaction between <code>trainable</code> and <code>compile()</code> Calling <code>compile()</code> on a model is meant to "freeze" the behavior of that model. This implies that the <code>trainable</code> attribute values at the time the model is compiled should be preserved throughout the lifetime of that model, until <code>compile</code> is called again. Hence, if you change <code>trainable</code>, make sure to call <code>compile()</code> again on your model for your changes to be taken into account. For instance, if two models A & B share some layers, and: <ul> <li>Model A gets compiled</li> <li>The <code>trainable</code> attribute value on the shared layers is changed</li> <li>Model B is compiled</li> </ul> Then model A and B are using different <code>trainable</code> values for the shared layers. This mechanism is critical for most existing GAN implementations, which do: <div class="codehilite"><pre><code>discriminator.compile(...) # the weights of `discriminator` should be updated when `discriminator` is trained discriminator.trainable = False gan.compile(...) # `discriminator` is a submodel of `gan`, which should not be updated when `gan` is trained </code></pre></div> <hr /> <h3 id="whats-the-difference-between-the-training-argument-in-call-and-the-trainable-attribute">What's the difference between the <code>training</code> argument in <code>call()</code> and the <code>trainable</code> attribute?</h3> <code>training</code> is a boolean argument in <code>call</code> that determines whether the call should be run in inference mode or training mode. For example, in training mode, a <code>Dropout</code> layer applies random dropout and rescales the output. In inference mode, the same layer does nothing. Example: <div class="codehilite"><pre><code>y = Dropout(0.5)(x, training=True) # Applies dropout at training time *and* inference time </code></pre></div> <code>trainable</code> is a boolean layer attribute that determines the trainable weights of the layer should be updated to minimize the loss during training. If <code>layer.trainable</code> is set to <code>False</code>, then <code>layer.trainable_weights</code> will always be an empty list. Example: <div class="codehilite"><pre><code>model = Sequential([ ResNet50Base(input_shape=(32, 32, 3), weights='pretrained'), Dense(10), ]) model.layers[0].trainable = False # Freeze ResNet50Base. assert model.layers[0].trainable_weights == [] # ResNet50Base has no trainable weights. assert len(model.trainable_weights) == 2 # Just the bias & kernel of the Dense layer. model.compile(...) model.fit(...) # Train Dense while excluding ResNet50Base. </code></pre></div> As you can see, "inference mode vs training mode" and "layer weight trainability" are two very different concepts. You could imagine the following: a dropout layer where the scaling factor is learned during training, via backpropagation. Let's name it <code>AutoScaleDropout</code>. This layer would have simultaneously a trainable state, and a different behavior in inference and training. Because the <code>trainable</code> attribute and the <code>training</code> call argument are independent, you can do the following: <div class="codehilite"><pre><code>layer = AutoScaleDropout(0.5) # Applies dropout at training time *and* inference time # *and* learns the scaling factor during training y = layer(x, training=True) assert len(layer.trainable_weights) == 1 </code></pre></div> <div class="codehilite"><pre><code># Applies dropout at training time *and* inference time # with a *frozen* scaling factor layer = AutoScaleDropout(0.5) layer.trainable = False y = layer(x, training=True) </code></pre></div> Special case of the <code>BatchNormalization</code> layer For a <code>BatchNormalization</code> layer, setting <code>bn.trainable = False</code> will also make its <code>training</code> call argument default to <code>False</code>, meaning that the layer will no update its state during training. This behavior only applies for <code>BatchNormalization</code>. For every other layer, weight trainability and "inference vs training mode" remain independent. <hr /> <h3 id="in-fit-how-is-the-validation-split-computed">In <code>fit()</code>, how is the validation split computed?</h3> If you set the <code>validation_split</code> argument in <code>model.fit</code> to e.g. 0.1, then the validation data used will be the last 10% of the data. If you set it to 0.25, it will be the last 25% of the data, etc. Note that the data isn't shuffled before extracting the validation split, so the validation is literally just the last x% of samples in the input you passed. The same validation set is used for all epochs (within the same call to <code>fit</code>). Note that the <code>validation_split</code> option is only available if your data is passed as Numpy arrays (not <a href="https://www.tensorflow.org/api_docs/python/tf/data/Datasets"><code>tf.data.Datasets</code></a>, which are not indexable). <hr /> <h3 id="in-fit-is-the-data-shuffled-during-training">In <code>fit()</code>, is the data shuffled during training?</h3> If you pass your data as NumPy arrays and if the <code>shuffle</code> argument in <code>model.fit()</code> is set to <code>True</code> (which is the default), the training data will be globally randomly shuffled at each epoch. If you pass your data as a <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset"><code>tf.data.Dataset</code></a> object and if the <code>shuffle</code> argument in <code>model.fit()</code> is set to <code>True</code>, the dataset will be locally shuffled (buffered shuffling). When using <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset"><code>tf.data.Dataset</code></a> objects, prefer shuffling your data beforehand (e.g. by calling <code>dataset = dataset.shuffle(buffer_size)</code>) so as to be in control of the buffer size. Validation data is never shuffled. <hr /> <h3 id="whats-the-recommended-way-to-monitor-my-metrics-when-training-with-fit">What's the recommended way to monitor my metrics when training with <code>fit()</code>?</h3> Loss values and metric values are reported via the default progress bar displayed by calls to <code>fit()</code>. However, staring at changing ascii numbers in a console is not an optimal metric-monitoring experience. We recommend the use of <a href="https://www.tensorflow.org/tensorboard">TensorBoard</a>, which will display nice-looking graphs of your training and validation metrics, regularly updated during training, which you can access from your browser. You can use TensorBoard with <code>fit()</code> via the <a href="/api/callbacks/tensorboard/"><code>TensorBoard</code> callback</a>. <hr /> <h3 id="what-if-i-need-to-customize-what-fit-does">What if I need to customize what <code>fit()</code> does?</h3> You have two options: 1) Subclass the <code>Model</code> class and override the <code>train_step</code> (and <code>test_step</code>) methods This is a better option if you want to use custom update rules but still want to leverage the functionality provided by <code>fit()</code>, such as callbacks, efficient step fusing, etc. Note that this pattern does not prevent you from building models with the Functional API, in which case you will use the class you created to instantiate the model with the <code>inputs</code> and <code>outputs</code>. Same goes for Sequential models, in which case you will subclass <a href="/api/models/sequential#sequential-class"><code>keras.Sequential</code></a> and override its <code>train_step</code> instead of <a href="/api/models/model#model-class"><code>keras.Model</code></a>. See the following guides: <ul> <li><a href="/guides/custom_train_step_in_jax/">Writing a custom train step in JAX</a></li> <li><a href="/guides/custom_train_step_in_tensorflow/">Writing a custom train step in TensorFlow</a></li> <li><a href="/guides/custom_train_step_in_torch/">Writing a custom train step in PyTorch</a></li> </ul> 2) Write a low-level custom training loop This is a good option if you want to be in control of every last little detail – though it can be somewhat verbose. See the following guides: <ul> <li><a href="/guides/writing_a_custom_training_loop_in_jax/">Writing a custom training loop in JAX</a></li> <li><a href="/guides/writing_a_custom_training_loop_in_tensorflow/">Writing a custom training loop in TensorFlow</a></li> <li><a href="/guides/writing_a_custom_training_loop_in_torch/">Writing a custom training loop in PyTorch</a></li> </ul> <hr /> <h3 id="whats-the-difference-between-model-methods-predict-and-call">What's the difference between <code>Model</code> methods <code>predict()</code> and <code>__call__()</code>?</h3> Let's answer with an extract from <a href="https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras">Deep Learning with Python, Second Edition</a>: <blockquote> Both <code>y = model.predict(x)</code> and <code>y = model(x)</code> (where <code>x</code> is an array of input data) mean "run the model on <code>x</code> and retrieve the output <code>y</code>." Yet they aren't exactly the same thing. <code>predict()</code> loops over the data in batches (in fact, you can specify the batch size via <code>predict(x, batch_size=64)</code>), and it extracts the NumPy value of the outputs. It's schematically equivalent to this: </blockquote> <div class="codehilite"><pre><code>def predict(x): y_batches = [] for x_batch in get_batches(x): y_batch = model(x_batch).numpy() y_batches.append(y_batch) return np.concatenate(y_batches) </code></pre></div> <blockquote> This means that <code>predict()</code> calls can scale to very large arrays. Meanwhile, <code>model(x)</code> happens in-memory and doesn't scale. On the other hand, <code>predict()</code> is not differentiable: you cannot retrieve its gradient if you call it in a <code>GradientTape</code> scope. You should use <code>model(x)</code> when you need to retrieve the gradients of the model call, and you should use <code>predict()</code> if you just need the output value. In other words, always use <code>predict()</code> unless you're in the middle of writing a low-level gradient descent loop (as we are now). </blockquote> <hr /> <h2 id="modelingrelated-questions">Modeling-related questions</h2> <h3 id="how-can-i-obtain-the-output-of-an-intermediate-layer-feature-extraction">How can I obtain the output of an intermediate layer (feature extraction)?</h3> In the Functional API and Sequential API, if a layer has been called exactly once, you can retrieve its output via <code>layer.output</code> and its input via <code>layer.input</code>. This enables you do quickly instantiate feature-extraction models, like this one: <div class="codehilite"><pre><code>import keras from keras import layers model = Sequential([ layers.Conv2D(32, 3, activation='relu'), layers.Conv2D(32, 3, activation='relu'), layers.MaxPooling2D(2), layers.Conv2D(32, 3, activation='relu'), layers.Conv2D(32, 3, activation='relu'), layers.GlobalMaxPooling2D(), layers.Dense(10), ]) extractor = keras.Model(inputs=model.inputs, outputs=[layer.output for layer in model.layers]) features = extractor(data) </code></pre></div> Naturally, this is not possible with models that are subclasses of <code>Model</code> that override <code>call</code>. Here's another example: instantiating a <code>Model</code> that returns the output of a specific named layer: <div class="codehilite"><pre><code>model = ... # create the original model layer_name = 'my_layer' intermediate_layer_model = keras.Model(inputs=model.input, outputs=model.get_layer(layer_name).output) intermediate_output = intermediate_layer_model(data) </code></pre></div> <hr /> <h3 id="how-can-i-use-pretrained-models-in-keras">How can I use pre-trained models in Keras?</h3> You could leverage the <a href="/api/applications/">models available in <code>keras.applications</code></a>, or the models available in <a href="/keras_cv/">KerasCV</a> and <a href="/keras_hub/">KerasHub</a>. <hr /> <h3 id="how-can-i-use-stateful-rnns">How can I use stateful RNNs?</h3> Making a RNN stateful means that the states for the samples of each batch will be reused as initial states for the samples in the next batch. When using stateful RNNs, it is therefore assumed that: <ul> <li>all batches have the same number of samples</li> <li>If <code>x1</code> and <code>x2</code> are successive batches of samples, then <code>x2[i]</code> is the follow-up sequence to <code>x1[i]</code>, for every <code>i</code>.</li> </ul> To use statefulness in RNNs, you need to: <ul> <li>explicitly specify the batch size you are using, by passing a <code>batch_size</code> argument to the first layer in your model. E.g. <code>batch_size=32</code> for a 32-samples batch of sequences of 10 timesteps with 16 features per timestep.</li> <li>set <code>stateful=True</code> in your RNN layer(s).</li> <li>specify <code>shuffle=False</code> when calling <code>fit()</code>.</li> </ul> To reset the states accumulated: <ul> <li>use <code>model.reset_states()</code> to reset the states of all layers in the model</li> <li>use <code>layer.reset_states()</code> to reset the states of a specific stateful RNN layer</li> </ul> Example: <div class="codehilite"><pre><code>import keras from keras import layers import numpy as np x = np.random.random((32, 21, 16)) # this is our input data, of shape (32, 21, 16) # we will feed it to our model in sequences of length 10 model = keras.Sequential() model.add(layers.LSTM(32, input_shape=(10, 16), batch_size=32, stateful=True)) model.add(layers.Dense(16, activation='softmax')) model.compile(optimizer='rmsprop', loss='categorical_crossentropy') # we train the network to predict the 11th timestep given the first 10: model.train_on_batch(x[:, :10, :], np.reshape(x[:, 10, :], (32, 16))) # the state of the network has changed. We can feed the follow-up sequences: model.train_on_batch(x[:, 10:20, :], np.reshape(x[:, 20, :], (32, 16))) # let's reset the states of the LSTM layer: model.reset_states() # another way to do it in this case: model.layers[0].reset_states() </code></pre></div> Note that the methods <code>predict</code>, <code>fit</code>, <code>train_on_batch</code>, etc. will all update the states of the stateful layers in a model. This allows you to do not only stateful training, but also stateful prediction. <hr /> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>