Visualize the hyperparameter tuning process

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/guides/keras_tuner/visualize_tuning/" />  <meta property="og:title" content="Keras documentation: Visualize the hyperparameter tuning process"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Visualize the hyperparameter tuning process"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Visualize the hyperparameter tuning process</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link active" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-sublink" href="/guides/functional_api/">The Functional API</a> <a class="nav-sublink" href="/guides/sequential_model/">The Sequential model</a> <a class="nav-sublink" href="/guides/making_new_layers_and_models_via_subclassing/">Making new layers & models via subclassing</a> <a class="nav-sublink" href="/guides/training_with_built_in_methods/">Training & evaluation with the built-in methods</a> <a class="nav-sublink" href="/guides/custom_train_step_in_jax/">Customizing `fit()` with JAX</a> <a class="nav-sublink" href="/guides/custom_train_step_in_tensorflow/">Customizing `fit()` with TensorFlow</a> <a class="nav-sublink" href="/guides/custom_train_step_in_torch/">Customizing `fit()` with PyTorch</a> <a class="nav-sublink" href="/guides/writing_a_custom_training_loop_in_jax/">Writing a custom training loop in JAX</a> <a class="nav-sublink" href="/guides/writing_a_custom_training_loop_in_tensorflow/">Writing a custom training loop in TensorFlow</a> <a class="nav-sublink" href="/guides/writing_a_custom_training_loop_in_torch/">Writing a custom training loop in PyTorch</a> <a class="nav-sublink" href="/guides/serialization_and_saving/">Serialization & saving</a> <a class="nav-sublink" href="/guides/customizing_saving_and_serialization/">Customizing saving & serialization</a> <a class="nav-sublink" href="/guides/writing_your_own_callbacks/">Writing your own callbacks</a> <a class="nav-sublink" href="/guides/transfer_learning/">Transfer learning & fine-tuning</a> <a class="nav-sublink" href="/guides/distributed_training_with_jax/">Distributed training with JAX</a> <a class="nav-sublink" href="/guides/distributed_training_with_tensorflow/">Distributed training with TensorFlow</a> <a class="nav-sublink" href="/guides/distributed_training_with_torch/">Distributed training with PyTorch</a> <a class="nav-sublink" href="/guides/distribution/">Distributed training with Keras 3</a> <a class="nav-sublink" href="/guides/migrating_to_keras_3/">Migrating Keras 2 code to Keras 3</a> <a class="nav-sublink active" href="/guides/keras_tuner/">Hyperparameter Tuning</a> <a class="nav-sublink2" href="/guides/keras_tuner/getting_started/">Getting started with KerasTuner</a> <a class="nav-sublink2" href="/guides/keras_tuner/distributed_tuning/">Distributed hyperparameter tuning with KerasTuner</a> <a class="nav-sublink2" href="/guides/keras_tuner/custom_tuner/">Tune hyperparameters in your custom training loop</a> <a class="nav-sublink2 active" href="/guides/keras_tuner/visualize_tuning/">Visualize the hyperparameter tuning process</a> <a class="nav-sublink2" href="/guides/keras_tuner/failed_trials/">Handling failed trials in KerasTuner</a> <a class="nav-sublink2" href="/guides/keras_tuner/tailor_the_search_space/">Tailor the search space</a> <a class="nav-sublink" href="/guides/keras_cv/">KerasCV</a> <a class="nav-sublink" href="/guides/keras_nlp/">KerasNLP</a> <a class="nav-sublink" href="/guides/keras_hub/">KerasHub</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparameter Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> <a class="nav-link" href="/keras_cv/" role="tab" aria-selected="">KerasCV: Computer Vision Workflows</a> <a class="nav-link" href="/keras_nlp/" role="tab" aria-selected="">KerasNLP: Natural Language Workflows</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/guides/'>Developer guides</a> / <a href='/guides/keras_tuner/'>Hyperparameter Tuning</a> / Visualize the hyperparameter tuning process </div> <div class='k-content'> <h1 id="visualize-the-hyperparameter-tuning-process">Visualize the hyperparameter tuning process</h1> Author: Haifeng Jin Date created: 2021/06/25 Last modified: 2021/06/05 Description: Using TensorBoard to visualize the hyperparameter tuning process in KerasTuner. <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/guides/ipynb/keras_tuner/visualize_tuning.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/guides/keras_tuner/visualize_tuning.py">GitHub source</a> <div class="codehilite"><pre><code>!pip install keras-tuner -q </code></pre></div> <hr /> <h2 id="introduction">Introduction</h2> KerasTuner prints the logs to screen including the values of the hyperparameters in each trial for the user to monitor the progress. However, reading the logs is not intuitive enough to sense the influences of hyperparameters have on the results, Therefore, we provide a method to visualize the hyperparameter values and the corresponding evaluation results with interactive figures using TensorBoard. <a href="https://www.tensorflow.org/tensorboard">TensorBoard</a> is a useful tool for visualizing the machine learning experiments. It can monitor the losses and metrics during the model training and visualize the model architectures. Running KerasTuner with TensorBoard will give you additional features for visualizing hyperparameter tuning results using its HParams plugin. We will use a simple example of tuning a model for the MNIST image classification dataset to show how to use KerasTuner with TensorBoard. The first step is to download and format the data. <div class="codehilite"><pre><code>import numpy as np import keras_tuner import keras from keras import layers (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data() # Normalize the pixel values to the range of [0, 1]. x_train = x_train.astype("float32") / 255 x_test = x_test.astype("float32") / 255 # Add the channel dimension to the images. x_train = np.expand_dims(x_train, -1) x_test = np.expand_dims(x_test, -1) # Print the shapes of the data. print(x_train.shape) print(y_train.shape) print(x_test.shape) print(y_test.shape) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>(60000, 28, 28, 1) (60000,) (10000, 28, 28, 1) (10000,) </code></pre></div> </div> Then, we write a <code>build_model</code> function to build the model with hyperparameters and return the model. The hyperparameters include the type of model to use (multi-layer perceptron or convolutional neural network), the number of layers, the number of units or filters, whether to use dropout. <div class="codehilite"><pre><code>def build_model(hp): inputs = keras.Input(shape=(28, 28, 1)) # Model type can be MLP or CNN. model_type = hp.Choice("model_type", ["mlp", "cnn"]) x = inputs if model_type == "mlp": x = layers.Flatten()(x) # Number of layers of the MLP is a hyperparameter. for i in range(hp.Int("mlp_layers", 1, 3)): # Number of units of each layer are # different hyperparameters with different names. x = layers.Dense( units=hp.Int(f"units_{i}", 32, 128, step=32), activation="relu", )(x) else: # Number of layers of the CNN is also a hyperparameter. for i in range(hp.Int("cnn_layers", 1, 3)): x = layers.Conv2D( hp.Int(f"filters_{i}", 32, 128, step=32), kernel_size=(3, 3), activation="relu", )(x) x = layers.MaxPooling2D(pool_size=(2, 2))(x) x = layers.Flatten()(x) # A hyperparamter for whether to use dropout layer. if hp.Boolean("dropout"): x = layers.Dropout(0.5)(x) # The last layer contains 10 units, # which is the same as the number of classes. outputs = layers.Dense(units=10, activation="softmax")(x) model = keras.Model(inputs=inputs, outputs=outputs) # Compile the model. model.compile( loss="sparse_categorical_crossentropy", metrics=["accuracy"], optimizer="adam", ) return model </code></pre></div> We can do a quick test of the models to check if it build successfully for both CNN and MLP. <div class="codehilite"><pre><code># Initialize the `HyperParameters` and set the values. hp = keras_tuner.HyperParameters() hp.values["model_type"] = "cnn" # Build the model using the `HyperParameters`. model = build_model(hp) # Test if the model runs with our data. model(x_train[:100]) # Print a summary of the model. model.summary() # Do the same for MLP model. hp.values["model_type"] = "mlp" model = build_model(hp) model(x_train[:100]) model.summary() </code></pre></div> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">Model: "functional_1" </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ input_layer (InputLayer) │ (None, 28, 28, 1) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ conv2d (Conv2D) │ (None, 26, 26, 32) │ 320 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ max_pooling2d (MaxPooling2D) │ (None, 13, 13, 32) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ flatten (Flatten) │ (None, 5408) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ dense (Dense) │ (None, 10) │ 54,090 │ └─────────────────────────────────┴───────────────────────────┴────────────┘ </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Total params: 54,410 (212.54 KB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Trainable params: 54,410 (212.54 KB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Non-trainable params: 0 (0.00 B) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">Model: "functional_3" </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ input_layer_1 (InputLayer) │ (None, 28, 28, 1) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ flatten_1 (Flatten) │ (None, 784) │ 0 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ dense_1 (Dense) │ (None, 32) │ 25,120 │ ├─────────────────────────────────┼───────────────────────────┼────────────┤ │ dense_2 (Dense) │ (None, 10) │ 330 │ └─────────────────────────────────┴───────────────────────────┴────────────┘ </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Total params: 25,450 (99.41 KB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Trainable params: 25,450 (99.41 KB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Non-trainable params: 0 (0.00 B) </pre> Initialize the <code>RandomSearch</code> tuner with 10 trials and using validation accuracy as the metric for selecting models. <div class="codehilite"><pre><code>tuner = keras_tuner.RandomSearch( build_model, max_trials=10, # Do not resume the previous search in the same directory. overwrite=True, objective="val_accuracy", # Set a directory to store the intermediate results. directory="/tmp/tb", ) </code></pre></div> Start the search by calling <code>tuner.search(...)</code>. To use TensorBoard, we need to pass a <a href="/api/callbacks/tensorboard#tensorboard-class"><code>keras.callbacks.TensorBoard</code></a> instance to the callbacks. <div class="codehilite"><pre><code>tuner.search( x_train, y_train, validation_split=0.2, epochs=2, # Use the TensorBoard callback. # The logs will be write to "/tmp/tb_logs". callbacks=[keras.callbacks.TensorBoard("/tmp/tb_logs")], ) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Trial 10 Complete [00h 00m 06s] val_accuracy: 0.9617499709129333 </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Best val_accuracy So Far: 0.9837499856948853 Total elapsed time: 00h 08m 32s </code></pre></div> </div> If running in Colab, the following two commands will show you the TensorBoard inside Colab. <code>%load_ext tensorboard</code> <code>%tensorboard --logdir /tmp/tb_logs</code> You have access to all the common features of the TensorBoard. For example, you can view the loss and metrics curves and visualize the computational graph of the models in different trials. <img alt="Loss and metrics curves" src="https://i.imgur.com/ShulDtI.png" /> <img alt="Computational graphs" src="https://i.imgur.com/8sRiT1I.png" /> In addition to these features, we also have a HParams tab, in which there are three views. In the table view, you can view the 10 different trials in a table with the different hyperparameter values and evaluation metrics. <img alt="Table view" src="https://i.imgur.com/OMcQdOw.png" /> On the left side, you can specify the filters for certain hyperparameters. For example, you can specify to only view the MLP models without the dropout layer and with 1 to 2 dense layers. <img alt="Filtered table view" src="https://i.imgur.com/yZpfaxN.png" /> Besides the table view, it also provides two other views, parallel coordinates view and scatter plot matrix view. They are just different visualization methods for the same data. You can still use the panel on the left to filter the results. In the parallel coordinates view, each colored line is a trial. The axes are the hyperparameters and evaluation metrics. <img alt="Parallel coordinates view" src="https://i.imgur.com/PJ7HQUQ.png" /> In the scatter plot matrix view, each dot is a trial. The plots are projections of the trials on planes with different hyperparameter and metrics as the axes. <img alt="Scatter plot matrix view" src="https://i.imgur.com/zjPjh6o.png" /> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#visualize-the-hyperparameter-tuning-process'>Visualize the hyperparameter tuning process</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>

CINXE.COM

Visualize the hyperparameter tuning process