Tailor the search space

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/guides/keras_tuner/tailor_the_search_space/" />  <meta property="og:title" content="Keras documentation: Tailor the search space"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Tailor the search space"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Tailor the search space</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link active" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-sublink" href="/guides/functional_api/">The Functional API</a> <a class="nav-sublink" href="/guides/sequential_model/">The Sequential model</a> <a class="nav-sublink" href="/guides/making_new_layers_and_models_via_subclassing/">Making new layers & models via subclassing</a> <a class="nav-sublink" href="/guides/training_with_built_in_methods/">Training & evaluation with the built-in methods</a> <a class="nav-sublink" href="/guides/custom_train_step_in_jax/">Customizing `fit()` with JAX</a> <a class="nav-sublink" href="/guides/custom_train_step_in_tensorflow/">Customizing `fit()` with TensorFlow</a> <a class="nav-sublink" href="/guides/custom_train_step_in_torch/">Customizing `fit()` with PyTorch</a> <a class="nav-sublink" href="/guides/writing_a_custom_training_loop_in_jax/">Writing a custom training loop in JAX</a> <a class="nav-sublink" href="/guides/writing_a_custom_training_loop_in_tensorflow/">Writing a custom training loop in TensorFlow</a> <a class="nav-sublink" href="/guides/writing_a_custom_training_loop_in_torch/">Writing a custom training loop in PyTorch</a> <a class="nav-sublink" href="/guides/serialization_and_saving/">Serialization & saving</a> <a class="nav-sublink" href="/guides/customizing_saving_and_serialization/">Customizing saving & serialization</a> <a class="nav-sublink" href="/guides/writing_your_own_callbacks/">Writing your own callbacks</a> <a class="nav-sublink" href="/guides/transfer_learning/">Transfer learning & fine-tuning</a> <a class="nav-sublink" href="/guides/distributed_training_with_jax/">Distributed training with JAX</a> <a class="nav-sublink" href="/guides/distributed_training_with_tensorflow/">Distributed training with TensorFlow</a> <a class="nav-sublink" href="/guides/distributed_training_with_torch/">Distributed training with PyTorch</a> <a class="nav-sublink" href="/guides/distribution/">Distributed training with Keras 3</a> <a class="nav-sublink" href="/guides/migrating_to_keras_3/">Migrating Keras 2 code to Keras 3</a> <a class="nav-sublink active" href="/guides/keras_tuner/">Hyperparameter Tuning</a> <a class="nav-sublink2" href="/guides/keras_tuner/getting_started/">Getting started with KerasTuner</a> <a class="nav-sublink2" href="/guides/keras_tuner/distributed_tuning/">Distributed hyperparameter tuning with KerasTuner</a> <a class="nav-sublink2" href="/guides/keras_tuner/custom_tuner/">Tune hyperparameters in your custom training loop</a> <a class="nav-sublink2" href="/guides/keras_tuner/visualize_tuning/">Visualize the hyperparameter tuning process</a> <a class="nav-sublink2" href="/guides/keras_tuner/failed_trials/">Handling failed trials in KerasTuner</a> <a class="nav-sublink2 active" href="/guides/keras_tuner/tailor_the_search_space/">Tailor the search space</a> <a class="nav-sublink" href="/guides/keras_cv/">KerasCV</a> <a class="nav-sublink" href="/guides/keras_nlp/">KerasNLP</a> <a class="nav-sublink" href="/guides/keras_hub/">KerasHub</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparameter Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> <a class="nav-link" href="/keras_cv/" role="tab" aria-selected="">KerasCV: Computer Vision Workflows</a> <a class="nav-link" href="/keras_nlp/" role="tab" aria-selected="">KerasNLP: Natural Language Workflows</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/guides/'>Developer guides</a> / <a href='/guides/keras_tuner/'>Hyperparameter Tuning</a> / Tailor the search space </div> <div class='k-content'> <h1 id="tailor-the-search-space">Tailor the search space</h1> Authors: Luca Invernizzi, James Long, Francois Chollet, Tom O'Malley, Haifeng Jin Date created: 2019/05/31 Last modified: 2021/10/27 Description: Tune a subset of the hyperparameters without changing the hypermodel. <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/guides/ipynb/keras_tuner/tailor_the_search_space.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/guides/keras_tuner/tailor_the_search_space.py">GitHub source</a> <div class="codehilite"><pre><code>!pip install keras-tuner -q </code></pre></div> In this guide, we will show how to tailor the search space without changing the <code>HyperModel</code> code directly. For example, you can only tune some of the hyperparameters and keep the rest fixed, or you can override the compile arguments, like <code>optimizer</code>, <code>loss</code>, and <code>metrics</code>. <hr /> <h2 id="the-default-value-of-a-hyperparameter">The default value of a hyperparameter</h2> Before we tailor the search space, it is important to know that every hyperparameter has a default value. This default value is used as the hyperparameter value when not tuning it during our tailoring the search space. Whenever you register a hyperparameter, you can use the <code>default</code> argument to specify a default value: <div class="codehilite"><pre><code>hp.Int("units", min_value=32, max_value=128, step=32, default=64) </code></pre></div> If you don't, hyperparameters always have a default default (for <code>Int</code>, it is equal to <code>min_value</code>). In the following model-building function, we specified the default value for the <code>units</code> hyperparameter as 64. <div class="codehilite"><pre><code>import keras from keras import layers import keras_tuner import numpy as np def build_model(hp): model = keras.Sequential() model.add(layers.Flatten()) model.add( layers.Dense( units=hp.Int("units", min_value=32, max_value=128, step=32, default=64) ) ) if hp.Boolean("dropout"): model.add(layers.Dropout(rate=0.25)) model.add(layers.Dense(units=10, activation="softmax")) model.compile( optimizer=keras.optimizers.Adam( learning_rate=hp.Choice("learning_rate", values=[1e-2, 1e-3, 1e-4]) ), loss="sparse_categorical_crossentropy", metrics=["accuracy"], ) return model </code></pre></div> We will reuse this search space in the rest of the tutorial by overriding the hyperparameters without defining a new search space. <hr /> <h2 id="search-a-few-and-fix-the-rest">Search a few and fix the rest</h2> If you have an existing hypermodel, and you want to search over only a few hyperparameters, and keep the rest fixed, you don't have to change the code in the model-building function or the <code>HyperModel</code>. You can pass a <code>HyperParameters</code> to the <code>hyperparameters</code> argument to the tuner constructor with all the hyperparameters you want to tune. Specify <code>tune_new_entries=False</code> to prevent it from tuning other hyperparameters, the default value of which would be used. In the following example, we only tune the <code>learning_rate</code> hyperparameter, and changed its type and value ranges. <div class="codehilite"><pre><code>hp = keras_tuner.HyperParameters() # This will override the `learning_rate` parameter with your # own selection of choices hp.Float("learning_rate", min_value=1e-4, max_value=1e-2, sampling="log") tuner = keras_tuner.RandomSearch( hypermodel=build_model, hyperparameters=hp, # Prevents unlisted parameters from being tuned tune_new_entries=False, objective="val_accuracy", max_trials=3, overwrite=True, directory="my_dir", project_name="search_a_few", ) # Generate random data x_train = np.random.rand(100, 28, 28, 1) y_train = np.random.randint(0, 10, (100, 1)) x_val = np.random.rand(20, 28, 28, 1) y_val = np.random.randint(0, 10, (20, 1)) # Run the search tuner.search(x_train, y_train, epochs=1, validation_data=(x_val, y_val)) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Trial 3 Complete [00h 00m 01s] val_accuracy: 0.20000000298023224 </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Best val_accuracy So Far: 0.25 Total elapsed time: 00h 00m 03s </code></pre></div> </div> If you summarize the search space, you will see only one hyperparameter. <div class="codehilite"><pre><code>tuner.search_space_summary() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Search space summary Default search space size: 1 learning_rate (Float) {'default': 0.0001, 'conditions': [], 'min_value': 0.0001, 'max_value': 0.01, 'step': None, 'sampling': 'log'} </code></pre></div> </div> <hr /> <h2 id="fix-a-few-and-tune-the-rest">Fix a few and tune the rest</h2> In the example above we showed how to tune only a few hyperparameters and keep the rest fixed. You can also do the reverse: only fix a few hyperparameters and tune all the rest. In the following example, we fixed the value of the <code>learning_rate</code> hyperparameter. Pass a <code>hyperparameters</code> argument with a <code>Fixed</code> entry (or any number of <code>Fixed</code> entries). Also remember to specify <code>tune_new_entries=True</code>, which allows us to tune the rest of the hyperparameters. <div class="codehilite"><pre><code>hp = keras_tuner.HyperParameters() hp.Fixed("learning_rate", value=1e-4) tuner = keras_tuner.RandomSearch( build_model, hyperparameters=hp, tune_new_entries=True, objective="val_accuracy", max_trials=3, overwrite=True, directory="my_dir", project_name="fix_a_few", ) tuner.search(x_train, y_train, epochs=1, validation_data=(x_val, y_val)) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Trial 3 Complete [00h 00m 01s] val_accuracy: 0.15000000596046448 </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Best val_accuracy So Far: 0.15000000596046448 Total elapsed time: 00h 00m 03s </code></pre></div> </div> If you summarize the search space, you will see the <code>learning_rate</code> is marked as fixed, and the rest of the hyperparameters are being tuned. <div class="codehilite"><pre><code>tuner.search_space_summary() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Search space summary Default search space size: 3 learning_rate (Fixed) {'conditions': [], 'value': 0.0001} units (Int) {'default': 64, 'conditions': [], 'min_value': 32, 'max_value': 128, 'step': 32, 'sampling': 'linear'} dropout (Boolean) {'default': False, 'conditions': []} </code></pre></div> </div> <hr /> <h2 id="overriding-compilation-arguments">Overriding compilation arguments</h2> If you have a hypermodel for which you want to change the existing optimizer, loss, or metrics, you can do so by passing these arguments to the tuner constructor: <div class="codehilite"><pre><code>tuner = keras_tuner.RandomSearch( build_model, optimizer=keras.optimizers.Adam(1e-3), loss="mse", metrics=[ "sparse_categorical_crossentropy", ], objective="val_loss", max_trials=3, overwrite=True, directory="my_dir", project_name="override_compile", ) tuner.search(x_train, y_train, epochs=1, validation_data=(x_val, y_val)) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Trial 3 Complete [00h 00m 01s] val_loss: 29.39796257019043 </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Best val_loss So Far: 29.39630699157715 Total elapsed time: 00h 00m 04s </code></pre></div> </div> If you get the best model, you can see the loss function has changed to MSE. <div class="codehilite"><pre><code>tuner.get_best_models()[0].loss </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>/usr/local/python/3.10.13/lib/python3.10/site-packages/keras/src/saving/saving_lib.py:388: UserWarning: Skipping variable loading for optimizer 'adam', because it has 2 variables whereas the saved optimizer has 10 variables. trackable.load_own_variables(weights_store.get(inner_path)) 'mse' </code></pre></div> </div> <hr /> <h2 id="tailor-the-search-space-of-prebuild-hypermodels">Tailor the search space of pre-build HyperModels</h2> You can also use these techniques with the pre-build models in KerasTuner, like <code>HyperResNet</code> or <code>HyperXception</code>. However, to see what hyperparameters are in these pre-build <code>HyperModel</code>s, you will have to read the source code. In the following example, we only tune the <code>learning_rate</code> of <code>HyperXception</code> and fixed all the rest of the hyperparameters. Because the default loss of <code>HyperXception</code> is <code>categorical_crossentropy</code>, which expect the labels to be one-hot encoded, which doesn't match our raw integer label data, we need to change it by overriding the <code>loss</code> in the compile args to <code>sparse_categorical_crossentropy</code>. <div class="codehilite"><pre><code>hypermodel = keras_tuner.applications.HyperXception(input_shape=(28, 28, 1), classes=10) hp = keras_tuner.HyperParameters() # This will override the `learning_rate` parameter with your # own selection of choices hp.Choice("learning_rate", values=[1e-2, 1e-3, 1e-4]) tuner = keras_tuner.RandomSearch( hypermodel, hyperparameters=hp, # Prevents unlisted parameters from being tuned tune_new_entries=False, # Override the loss. loss="sparse_categorical_crossentropy", metrics=["accuracy"], objective="val_accuracy", max_trials=3, overwrite=True, directory="my_dir", project_name="helloworld", ) # Run the search tuner.search(x_train, y_train, epochs=1, validation_data=(x_val, y_val)) tuner.search_space_summary() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Trial 3 Complete [00h 00m 19s] val_accuracy: 0.15000000596046448 </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Best val_accuracy So Far: 0.20000000298023224 Total elapsed time: 00h 00m 58s Search space summary Default search space size: 1 learning_rate (Choice) {'default': 0.01, 'conditions': [], 'values': [0.01, 0.001, 0.0001], 'ordered': True} </code></pre></div> </div> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#tailor-the-search-space'>Tailor the search space</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#the-default-value-of-a-hyperparameter'>The default value of a hyperparameter</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#search-a-few-and-fix-the-rest'>Search a few and fix the rest</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#fix-a-few-and-tune-the-rest'>Fix a few and tune the rest</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#overriding-compilation-arguments'>Overriding compilation arguments</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#tailor-the-search-space-of-prebuild-hypermodels'>Tailor the search space of pre-build HyperModels</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>

CINXE.COM

Tailor the search space