Layer weight regularizers

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/api/layers/regularizers/" />  <meta property="og:title" content="Keras documentation: Layer weight regularizers"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Layer weight regularizers"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Layer weight regularizers</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link active" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-sublink" href="/api/models/">Models API</a> <a class="nav-sublink active" href="/api/layers/">Layers API</a> <a class="nav-sublink2" href="/api/layers/base_layer/">The base Layer class</a> <a class="nav-sublink2" href="/api/layers/activations/">Layer activations</a> <a class="nav-sublink2" href="/api/layers/initializers/">Layer weight initializers</a> <a class="nav-sublink2 active" href="/api/layers/regularizers/">Layer weight regularizers</a> <a class="nav-sublink2" href="/api/layers/constraints/">Layer weight constraints</a> <a class="nav-sublink2" href="/api/layers/core_layers/">Core layers</a> <a class="nav-sublink2" href="/api/layers/convolution_layers/">Convolution layers</a> <a class="nav-sublink2" href="/api/layers/pooling_layers/">Pooling layers</a> <a class="nav-sublink2" href="/api/layers/recurrent_layers/">Recurrent layers</a> <a class="nav-sublink2" href="/api/layers/preprocessing_layers/">Preprocessing layers</a> <a class="nav-sublink2" href="/api/layers/normalization_layers/">Normalization layers</a> <a class="nav-sublink2" href="/api/layers/regularization_layers/">Regularization layers</a> <a class="nav-sublink2" href="/api/layers/attention_layers/">Attention layers</a> <a class="nav-sublink2" href="/api/layers/reshaping_layers/">Reshaping layers</a> <a class="nav-sublink2" href="/api/layers/merging_layers/">Merging layers</a> <a class="nav-sublink2" href="/api/layers/activation_layers/">Activation layers</a> <a class="nav-sublink2" href="/api/layers/backend_specific_layers/">Backend-specific layers</a> <a class="nav-sublink" href="/api/callbacks/">Callbacks API</a> <a class="nav-sublink" href="/api/ops/">Ops API</a> <a class="nav-sublink" href="/api/optimizers/">Optimizers</a> <a class="nav-sublink" href="/api/metrics/">Metrics</a> <a class="nav-sublink" href="/api/losses/">Losses</a> <a class="nav-sublink" href="/api/data_loading/">Data loading</a> <a class="nav-sublink" href="/api/datasets/">Built-in small datasets</a> <a class="nav-sublink" href="/api/applications/">Keras Applications</a> <a class="nav-sublink" href="/api/mixed_precision/">Mixed precision</a> <a class="nav-sublink" href="/api/distribution/">Multi-device distribution</a> <a class="nav-sublink" href="/api/random/">RNG API</a> <a class="nav-sublink" href="/api/utils/">Utilities</a> <a class="nav-sublink" href="/api/keras_tuner/">KerasTuner</a> <a class="nav-sublink" href="/api/keras_cv/">KerasCV</a> <a class="nav-sublink" href="/api/keras_nlp/">KerasNLP</a> <a class="nav-sublink" href="/api/keras_hub/">KerasHub</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparameter Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> <a class="nav-link" href="/keras_cv/" role="tab" aria-selected="">KerasCV: Computer Vision Workflows</a> <a class="nav-link" href="/keras_nlp/" role="tab" aria-selected="">KerasNLP: Natural Language Workflows</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/api/'>Keras 3 API documentation</a> / <a href='/api/layers/'>Layers API</a> / Layer weight regularizers </div> <div class='k-content'> <h1 id="layer-weight-regularizers">Layer weight regularizers</h1> Regularizers allow you to apply penalties on layer parameters or layer activity during optimization. These penalties are summed into the loss function that the network optimizes. Regularization penalties are applied on a per-layer basis. The exact API will depend on the layer, but many layers (e.g. <code>Dense</code>, <code>Conv1D</code>, <code>Conv2D</code> and <code>Conv3D</code>) have a unified API. These layers expose 3 keyword arguments: <ul> <li><code>kernel_regularizer</code>: Regularizer to apply a penalty on the layer's kernel</li> <li><code>bias_regularizer</code>: Regularizer to apply a penalty on the layer's bias</li> <li><code>activity_regularizer</code>: Regularizer to apply a penalty on the layer's output</li> </ul> <div class="codehilite"><pre><code>from keras import layers from keras import regularizers layer = layers.Dense( units=64, kernel_regularizer=regularizers.L1L2(l1=1e-5, l2=1e-4), bias_regularizer=regularizers.L2(1e-4), activity_regularizer=regularizers.L2(1e-5) ) </code></pre></div> The value returned by the <code>activity_regularizer</code> object gets divided by the input batch size so that the relative weighting between the weight regularizers and the activity regularizers does not change with the batch size. You can access a layer's regularization penalties by calling <code>layer.losses</code> after calling the layer on inputs: <div class="codehilite"><pre><code>from keras import ops layer = layers.Dense(units=5, kernel_initializer='ones', kernel_regularizer=regularizers.L1(0.01), activity_regularizer=regularizers.L2(0.01)) tensor = ops.ones(shape=(5, 5)) * 2.0 out = layer(tensor) # The kernel regularization term is 0.25 # The activity regularization term (after dividing by the batch size) is 5 print(ops.sum(layer.losses)) # 5.25 (= 5 + 0.25) </code></pre></div> <h2 id="available-regularizers">Available regularizers</h2> The following built-in regularizers are available as part of the <code>keras.regularizers</code> module: <a href="https://github.com/keras-team/keras/tree/v3.6.0/keras/src/regularizers/regularizers.py#L8">[source]</a> <h3 id="regularizer-class"><code>Regularizer</code> class</h3> <div class="codehilite"><pre><code>keras.regularizers.Regularizer() </code></pre></div> Regularizer base class. Regularizers allow you to apply penalties on layer parameters or layer activity during optimization. These penalties are summed into the loss function that the network optimizes. Regularization penalties are applied on a per-layer basis. The exact API will depend on the layer, but many layers (e.g. <code>Dense</code>, <code>Conv1D</code>, <code>Conv2D</code> and <code>Conv3D</code>) have a unified API. These layers expose 3 keyword arguments: <ul> <li><code>kernel_regularizer</code>: Regularizer to apply a penalty on the layer's kernel</li> <li><code>bias_regularizer</code>: Regularizer to apply a penalty on the layer's bias</li> <li><code>activity_regularizer</code>: Regularizer to apply a penalty on the layer's output</li> </ul> All layers (including custom layers) expose <code>activity_regularizer</code> as a settable property, whether or not it is in the constructor arguments. The value returned by the <code>activity_regularizer</code> is divided by the input batch size so that the relative weighting between the weight regularizers and the activity regularizers does not change with the batch size. You can access a layer's regularization penalties by calling <code>layer.losses</code> after calling the layer on inputs. <h2 id="example">Example</h2> <div class="codehilite"><pre><code>>>> layer = Dense( ... 5, input_dim=5, ... kernel_initializer='ones', ... kernel_regularizer=L1(0.01), ... activity_regularizer=L2(0.01)) >>> tensor = ops.ones(shape=(5, 5)) * 2.0 >>> out = layer(tensor) </code></pre></div> <div class="codehilite"><pre><code>>>> # The kernel regularization term is 0.25 >>> # The activity regularization term (after dividing by the batch size) >>> # is 5 >>> ops.sum(layer.losses) 5.25 </code></pre></div> <h2 id="available-penalties">Available penalties</h2> <div class="codehilite"><pre><code>L1(0.3) # L1 Regularization Penalty L2(0.1) # L2 Regularization Penalty L1L2(l1=0.01, l2=0.01) # L1 + L2 penalties </code></pre></div> <h2 id="directly-calling-a-regularizer">Directly calling a regularizer</h2> Compute a regularization loss on a tensor by directly calling a regularizer as if it is a one-argument function. E.g. <div class="codehilite"><pre><code>>>> regularizer = L2(2.) >>> tensor = ops.ones(shape=(5, 5)) >>> regularizer(tensor) 50.0 </code></pre></div> <h2 id="developing-new-regularizers">Developing new regularizers</h2> Any function that takes in a weight matrix and returns a scalar tensor can be used as a regularizer, e.g.: <div class="codehilite"><pre><code>>>> def l1_reg(weight_matrix): ... return 0.01 * ops.sum(ops.absolute(weight_matrix)) ... >>> layer = Dense(5, input_dim=5, ... kernel_initializer='ones', kernel_regularizer=l1_reg) >>> tensor = ops.ones(shape=(5, 5)) >>> out = layer(tensor) >>> layer.losses 0.25 </code></pre></div> Alternatively, you can write your custom regularizers in an object-oriented way by extending this regularizer base class, e.g.: <div class="codehilite"><pre><code>>>> class L2Regularizer(Regularizer): ... def __init__(self, l2=0.): ... self.l2 = l2 ... ... def __call__(self, x): ... return self.l2 * ops.sum(ops.square(x)) ... ... def get_config(self): ... return {'l2': float(self.l2)} ... >>> layer = Dense( ... 5, input_dim=5, kernel_initializer='ones', ... kernel_regularizer=L2Regularizer(l2=0.5)) </code></pre></div> <div class="codehilite"><pre><code>>>> tensor = ops.ones(shape=(5, 5)) >>> out = layer(tensor) >>> layer.losses 12.5 </code></pre></div> <h3 id="a-note-on-serialization-and-deserialization">A note on serialization and deserialization:</h3> Registering the regularizers as serializable is optional if you are just training and executing models, exporting to and from SavedModels, or saving and loading weight checkpoints. Registration is required for saving and loading models to HDF5 format, Keras model cloning, some visualization utilities, and exporting models to and from JSON. If using this functionality, you must make sure any python process running your model has also defined and registered your custom regularizer. <hr /> <a href="https://github.com/keras-team/keras/tree/v3.6.0/keras/src/regularizers/regularizers.py#L213">[source]</a> <h3 id="l1-class"><code>L1</code> class</h3> <div class="codehilite"><pre><code>keras.regularizers.L1(l1=0.01) </code></pre></div> A regularizer that applies a L1 regularization penalty. The L1 regularization penalty is computed as: <code>loss = l1 * reduce_sum(abs(x))</code> L1 may be passed to a layer as a string identifier: <div class="codehilite"><pre><code>>>> dense = Dense(3, kernel_regularizer='l1') </code></pre></div> In this case, the default value used is <code>l1=0.01</code>. Arguments <ul> <li>l1: float, L1 regularization factor.</li> </ul> <hr /> <a href="https://github.com/keras-team/keras/tree/v3.6.0/keras/src/regularizers/regularizers.py#L242">[source]</a> <h3 id="l2-class"><code>L2</code> class</h3> <div class="codehilite"><pre><code>keras.regularizers.L2(l2=0.01) </code></pre></div> A regularizer that applies a L2 regularization penalty. The L2 regularization penalty is computed as: <code>loss = l2 * reduce_sum(square(x))</code> L2 may be passed to a layer as a string identifier: <div class="codehilite"><pre><code>>>> dense = Dense(3, kernel_regularizer='l2') </code></pre></div> In this case, the default value used is <code>l2=0.01</code>. Arguments <ul> <li>l2: float, L2 regularization factor.</li> </ul> <hr /> <a href="https://github.com/keras-team/keras/tree/v3.6.0/keras/src/regularizers/regularizers.py#L168">[source]</a> <h3 id="l1l2-class"><code>L1L2</code> class</h3> <div class="codehilite"><pre><code>keras.regularizers.L1L2(l1=0.0, l2=0.0) </code></pre></div> A regularizer that applies both L1 and L2 regularization penalties. The L1 regularization penalty is computed as: <code>loss = l1 * reduce_sum(abs(x))</code> The L2 regularization penalty is computed as <code>loss = l2 * reduce_sum(square(x))</code> L1L2 may be passed to a layer as a string identifier: <div class="codehilite"><pre><code>>>> dense = Dense(3, kernel_regularizer='l1_l2') </code></pre></div> In this case, the default values used are <code>l1=0.01</code> and <code>l2=0.01</code>. Arguments <ul> <li>l1: float, L1 regularization factor.</li> <li>l2: float, L2 regularization factor.</li> </ul> <hr /> <a href="https://github.com/keras-team/keras/tree/v3.6.0/keras/src/regularizers/regularizers.py#L271">[source]</a> <h3 id="orthogonalregularizer-class"><code>OrthogonalRegularizer</code> class</h3> <div class="codehilite"><pre><code>keras.regularizers.OrthogonalRegularizer(factor=0.01, mode="rows") </code></pre></div> Regularizer that encourages input vectors to be orthogonal to each other. It can be applied to either the rows of a matrix (<code>mode="rows"</code>) or its columns (<code>mode="columns"</code>). When applied to a <code>Dense</code> kernel of shape <code>(input_dim, units)</code>, rows mode will seek to make the feature vectors (i.e. the basis of the output space) orthogonal to each other. Arguments <ul> <li>factor: Float. The regularization factor. The regularization penalty will be proportional to <code>factor</code> times the mean of the dot products between the L2-normalized rows (if <code>mode="rows"</code>, or columns if <code>mode="columns"</code>) of the inputs, excluding the product of each row/column with itself. Defaults to <code>0.01</code>.</li> <li>mode: String, one of <code>{"rows", "columns"}</code>. Defaults to <code>"rows"</code>. In rows mode, the regularization effect seeks to make the rows of the input orthogonal to each other. In columns mode, it seeks to make the columns of the input orthogonal to each other.</li> </ul> Example <div class="codehilite"><pre><code>>>> regularizer = OrthogonalRegularizer(factor=0.01) >>> layer = Dense(units=4, kernel_regularizer=regularizer) </code></pre></div> <hr /> <h2 id="creating-custom-regularizers">Creating custom regularizers</h2> <h3 id="simple-callables">Simple callables</h3> A weight regularizer can be any callable that takes as input a weight tensor (e.g. the kernel of a <code>Conv2D</code> layer), and returns a scalar loss. Like this: <div class="codehilite"><pre><code>def my_regularizer(x): return 1e-3 * ops.sum(ops.square(x)) </code></pre></div> <h3 id="regularizer-subclasses"><code>Regularizer</code> subclasses</h3> If you need to configure your regularizer via various arguments (e.g. <code>l1</code> and <code>l2</code> arguments in <code>l1_l2</code>), you should implement it as a subclass of <a href="/api/layers/regularizers#regularizer-class"><code>keras.regularizers.Regularizer</code></a>. Here's a simple example: <div class="codehilite"><pre><code>class MyRegularizer(regularizers.Regularizer): def __init__(self, strength): self.strength = strength def __call__(self, x): return self.strength * ops.sum(ops.square(x)) </code></pre></div> Optionally, you can also implement the method <code>get_config</code> and the class method <code>from_config</code> in order to support serialization – just like with any Keras object. Example: <div class="codehilite"><pre><code>class MyRegularizer(regularizers.Regularizer): def __init__(self, strength): self.strength = strength def __call__(self, x): return self.strength * ops.sum(ops.square(x)) def get_config(self): return {'strength': self.strength} </code></pre></div> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#layer-weight-regularizers'>Layer weight regularizers</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#available-regularizers'>Available regularizers</a> </div> <div class='k-outline-depth-3'> <a href='#regularizer-class'><code>Regularizer</code> class</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#example'>Example</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#available-penalties'>Available penalties</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#directly-calling-a-regularizer'>Directly calling a regularizer</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#developing-new-regularizers'>Developing new regularizers</a> </div> <div class='k-outline-depth-3'> <a href='#a-note-on-serialization-and-deserialization'>A note on serialization and deserialization:</a> </div> <div class='k-outline-depth-3'> <a href='#l1-class'><code>L1</code> class</a> </div> <div class='k-outline-depth-3'> <a href='#l2-class'><code>L2</code> class</a> </div> <div class='k-outline-depth-3'> <a href='#l1l2-class'><code>L1L2</code> class</a> </div> <div class='k-outline-depth-3'> <a href='#orthogonalregularizer-class'><code>OrthogonalRegularizer</code> class</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#creating-custom-regularizers'>Creating custom regularizers</a> </div> <div class='k-outline-depth-3'> <a href='#simple-callables'>Simple callables</a> </div> <div class='k-outline-depth-3'> <a href='#regularizer-subclasses'><code>Regularizer</code> subclasses</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>

CINXE.COM

Layer weight regularizers