CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/structured_data/deep_neural_decision_forests/" />  <meta property="og:title" content="Keras documentation: Classification with Neural Decision Forests"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Classification with Neural Decision Forests"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Classification with Neural Decision Forests</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink active" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink2" href="/examples/structured_data/structured_data_classification_with_feature_space/">Structured data classification with FeatureSpace</a> <a class="nav-sublink2" href="/examples/structured_data/feature_space_advanced/">FeatureSpace advanced use cases</a> <a class="nav-sublink2" href="/examples/structured_data/imbalanced_classification/">Imbalanced classification: credit card fraud detection</a> <a class="nav-sublink2" href="/examples/structured_data/structured_data_classification_from_scratch/">Structured data classification from scratch</a> <a class="nav-sublink2" href="/examples/structured_data/wide_deep_cross_networks/">Structured data learning with Wide, Deep, and Cross networks</a> <a class="nav-sublink2" href="/examples/structured_data/classification_with_grn_and_vsn/">Classification with Gated Residual and Variable Selection Networks</a> <a class="nav-sublink2" href="/examples/structured_data/classification_with_tfdf/">Classification with TensorFlow Decision Forests</a> <a class="nav-sublink2 active" href="/examples/structured_data/deep_neural_decision_forests/">Classification with Neural Decision Forests</a> <a class="nav-sublink2" href="/examples/structured_data/tabtransformer/">Structured data learning with TabTransformer</a> <a class="nav-sublink2" href="/examples/structured_data/collaborative_filtering_movielens/">Collaborative Filtering for Movie Recommendations</a> <a class="nav-sublink2" href="/examples/structured_data/movielens_recommendations_transformers/">A Transformer-based recommendation system</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparameter Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> <a class="nav-link" href="/keras_cv/" role="tab" aria-selected="">KerasCV: Computer Vision Workflows</a> <a class="nav-link" href="/keras_nlp/" role="tab" aria-selected="">KerasNLP: Natural Language Workflows</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/structured_data/'>Structured Data</a> / Classification with Neural Decision Forests </div> <div class='k-content'> <h1 id="classification-with-neural-decision-forests">Classification with Neural Decision Forests</h1> Author: <a href="https://www.linkedin.com/in/khalid-salama-24403144/">Khalid Salama</a> Date created: 2021/01/15 Last modified: 2021/01/15 Description: How to train differentiable decision trees for end-to-end learning in deep neural networks. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/structured_data/ipynb/deep_neural_decision_forests.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/structured_data/deep_neural_decision_forests.py">GitHub source</a> <hr /> <h2 id="introduction">Introduction</h2> This example provides an implementation of the <a href="https://ieeexplore.ieee.org/document/7410529">Deep Neural Decision Forest</a> model introduced by P. Kontschieder et al. for structured data classification. It demonstrates how to build a stochastic and differentiable decision tree model, train it end-to-end, and unify decision trees with deep representation learning. <hr /> <h2 id="the-dataset">The dataset</h2> This example uses the <a href="https://archive.ics.uci.edu/ml/datasets/census+income">United States Census Income Dataset</a> provided by the <a href="https://archive.ics.uci.edu/ml/index.php">UC Irvine Machine Learning Repository</a>. The task is binary classification to predict whether a person is likely to be making over USD 50,000 a year. The dataset includes 48,842 instances with 14 input features (such as age, work class, education, occupation, and so on): 5 numerical features and 9 categorical features. <hr /> <h2 id="setup">Setup</h2> <div class="codehilite"><pre><code>import keras from keras import layers from keras.layers import StringLookup from keras import ops from tensorflow import data as tf_data import numpy as np import pandas as pd import math </code></pre></div> <hr /> <h2 id="prepare-the-data">Prepare the data</h2> <div class="codehilite"><pre><code>CSV_HEADER = [ "age", "workclass", "fnlwgt", "education", "education_num", "marital_status", "occupation", "relationship", "race", "gender", "capital_gain", "capital_loss", "hours_per_week", "native_country", "income_bracket", ] train_data_url = ( "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data" ) train_data = pd.read_csv(train_data_url, header=None, names=CSV_HEADER) test_data_url = ( "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test" ) test_data = pd.read_csv(test_data_url, header=None, names=CSV_HEADER) print(f"Train dataset shape: {train_data.shape}") print(f"Test dataset shape: {test_data.shape}") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Train dataset shape: (32561, 15) Test dataset shape: (16282, 15) </code></pre></div> </div> Remove the first record (because it is not a valid data example) and a trailing 'dot' in the class labels. <div class="codehilite"><pre><code>test_data = test_data[1:] test_data.income_bracket = test_data.income_bracket.apply( lambda value: value.replace(".", "") ) </code></pre></div> We store the training and test data splits locally as CSV files. <div class="codehilite"><pre><code>train_data_file = "train_data.csv" test_data_file = "test_data.csv" train_data.to_csv(train_data_file, index=False, header=False) test_data.to_csv(test_data_file, index=False, header=False) </code></pre></div> <hr /> <h2 id="define-dataset-metadata">Define dataset metadata</h2> Here, we define the metadata of the dataset that will be useful for reading and parsing and encoding input features. <div class="codehilite"><pre><code># A list of the numerical feature names. NUMERIC_FEATURE_NAMES = [ "age", "education_num", "capital_gain", "capital_loss", "hours_per_week", ] # A dictionary of the categorical features and their vocabulary. CATEGORICAL_FEATURES_WITH_VOCABULARY = { "workclass": sorted(list(train_data["workclass"].unique())), "education": sorted(list(train_data["education"].unique())), "marital_status": sorted(list(train_data["marital_status"].unique())), "occupation": sorted(list(train_data["occupation"].unique())), "relationship": sorted(list(train_data["relationship"].unique())), "race": sorted(list(train_data["race"].unique())), "gender": sorted(list(train_data["gender"].unique())), "native_country": sorted(list(train_data["native_country"].unique())), } # A list of the columns to ignore from the dataset. IGNORE_COLUMN_NAMES = ["fnlwgt"] # A list of the categorical feature names. CATEGORICAL_FEATURE_NAMES = list(CATEGORICAL_FEATURES_WITH_VOCABULARY.keys()) # A list of all the input features. FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES # A list of column default values for each feature. COLUMN_DEFAULTS = [ [0.0] if feature_name in NUMERIC_FEATURE_NAMES + IGNORE_COLUMN_NAMES else ["NA"] for feature_name in CSV_HEADER ] # The name of the target feature. TARGET_FEATURE_NAME = "income_bracket" # A list of the labels of the target features. TARGET_LABELS = [" <=50K", " >50K"] </code></pre></div> <hr /> <h2 id="create-tfdatadataset-objects-for-training-and-validation">Create <code>tf_data.Dataset</code> objects for training and validation</h2> We create an input function to read and parse the file, and convert features and labels into a <a href="https://www.tensorflow.org/guide/datasets"><code>tf_data.Dataset</code></a> for training and validation. We also preprocess the input by mapping the target label to an index. <div class="codehilite"><pre><code>target_label_lookup = StringLookup( vocabulary=TARGET_LABELS, mask_token=None, num_oov_indices=0 ) lookup_dict = {} for feature_name in CATEGORICAL_FEATURE_NAMES: vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY[feature_name] # Create a lookup to convert a string values to an integer indices. # Since we are not using a mask token, nor expecting any out of vocabulary # (oov) token, we set mask_token to None and num_oov_indices to 0. lookup = StringLookup(vocabulary=vocabulary, mask_token=None, num_oov_indices=0) lookup_dict[feature_name] = lookup def encode_categorical(batch_x, batch_y): for feature_name in CATEGORICAL_FEATURE_NAMES: batch_x[feature_name] = lookup_dict[feature_name](batch_x[feature_name]) return batch_x, batch_y def get_dataset_from_csv(csv_file_path, shuffle=False, batch_size=128): dataset = ( tf_data.experimental.make_csv_dataset( csv_file_path, batch_size=batch_size, column_names=CSV_HEADER, column_defaults=COLUMN_DEFAULTS, label_name=TARGET_FEATURE_NAME, num_epochs=1, header=False, na_value="?", shuffle=shuffle, ) .map(lambda features, target: (features, target_label_lookup(target))) .map(encode_categorical) ) return dataset.cache() </code></pre></div> <hr /> <h2 id="create-model-inputs">Create model inputs</h2> <div class="codehilite"><pre><code>def create_model_inputs(): inputs = {} for feature_name in FEATURE_NAMES: if feature_name in NUMERIC_FEATURE_NAMES: inputs[feature_name] = layers.Input( name=feature_name, shape=(), dtype="float32" ) else: inputs[feature_name] = layers.Input( name=feature_name, shape=(), dtype="int32" ) return inputs </code></pre></div> <hr /> <h2 id="encode-input-features">Encode input features</h2> <div class="codehilite"><pre><code>def encode_inputs(inputs): encoded_features = [] for feature_name in inputs: if feature_name in CATEGORICAL_FEATURE_NAMES: vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY[feature_name] # Create a lookup to convert a string values to an integer indices. # Since we are not using a mask token, nor expecting any out of vocabulary # (oov) token, we set mask_token to None and num_oov_indices to 0. value_index = inputs[feature_name] embedding_dims = int(math.sqrt(lookup.vocabulary_size())) # Create an embedding layer with the specified dimensions. embedding = layers.Embedding( input_dim=lookup.vocabulary_size(), output_dim=embedding_dims ) # Convert the index values to embedding representations. encoded_feature = embedding(value_index) else: # Use the numerical features as-is. encoded_feature = inputs[feature_name] if inputs[feature_name].shape[-1] is None: encoded_feature = keras.ops.expand_dims(encoded_feature, -1) encoded_features.append(encoded_feature) encoded_features = layers.concatenate(encoded_features) return encoded_features </code></pre></div> <hr /> <h2 id="deep-neural-decision-tree">Deep Neural Decision Tree</h2> A neural decision tree model has two sets of weights to learn. The first set is <code>pi</code>, which represents the probability distribution of the classes in the tree leaves. The second set is the weights of the routing layer <code>decision_fn</code>, which represents the probability of going to each leave. The forward pass of the model works as follows: <ol> <li>The model expects input <code>features</code> as a single vector encoding all the features of an instance in the batch. This vector can be generated from a Convolution Neural Network (CNN) applied to images or dense transformations applied to structured data features.</li> <li>The model first applies a <code>used_features_mask</code> to randomly select a subset of input features to use.</li> <li>Then, the model computes the probabilities (<code>mu</code>) for the input instances to reach the tree leaves by iteratively performing a stochastic routing throughout the tree levels.</li> <li>Finally, the probabilities of reaching the leaves are combined by the class probabilities at the leaves to produce the final <code>outputs</code>.</li> </ol> <div class="codehilite"><pre><code>class NeuralDecisionTree(keras.Model): def __init__(self, depth, num_features, used_features_rate, num_classes): super().__init__() self.depth = depth self.num_leaves = 2**depth self.num_classes = num_classes # Create a mask for the randomly selected features. num_used_features = int(num_features * used_features_rate) one_hot = np.eye(num_features) sampled_feature_indices = np.random.choice( np.arange(num_features), num_used_features, replace=False ) self.used_features_mask = ops.convert_to_tensor( one_hot[sampled_feature_indices], dtype="float32" ) # Initialize the weights of the classes in leaves. self.pi = self.add_weight( initializer="random_normal", shape=[self.num_leaves, self.num_classes], dtype="float32", trainable=True, ) # Initialize the stochastic routing layer. self.decision_fn = layers.Dense( units=self.num_leaves, activation="sigmoid", name="decision" ) def call(self, features): batch_size = ops.shape(features)[0] # Apply the feature mask to the input features. features = ops.matmul( features, ops.transpose(self.used_features_mask) ) # [batch_size, num_used_features] # Compute the routing probabilities. decisions = ops.expand_dims( self.decision_fn(features), axis=2 ) # [batch_size, num_leaves, 1] # Concatenate the routing probabilities with their complements. decisions = layers.concatenate( [decisions, 1 - decisions], axis=2 ) # [batch_size, num_leaves, 2] mu = ops.ones([batch_size, 1, 1]) begin_idx = 1 end_idx = 2 # Traverse the tree in breadth-first order. for level in range(self.depth): mu = ops.reshape(mu, [batch_size, -1, 1]) # [batch_size, 2 ** level, 1] mu = ops.tile(mu, (1, 1, 2)) # [batch_size, 2 ** level, 2] level_decisions = decisions[ :, begin_idx:end_idx, : ] # [batch_size, 2 ** level, 2] mu = mu * level_decisions # [batch_size, 2**level, 2] begin_idx = end_idx end_idx = begin_idx + 2 ** (level + 1) mu = ops.reshape(mu, [batch_size, self.num_leaves]) # [batch_size, num_leaves] probabilities = keras.activations.softmax(self.pi) # [num_leaves, num_classes] outputs = ops.matmul(mu, probabilities) # [batch_size, num_classes] return outputs </code></pre></div> <hr /> <h2 id="deep-neural-decision-forest">Deep Neural Decision Forest</h2> The neural decision forest model consists of a set of neural decision trees that are trained simultaneously. The output of the forest model is the average outputs of its trees. <div class="codehilite"><pre><code>class NeuralDecisionForest(keras.Model): def __init__(self, num_trees, depth, num_features, used_features_rate, num_classes): super().__init__() self.ensemble = [] # Initialize the ensemble by adding NeuralDecisionTree instances. # Each tree will have its own randomly selected input features to use. for _ in range(num_trees): self.ensemble.append( NeuralDecisionTree(depth, num_features, used_features_rate, num_classes) ) def call(self, inputs): # Initialize the outputs: a [batch_size, num_classes] matrix of zeros. batch_size = ops.shape(inputs)[0] outputs = ops.zeros([batch_size, num_classes]) # Aggregate the outputs of trees in the ensemble. for tree in self.ensemble: outputs += tree(inputs) # Divide the outputs by the ensemble size to get the average. outputs /= len(self.ensemble) return outputs </code></pre></div> Finally, let's set up the code that will train and evaluate the model. <div class="codehilite"><pre><code>learning_rate = 0.01 batch_size = 265 num_epochs = 10 def run_experiment(model): model.compile( optimizer=keras.optimizers.Adam(learning_rate=learning_rate), loss=keras.losses.SparseCategoricalCrossentropy(), metrics=[keras.metrics.SparseCategoricalAccuracy()], ) print("Start training the model...") train_dataset = get_dataset_from_csv( train_data_file, shuffle=True, batch_size=batch_size ) model.fit(train_dataset, epochs=num_epochs) print("Model training finished") print("Evaluating the model on the test data...") test_dataset = get_dataset_from_csv(test_data_file, batch_size=batch_size) _, accuracy = model.evaluate(test_dataset) print(f"Test accuracy: {round(accuracy * 100, 2)}%") </code></pre></div> <hr /> <h2 id="experiment-1-train-a-decision-tree-model">Experiment 1: train a decision tree model</h2> In this experiment, we train a single neural decision tree model where we use all input features. <div class="codehilite"><pre><code>num_trees = 10 depth = 10 used_features_rate = 1.0 num_classes = len(TARGET_LABELS) def create_tree_model(): inputs = create_model_inputs() features = encode_inputs(inputs) features = layers.BatchNormalization()(features) num_features = features.shape[1] tree = NeuralDecisionTree(depth, num_features, used_features_rate, num_classes) outputs = tree(features) model = keras.Model(inputs=inputs, outputs=outputs) return model tree_model = create_tree_model() run_experiment(tree_model) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Start training the model... Epoch 1/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 5s 26ms/step - loss: 0.5308 - sparse_categorical_accuracy: 0.8150 Epoch 2/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - loss: 0.3476 - sparse_categorical_accuracy: 0.8429 Epoch 3/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - loss: 0.3312 - sparse_categorical_accuracy: 0.8478 Epoch 4/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - loss: 0.3247 - sparse_categorical_accuracy: 0.8495 Epoch 5/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - loss: 0.3202 - sparse_categorical_accuracy: 0.8512 Epoch 6/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - loss: 0.3158 - sparse_categorical_accuracy: 0.8536 Epoch 7/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - loss: 0.3116 - sparse_categorical_accuracy: 0.8572 Epoch 8/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - loss: 0.3071 - sparse_categorical_accuracy: 0.8608 Epoch 9/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - loss: 0.3026 - sparse_categorical_accuracy: 0.8630 Epoch 10/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - loss: 0.2975 - sparse_categorical_accuracy: 0.8653 Model training finished Evaluating the model on the test data... 62/62 ━━━━━━━━━━━━━━━━━━━━ 1s 13ms/step - loss: 0.3279 - sparse_categorical_accuracy: 0.8463 Test accuracy: 85.08% </code></pre></div> </div> <hr /> <h2 id="experiment-2-train-a-forest-model">Experiment 2: train a forest model</h2> In this experiment, we train a neural decision forest with <code>num_trees</code> trees where each tree uses randomly selected 50% of the input features. You can control the number of features to be used in each tree by setting the <code>used_features_rate</code> variable. In addition, we set the depth to 5 instead of 10 compared to the previous experiment. <div class="codehilite"><pre><code>num_trees = 25 depth = 5 used_features_rate = 0.5 def create_forest_model(): inputs = create_model_inputs() features = encode_inputs(inputs) features = layers.BatchNormalization()(features) num_features = features.shape[1] forest_model = NeuralDecisionForest( num_trees, depth, num_features, used_features_rate, num_classes ) outputs = forest_model(features) model = keras.Model(inputs=inputs, outputs=outputs) return model forest_model = create_forest_model() run_experiment(forest_model) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Start training the model... Epoch 1/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 47s 202ms/step - loss: 0.5469 - sparse_categorical_accuracy: 0.7915 Epoch 2/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - loss: 0.3459 - sparse_categorical_accuracy: 0.8494 Epoch 3/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - loss: 0.3268 - sparse_categorical_accuracy: 0.8523 Epoch 4/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - loss: 0.3195 - sparse_categorical_accuracy: 0.8524 Epoch 5/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - loss: 0.3149 - sparse_categorical_accuracy: 0.8539 Epoch 6/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - loss: 0.3112 - sparse_categorical_accuracy: 0.8556 Epoch 7/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - loss: 0.3079 - sparse_categorical_accuracy: 0.8566 Epoch 8/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 9ms/step - loss: 0.3050 - sparse_categorical_accuracy: 0.8582 Epoch 9/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 9ms/step - loss: 0.3021 - sparse_categorical_accuracy: 0.8595 Epoch 10/10 123/123 ━━━━━━━━━━━━━━━━━━━━ 1s 9ms/step - loss: 0.2992 - sparse_categorical_accuracy: 0.8617 Model training finished Evaluating the model on the test data... 62/62 ━━━━━━━━━━━━━━━━━━━━ 5s 39ms/step - loss: 0.3145 - sparse_categorical_accuracy: 0.8503 Test accuracy: 85.55% </code></pre></div> </div> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#classification-with-neural-decision-forests'>Classification with Neural Decision Forests</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#the-dataset'>The dataset</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#setup'>Setup</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#prepare-the-data'>Prepare the data</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#define-dataset-metadata'>Define dataset metadata</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#create-tfdatadataset-objects-for-training-and-validation'>Create <code>tf_data.Dataset</code> objects for training and validation</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#create-model-inputs'>Create model inputs</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#encode-input-features'>Encode input features</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#deep-neural-decision-tree'>Deep Neural Decision Tree</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#deep-neural-decision-forest'>Deep Neural Decision Forest</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#experiment-1-train-a-decision-tree-model'>Experiment 1: train a decision tree model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#experiment-2-train-a-forest-model'>Experiment 2: train a forest model</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>