CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/keras_recipes/tf_serving/" />  <meta property="og:title" content="Keras documentation: Serving TensorFlow models with TFServing"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Serving TensorFlow models with TFServing"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Serving TensorFlow models with TFServing</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink active" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-sublink2" href="/examples/keras_recipes/parameter_efficient_finetuning_of_gemma_with_lora_and_qlora/">Parameter-efficient fine-tuning of Gemma with LoRA and QLoRA</a> <a class="nav-sublink2" href="/examples/keras_recipes/float8_training_and_inference_with_transformer/">Float8 training and inference with a simple Transformer model</a> <a class="nav-sublink2 active" href="/examples/keras_recipes/tf_serving/">Serving TensorFlow models with TFServing</a> <a class="nav-sublink2" href="/examples/keras_recipes/debugging_tips/">Keras debugging tips</a> <a class="nav-sublink2" href="/examples/keras_recipes/subclassing_conv_layers/">Customizing the convolution operation of a Conv2D layer</a> <a class="nav-sublink2" href="/examples/keras_recipes/trainer_pattern/">Trainer pattern</a> <a class="nav-sublink2" href="/examples/keras_recipes/endpoint_layer_pattern/">Endpoint layer pattern</a> <a class="nav-sublink2" href="/examples/keras_recipes/reproducibility_recipes/">Reproducibility in Keras Models</a> <a class="nav-sublink2" href="/examples/keras_recipes/tensorflow_numpy_models/">Writing Keras Models With TensorFlow NumPy</a> <a class="nav-sublink2" href="/examples/keras_recipes/antirectifier/">Simple custom layer example: Antirectifier</a> <a class="nav-sublink2" href="/examples/keras_recipes/sample_size_estimate/">Estimating required sample size for model training</a> <a class="nav-sublink2" href="/examples/keras_recipes/memory_efficient_embeddings/">Memory-efficient embeddings for recommendation systems</a> <a class="nav-sublink2" href="/examples/keras_recipes/creating_tfrecords/">Creating TFRecords</a> <a class="nav-sublink2" href="/examples/keras_recipes/packaging_keras_models_for_wide_distribution/">Packaging Keras models for wide distribution using Functional Subclassing</a> <a class="nav-sublink2" href="/examples/keras_recipes/approximating_non_function_mappings/">Approximating non-Function Mappings with Mixture Density Networks</a> <a class="nav-sublink2" href="/examples/keras_recipes/bayesian_neural_networks/">Probabilistic Bayesian Neural Networks</a> <a class="nav-sublink2" href="/examples/keras_recipes/better_knowledge_distillation/">Knowledge distillation recipes</a> <a class="nav-sublink2" href="/examples/keras_recipes/sklearn_metric_callbacks/">Evaluating and exporting scikit-learn metrics in a Keras callback</a> <a class="nav-sublink2" href="/examples/keras_recipes/tfrecord/">How to train a Keras model on TFRecord files</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparameter Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> <a class="nav-link" href="/keras_cv/" role="tab" aria-selected="">KerasCV: Computer Vision Workflows</a> <a class="nav-link" href="/keras_nlp/" role="tab" aria-selected="">KerasNLP: Natural Language Workflows</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/keras_recipes/'>Quick Keras Recipes</a> / Serving TensorFlow models with TFServing </div> <div class='k-content'> <h1 id="serving-tensorflow-models-with-tfserving">Serving TensorFlow models with TFServing</h1> Author: <a href="https://www.linkedin.com/in/dimitre-oliveira-7a1a0113a/">Dimitre Oliveira</a> Date created: 2023/01/02 Last modified: 2023/01/02 Description: How to serve TensorFlow models with TensorFlow Serving. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/keras_recipes/ipynb/tf_serving.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/keras_recipes/tf_serving.py">GitHub source</a> <hr /> <h2 id="introduction">Introduction</h2> Once you build a machine learning model, the next step is to serve it. You may want to do that by exposing your model as an endpoint service. There are many frameworks that you can use to do that, but the TensorFlow ecosystem has its own solution called <a href="https://www.tensorflow.org/tfx/guide/serving">TensorFlow Serving</a>. From the TensorFlow Serving <a href="https://github.com/tensorflow/serving">GitHub page</a>: <blockquote> TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. It deals with the inference aspect of machine learning, taking models after training and managing their lifetimes, providing clients with versioned access via a high-performance, reference-counted lookup table. TensorFlow Serving provides out-of-the-box integration with TensorFlow models, but can be easily extended to serve other types of models and data." </blockquote> To note a few features: <ul> <li>It can serve multiple models, or multiple versions of the same model simultaneously</li> <li>It exposes both gRPC as well as HTTP inference endpoints</li> <li>It allows deployment of new model versions without changing any client code</li> <li>It supports canarying new versions and A/B testing experimental models</li> <li>It adds minimal latency to inference time due to efficient, low-overhead implementation</li> <li>It features a scheduler that groups individual inference requests into batches for joint execution on GPU, with configurable latency controls</li> <li>It supports many servables: Tensorflow models, embeddings, vocabularies, feature transformations and even non-Tensorflow-based machine learning models</li> </ul> This guide creates a simple <a href="https://arxiv.org/abs/1704.04861">MobileNet</a> model using the <a href="https://keras.io/api/applications/">Keras applications API</a>, and then serves it with <a href="https://www.tensorflow.org/tfx/guide/serving">TensorFlow Serving</a>. The focus is on TensorFlow Serving, rather than the modeling and training in TensorFlow. <blockquote> Note: you can find a Colab notebook with the full working code at <a href="https://colab.research.google.com/drive/1nwuIJa4so1XzYU0ngq8tX_-SGTO295Mu?usp=sharing">this link</a>. </blockquote> <hr /> <h2 id="dependencies">Dependencies</h2> <div class="codehilite"><pre><code>import os os.environ["KERAS_BACKEND"] = "tensorflow" import json import shutil import requests import numpy as np import tensorflow as tf import keras import matplotlib.pyplot as plt </code></pre></div> <hr /> <h2 id="model">Model</h2> Here we load a pre-trained <a href="https://arxiv.org/abs/1704.04861">MobileNet</a> from the <a href="https://keras.io/api/applications/">Keras applications</a>, this is the model that we are going to serve. <div class="codehilite"><pre><code>model = keras.applications.MobileNet() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet/mobilenet_1_0_224_tf.h5 17225924/17225924 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step </code></pre></div> </div> <hr /> <h2 id="preprocessing">Preprocessing</h2> Most models don't work out of the box on raw data, they usually require some kind of preprocessing step to adjust the data to the model requirements, in the case of this MobileNet we can see from its <a href="https://keras.io/api/applications/mobilenet/">API page</a> that it requires three basic steps for its input images: <ul> <li>Pixel values normalized to the <code>[0, 1]</code> range</li> <li>Pixel values scaled to the <code>[-1, 1]</code> range</li> <li>Images with the shape of <code>(224, 224, 3)</code> meaning <code>(height, width, channels)</code></li> </ul> We can do all of that with the following function: <div class="codehilite"><pre><code>def preprocess(image, mean=0.5, std=0.5, shape=(224, 224)): """Scale, normalize and resizes images.""" image = image / 255.0 # Scale image = (image - mean) / std # Normalize image = tf.image.resize(image, shape) # Resize return image </code></pre></div> A note regarding preprocessing and postprocessing using the "keras.applications" API All models that are available at the <a href="https://keras.io/api/applications/">Keras applications</a> API also provide <code>preprocess_input</code> and <code>decode_predictions</code> functions, those functions are respectively responsible for the preprocessing and postprocessing of each model, and already contains all the logic necessary for those steps. That is the recommended way to process inputs and outputs when using Keras applications models. For this guide, we are not using them to present the advantages of custom signatures in a clearer way. <hr /> <h2 id="postprocessing">Postprocessing</h2> In the same context most models output values that need extra processing to meet the user requirements, for instance, the user does not want to know the logits values for each class given an image, what the user wants is to know from which class it belongs. For our model, this translates to the following transformations on top of the model outputs: <ul> <li>Get the index of the class with the highest prediction</li> <li>Get the name of the class from that index</li> </ul> <div class="codehilite"><pre><code># Download human-readable labels for ImageNet. imagenet_labels_url = ( "https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt" ) response = requests.get(imagenet_labels_url) # Skipping background class labels = [x for x in response.text.split("\n") if x != ""][1:] # Convert the labels to the TensorFlow data format tf_labels = tf.constant(labels, dtype=tf.string) def postprocess(prediction, labels=tf_labels): """Convert from probs to labels.""" indices = tf.argmax(prediction, axis=-1) # Index with highest prediction label = tf.gather(params=labels, indices=indices) # Class name return label </code></pre></div> Now let's download a banana picture and see how everything comes together. <div class="codehilite"><pre><code>response = requests.get("https://i.imgur.com/j9xCCzn.jpeg", stream=True) with open("banana.jpeg", "wb") as f: shutil.copyfileobj(response.raw, f) sample_img = plt.imread("./banana.jpeg") print(f"Original image shape: {sample_img.shape}") print(f"Original image pixel range: ({sample_img.min()}, {sample_img.max()})") plt.imshow(sample_img) plt.show() preprocess_img = preprocess(sample_img) print(f"Preprocessed image shape: {preprocess_img.shape}") print( f"Preprocessed image pixel range: ({preprocess_img.numpy().min()},", f"{preprocess_img.numpy().max()})", ) batched_img = tf.expand_dims(preprocess_img, axis=0) batched_img = tf.cast(batched_img, tf.float32) print(f"Batched image shape: {batched_img.shape}") model_outputs = model(batched_img) print(f"Model output shape: {model_outputs.shape}") print(f"Predicted class: {postprocess(model_outputs)}") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Original image shape: (540, 960, 3) Original image pixel range: (0, 255) </code></pre></div> </div> <img alt="png" src="/img/examples/keras_recipes/tf_serving/tf_serving_12_1.png" /> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Preprocessed image shape: (224, 224, 3) Preprocessed image pixel range: (-1.0, 1.0) Batched image shape: (1, 224, 224, 3) Model output shape: (1, 1000) Predicted class: [b'banana'] </code></pre></div> </div> <hr /> <h2 id="save-the-model">Save the model</h2> To load our trained model into TensorFlow Serving, we first need to save it in <a href="https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/saved_model">SavedModel</a> format. This will create a protobuf file in a well-defined directory hierarchy, and will include a version number. <a href="https://www.tensorflow.org/tfx/guide/serving">TensorFlow Serving</a> allows us to select which version of a model, or "servable" we want to use when we make inference requests. Each version will be exported to a different sub-directory under the given path. <div class="codehilite"><pre><code>model_dir = "./model" model_version = 1 model_export_path = f"{model_dir}/{model_version}" tf.saved_model.save( model, export_dir=model_export_path, ) print(f"SavedModel files: {os.listdir(model_export_path)}") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>INFO:tensorflow:Assets written to: ./model/1/assets INFO:tensorflow:Assets written to: ./model/1/assets SavedModel files: ['variables', 'saved_model.pb', 'assets', 'fingerprint.pb'] </code></pre></div> </div> <hr /> <h2 id="examine-your-saved-model">Examine your saved model</h2> We'll use the command line utility <code>saved_model_cli</code> to look at the <a href="https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/MetaGraphDef">MetaGraphDefs</a> (the models) and <a href="https://www.tensorflow.org/tfx/serving/signature_defs">SignatureDefs</a> (the methods you can call) in our SavedModel. See <a href="https://github.com/tensorflow/docs/blob/master/site/en/r1/guide/saved_model.md#cli-to-inspect-and-execute-savedmodel">this discussion of the SavedModel CLI</a> in the TensorFlow Guide. <div class="codehilite"><pre><code>!saved_model_cli show --dir {model_export_path} --tag_set serve --signature_def serving_default </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>The given SavedModel SignatureDef contains the following input(s): inputs['inputs'] tensor_info: dtype: DT_FLOAT shape: (-1, 224, 224, 3) name: serving_default_inputs:0 The given SavedModel SignatureDef contains the following output(s): outputs['output_0'] tensor_info: dtype: DT_FLOAT shape: (-1, 1000) name: StatefulPartitionedCall:0 Method name is: tensorflow/serving/predict </code></pre></div> </div> That tells us a lot about our model! For instance, we can see that its inputs have a 4D shape <code>(-1, 224, 224, 3)</code> which means <code>(batch_size, height, width, channels)</code>, also note that this model requires a specific image shape <code>(224, 224, 3)</code> this means that we may need to reshape our images before sending them to the model. We can also see that the model's outputs have a <code>(-1, 1000)</code> shape which are the logits for the 1000 classes of the <a href="https://www.image-net.org">ImageNet</a> dataset. This information doesn't tell us everything, like the fact that the pixel values needs to be in the <code>[-1, 1]</code> range, but it's a great start. <hr /> <h2 id="serve-your-model-with-tensorflow-serving">Serve your model with TensorFlow Serving</h2> <h3 id="install-tfserving">Install TFServing</h3> We're preparing to install TensorFlow Serving using <a href="https://wiki.debian.org/Aptitude">Aptitude</a> since this Colab runs in a Debian environment. We'll add the <code>tensorflow-model-server</code> package to the list of packages that Aptitude knows about. Note that we're running as root. <blockquote> Note: This example is running TensorFlow Serving natively, but <a href="https://www.tensorflow.org/tfx/serving/docker">you can also run it in a Docker container</a>, which is one of the easiest ways to get started using TensorFlow Serving. </blockquote> <div class="codehilite"><pre><code>wget 'http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-universal-2.8.0/t/tensorflow-model-server-universal/tensorflow-model-server-universal_2.8.0_all.deb' dpkg -i tensorflow-model-server-universal_2.8.0_all.deb </code></pre></div> <h3 id="start-running-tensorflow-serving">Start running TensorFlow Serving</h3> This is where we start running TensorFlow Serving and load our model. After it loads, we can start making inference requests using REST. There are some important parameters: <ul> <li><code>port</code>: The port that you'll use for gRPC requests.</li> <li><code>rest_api_port</code>: The port that you'll use for REST requests.</li> <li><code>model_name</code>: You'll use this in the URL of REST requests. It can be anything.</li> <li><code>model_base_path</code>: This is the path to the directory where you've saved your model.</li> </ul> Check the <a href="https://github.com/tensorflow/serving/blob/master/tensorflow_serving/model_servers/main.cc">TFServing API reference</a> to get all the parameters available. <div class="codehilite"><pre><code># Environment variable with the path to the model os.environ["MODEL_DIR"] = f"{model_dir}" </code></pre></div> <div class="codehilite"><pre><code>%%bash --bg nohup tensorflow_model_server \ --port=8500 \ --rest_api_port=8501 \ --model_name=model \ --model_base_path=$MODEL_DIR >server.log 2>&1 </code></pre></div> <div class="codehilite"><pre><code># We can check the logs to the server to help troubleshooting !cat server.log </code></pre></div> outputs: <div class="codehilite"><pre><code>[warn] getaddrinfo: address family for nodename not supported [evhttp_server.cc : 245] NET_LOG: Entering the event loop ... </code></pre></div> <div class="codehilite"><pre><code># Now we can check if tensorflow is in the active services !sudo lsof -i -P -n | grep LISTEN </code></pre></div> outputs: <div class="codehilite"><pre><code>node 7 root 21u IPv6 19100 0t0 TCP *:8080 (LISTEN) kernel_ma 34 root 7u IPv4 18874 0t0 TCP 172.28.0.12:6000 (LISTEN) colab-fil 63 root 5u IPv4 17975 0t0 TCP *:3453 (LISTEN) colab-fil 63 root 6u IPv6 17976 0t0 TCP *:3453 (LISTEN) jupyter-n 81 root 6u IPv4 18092 0t0 TCP 172.28.0.12:9000 (LISTEN) python3 101 root 23u IPv4 18252 0t0 TCP 127.0.0.1:44915 (LISTEN) python3 132 root 3u IPv4 20548 0t0 TCP 127.0.0.1:15264 (LISTEN) python3 132 root 4u IPv4 20549 0t0 TCP 127.0.0.1:37977 (LISTEN) python3 132 root 9u IPv4 20662 0t0 TCP 127.0.0.1:40689 (LISTEN) tensorflo 1101 root 5u IPv4 35543 0t0 TCP *:8500 (LISTEN) tensorflo 1101 root 12u IPv4 35548 0t0 TCP *:8501 (LISTEN) </code></pre></div> <hr /> <h2 id="make-a-request-to-your-model-in-tensorflow-serving">Make a request to your model in TensorFlow Serving</h2> Now let's create the JSON object for an inference request, and see how well our model classifies it: <h3 id="rest-api">REST API</h3> <h4 id="newest-version-of-the-servable">Newest version of the servable</h4> We'll send a predict request as a POST to our server's REST endpoint, and pass it as an example. We'll ask our server to give us the latest version of our servable by not specifying a particular version. <div class="codehilite"><pre><code>data = json.dumps( { "signature_name": "serving_default", "instances": batched_img.numpy().tolist(), } ) url = "http://localhost:8501/v1/models/model:predict" def predict_rest(json_data, url): json_response = requests.post(url, data=json_data) response = json.loads(json_response.text) rest_outputs = np.array(response["predictions"]) return rest_outputs </code></pre></div> <div class="codehilite"><pre><code>rest_outputs = predict_rest(data, url) print(f"REST output shape: {rest_outputs.shape}") print(f"Predicted class: {postprocess(rest_outputs)}") </code></pre></div> outputs: <div class="codehilite"><pre><code>REST output shape: (1, 1000) Predicted class: [b'banana'] </code></pre></div> <h3 id="grpc-api">gRPC API</h3> <a href="https://grpc.io/">gRPC</a> is based on the Remote Procedure Call (RPC) model and is a technology for implementing RPC APIs that uses HTTP 2.0 as its underlying transport protocol. gRPC is usually preferred for low-latency, highly scalable, and distributed systems. If you wanna know more about the REST vs gRPC tradeoffs, checkout <a href="https://cloud.google.com/blog/products/api-management/understanding-grpc-openapi-and-rest-and-when-to-use-them">this article</a>. <div class="codehilite"><pre><code>import grpc # Create a channel that will be connected to the gRPC port of the container channel = grpc.insecure_channel("localhost:8500") </code></pre></div> <div class="codehilite"><pre><code>pip install -q tensorflow_serving_api </code></pre></div> <div class="codehilite"><pre><code>from tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc # Create a stub made for prediction # This stub will be used to send the gRPCrequest to the TF Server stub = prediction_service_pb2_grpc.PredictionServiceStub(channel) </code></pre></div> <div class="codehilite"><pre><code># Get the serving_input key loaded_model = tf.saved_model.load(model_export_path) input_name = list( loaded_model.signatures["serving_default"].structured_input_signature[1].keys() )[0] </code></pre></div> <div class="codehilite"><pre><code>def predict_grpc(data, input_name, stub): # Create a gRPC request made for prediction request = predict_pb2.PredictRequest() # Set the name of the model, for this use case it is "model" request.model_spec.name = "model" # Set which signature is used to format the gRPC query # here the default one "serving_default" request.model_spec.signature_name = "serving_default" # Set the input as the data # tf.make_tensor_proto turns a TensorFlow tensor into a Protobuf tensor request.inputs[input_name].CopyFrom(tf.make_tensor_proto(data.numpy().tolist())) # Send the gRPC request to the TF Server result = stub.Predict(request) return result grpc_outputs = predict_grpc(batched_img, input_name, stub) grpc_outputs = np.array([grpc_outputs.outputs['predictions'].float_val]) print(f"gRPC output shape: {grpc_outputs.shape}") print(f"Predicted class: {postprocess(grpc_outputs)}") </code></pre></div> outputs: <div class="codehilite"><pre><code>gRPC output shape: (1, 1000) Predicted class: [b'banana'] </code></pre></div> <hr /> <h2 id="custom-signature">Custom signature</h2> Note that for this model we always need to preprocess and postprocess all samples to get the desired output, this can get quite tricky if are maintaining and serving several models developed by a large team, and each one of them might require different processing logic. TensorFlow allows us to customize the model graph to embed all of that processing logic, which makes model serving much easier, there are different ways to achieve this, but since we are going to server the models using TFServing we can customize the model graph straight into the serving signature. We can just use the following code to export the same model that already contains the preprocessing and postprocessing logic as the default signature, this allows this model to make predictions on raw data. <div class="codehilite"><pre><code>def export_model(model, labels): @tf.function(input_signature=[tf.TensorSpec([None, None, None, 3], tf.float32)]) def serving_fn(image): processed_img = preprocess(image) probs = model(processed_img) label = postprocess(probs) return {"label": label} return serving_fn model_sig_version = 2 model_sig_export_path = f"{model_dir}/{model_sig_version}" tf.saved_model.save( model, export_dir=model_sig_export_path, signatures={"serving_default": export_model(model, labels)}, ) </code></pre></div> <div class="codehilite"><pre><code>!saved_model_cli show --dir {model_sig_export_path} --tag_set serve --signature_def serving_default </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>INFO:tensorflow:Assets written to: ./model/2/assets INFO:tensorflow:Assets written to: ./model/2/assets The given SavedModel SignatureDef contains the following input(s): inputs['image'] tensor_info: dtype: DT_FLOAT shape: (-1, -1, -1, 3) name: serving_default_image:0 The given SavedModel SignatureDef contains the following output(s): outputs['label'] tensor_info: dtype: DT_STRING shape: (-1) name: StatefulPartitionedCall:0 Method name is: tensorflow/serving/predict </code></pre></div> </div> Note that this model has a different signature, its input is still 4D but now with a <code>(-1, -1, -1, 3)</code> shape, which means that it supports images with any height and width size. Its output also has a different shape, it no longer outputs the 1000-long logits. We can test the model's prediction using a specific signature using this API below: <div class="codehilite"><pre><code>batched_raw_img = tf.expand_dims(sample_img, axis=0) batched_raw_img = tf.cast(batched_raw_img, tf.float32) loaded_model = tf.saved_model.load(model_sig_export_path) loaded_model.signatures["serving_default"](**{"image": batched_raw_img}) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>{'label': <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'banana'], dtype=object)>} </code></pre></div> </div> <hr /> <h2 id="prediction-using-a-particular-version-of-the-servable">Prediction using a particular version of the servable</h2> Now let's specify a particular version of our servable. Note that when we saved the model with a custom signature we used a different folder, the first model was saved in folder <code>/1</code> (version 1), and the one with a custom signature in folder <code>/2</code> (version 2). By default, TFServing will serve all models that share the same base parent folder. <h3 id="rest-api">REST API</h3> <div class="codehilite"><pre><code>data = json.dumps( { "signature_name": "serving_default", "instances": batched_raw_img.numpy().tolist(), } ) url_sig = "http://localhost:8501/v1/models/model/versions/2:predict" </code></pre></div> <div class="codehilite"><pre><code>print(f"REST output shape: {rest_outputs.shape}") print(f"Predicted class: {rest_outputs}") </code></pre></div> outputs: <div class="codehilite"><pre><code>REST output shape: (1,) Predicted class: ['banana'] </code></pre></div> <h3 id="grpc-api">gRPC API</h3> <div class="codehilite"><pre><code>channel = grpc.insecure_channel("localhost:8500") stub = prediction_service_pb2_grpc.PredictionServiceStub(channel) </code></pre></div> <div class="codehilite"><pre><code>input_name = list( loaded_model.signatures["serving_default"].structured_input_signature[1].keys() )[0] </code></pre></div> <div class="codehilite"><pre><code>grpc_outputs = predict_grpc(batched_raw_img, input_name, stub) grpc_outputs = np.array([grpc_outputs.outputs['label'].string_val]) print(f"gRPC output shape: {grpc_outputs.shape}") print(f"Predicted class: {grpc_outputs}") </code></pre></div> outputs: <div class="codehilite"><pre><code>gRPC output shape: (1, 1) Predicted class: [[b'banana']] </code></pre></div> <hr /> <h2 id="additional-resources">Additional resources</h2> <ul> <li><a href="https://colab.research.google.com/drive/1nwuIJa4so1XzYU0ngq8tX_-SGTO295Mu?usp=sharing">Colab notebook with the full working code</a></li> <li><a href="https://www.tensorflow.org/tfx/tutorials/serving/rest_simple#make_a_request_to_your_model_in_tensorflow_serving">Train and serve a TensorFlow model with TensorFlow Serving - TensorFlow blog</a></li> <li><a href="https://www.youtube.com/playlist?list=PLQY2H8rRoyvwHdpVQVohY7-qcYf2s1UYK">TensorFlow Serving playlist - TensorFlow YouTube channel</a></li> </ul> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#serving-tensorflow-models-with-tfserving'>Serving TensorFlow models with TFServing</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#dependencies'>Dependencies</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#model'>Model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#preprocessing'>Preprocessing</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#postprocessing'>Postprocessing</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#save-the-model'>Save the model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#examine-your-saved-model'>Examine your saved model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#serve-your-model-with-tensorflow-serving'>Serve your model with TensorFlow Serving</a> </div> <div class='k-outline-depth-3'> <a href='#install-tfserving'>Install TFServing</a> </div> <div class='k-outline-depth-3'> <a href='#start-running-tensorflow-serving'>Start running TensorFlow Serving</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#make-a-request-to-your-model-in-tensorflow-serving'>Make a request to your model in TensorFlow Serving</a> </div> <div class='k-outline-depth-3'> <a href='#rest-api'>REST API</a> </div> <div class='k-outline-depth-3'> <a href='#grpc-api'>gRPC API</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#custom-signature'>Custom signature</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#prediction-using-a-particular-version-of-the-servable'>Prediction using a particular version of the servable</a> </div> <div class='k-outline-depth-3'> <a href='#rest-api'>REST API</a> </div> <div class='k-outline-depth-3'> <a href='#grpc-api'>gRPC API</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#additional-resources'>Additional resources</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>