CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/nlp/sentence_embeddings_with_sbert/" />  <meta property="og:title" content="Keras documentation: Sentence embeddings using Siamese RoBERTa-networks"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: Sentence embeddings using Siamese RoBERTa-networks"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>Sentence embeddings using Siamese RoBERTa-networks</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink active" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink2" href="/examples/nlp/text_classification_from_scratch/">Text classification from scratch</a> <a class="nav-sublink2" href="/examples/nlp/active_learning_review_classification/">Review Classification using Active Learning</a> <a class="nav-sublink2" href="/examples/nlp/fnet_classification_with_keras_hub/">Text Classification using FNet</a> <a class="nav-sublink2" href="/examples/nlp/multi_label_classification/">Large-scale multi-label text classification</a> <a class="nav-sublink2" href="/examples/nlp/text_classification_with_transformer/">Text classification with Transformer</a> <a class="nav-sublink2" href="/examples/nlp/text_classification_with_switch_transformer/">Text classification with Switch Transformer</a> <a class="nav-sublink2" href="/examples/nlp/tweet-classification-using-tfdf/">Text classification using Decision Forests and pretrained embeddings</a> <a class="nav-sublink2" href="/examples/nlp/pretrained_word_embeddings/">Using pre-trained word embeddings</a> <a class="nav-sublink2" href="/examples/nlp/bidirectional_lstm_imdb/">Bidirectional LSTM on IMDB</a> <a class="nav-sublink2" href="/examples/nlp/data_parallel_training_with_keras_hub/">Data Parallel Training with KerasHub and tf.distribute</a> <a class="nav-sublink2" href="/examples/nlp/neural_machine_translation_with_keras_hub/">English-to-Spanish translation with KerasHub</a> <a class="nav-sublink2" href="/examples/nlp/neural_machine_translation_with_transformer/">English-to-Spanish translation with a sequence-to-sequence Transformer</a> <a class="nav-sublink2" href="/examples/nlp/lstm_seq2seq/">Character-level recurrent sequence-to-sequence model</a> <a class="nav-sublink2" href="/examples/nlp/multimodal_entailment/">Multimodal entailment</a> <a class="nav-sublink2" href="/examples/nlp/ner_transformers/">Named Entity Recognition using Transformers</a> <a class="nav-sublink2" href="/examples/nlp/text_extraction_with_bert/">Text Extraction with BERT</a> <a class="nav-sublink2" href="/examples/nlp/addition_rnn/">Sequence to sequence learning for performing number addition</a> <a class="nav-sublink2" href="/examples/nlp/semantic_similarity_with_keras_hub/">Semantic Similarity with KerasHub</a> <a class="nav-sublink2" href="/examples/nlp/semantic_similarity_with_bert/">Semantic Similarity with BERT</a> <a class="nav-sublink2 active" href="/examples/nlp/sentence_embeddings_with_sbert/">Sentence embeddings using Siamese RoBERTa-networks</a> <a class="nav-sublink2" href="/examples/nlp/masked_language_modeling/">End-to-end Masked Language Modeling with BERT</a> <a class="nav-sublink2" href="/examples/nlp/abstractive_summarization_with_bart/">Abstractive Text Summarization with BART</a> <a class="nav-sublink2" href="/examples/nlp/pretraining_BERT/">Pretraining BERT with Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/nlp/parameter_efficient_finetuning_of_gpt2_with_lora/">Parameter-efficient fine-tuning of GPT-2 with LoRA</a> <a class="nav-sublink2" href="/examples/nlp/mlm_training_tpus/">Training a language model from scratch with 🤗 Transformers and TPUs</a> <a class="nav-sublink2" href="/examples/nlp/multiple_choice_task_with_transfer_learning/">MultipleChoice Task with Transfer Learning</a> <a class="nav-sublink2" href="/examples/nlp/question_answering/">Question Answering with Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/nlp/t5_hf_summarization/">Abstractive Summarization with Hugging Face Transformers</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparameter Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> <a class="nav-link" href="/keras_cv/" role="tab" aria-selected="">KerasCV: Computer Vision Workflows</a> <a class="nav-link" href="/keras_nlp/" role="tab" aria-selected="">KerasNLP: Natural Language Workflows</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/nlp/'>Natural Language Processing</a> / Sentence embeddings using Siamese RoBERTa-networks </div> <div class='k-content'> <h1 id="sentence-embeddings-using-siamese-robertanetworks">Sentence embeddings using Siamese RoBERTa-networks</h1> Author: <a href="https://github.com/abuelnasr0">Mohammed Abu El-Nasr</a> Date created: 2023/07/14 Last modified: 2023/07/14 Description: Fine-tune a RoBERTa model to generate sentence embeddings using KerasHub. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/nlp/ipynb/sentence_embeddings_with_sbert.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/nlp/sentence_embeddings_with_sbert.py">GitHub source</a> <hr /> <h2 id="introduction">Introduction</h2> BERT and RoBERTa can be used for semantic textual similarity tasks, where two sentences are passed to the model and the network predicts whether they are similar or not. But what if we have a large collection of sentences and want to find the most similar pairs in that collection? That will take n*(n-1)/2 inference computations, where n is the number of sentences in the collection. For example, if n = 10000, the required time will be 65 hours on a V100 GPU. A common method to overcome the time overhead issue is to pass one sentence to the model, then average the output of the model, or take the first token (the [CLS] token) and use them as a <a href="https://en.wikipedia.org/wiki/Sentence_embedding">sentence embedding</a>, then use a vector similarity measure like cosine similarity or Manhatten / Euclidean distance to find close sentences (semantically similar sentences). That will reduce the time to find the most similar pairs in a collection of 10,000 sentences from 65 hours to 5 seconds! If we use RoBERTa directly, that will yield rather bad sentence embeddings. But if we fine-tune RoBERTa using a Siamese network, that will generate semantically meaningful sentence embeddings. This will enable RoBERTa to be used for new tasks. These tasks include: <ul> <li>Large-scale semantic similarity comparison.</li> <li>Clustering.</li> <li>Information retrieval via semantic search.</li> </ul> In this example, we will show how to fine-tune a RoBERTa model using a Siamese network such that it will be able to produce semantically meaningful sentence embeddings and use them in a semantic search and clustering example. This method of fine-tuning was introduced in <a href="https://arxiv.org/abs/1908.10084">Sentence-BERT</a> <hr /> <h2 id="setup">Setup</h2> Let's install and import the libraries we need. We'll be using the KerasHub library in this example. We will also enable <a href="https://www.tensorflow.org/guide/mixed_precision">mixed precision</a> training. This will help us reduce the training time. <div class="codehilite"><pre><code>!pip install -q --upgrade keras-hub !pip install -q --upgrade keras # Upgrade to Keras 3. </code></pre></div> <div class="codehilite"><pre><code>import os os.environ["KERAS_BACKEND"] = "tensorflow" import keras import keras_hub import tensorflow as tf import tensorflow_datasets as tfds import sklearn.cluster as cluster keras.mixed_precision.set_global_policy("mixed_float16") </code></pre></div> <hr /> <h2 id="finetune-the-model-using-siamese-networks">Fine-tune the model using siamese networks</h2> <a href="https://en.wikipedia.org/wiki/Siamese_neural_network">Siamese network</a> is a neural network architecture that contains two or more subnetworks. The subnetworks share the same weights. It is used to generate feature vectors for each input and then compare them for similarity. For our example, the subnetwork will be a RoBERTa model that has a pooling layer on top of it to produce the embeddings of the input sentences. These embeddings will then be compared to each other to learn to produce semantically meaningful embeddings. The pooling strategies used are mean, max, and CLS pooling. Mean pooling produces the best results. We will use it in our examples. <h3 id="finetune-using-the-regression-objective-function">Fine-tune using the regression objective function</h3> For building the siamese network with the regression objective function, the siamese network is asked to predict the cosine similarity between the embeddings of the two input sentences. Cosine similarity indicates the angle between the sentence embeddings. If the cosine similarity is high, that means there is a small angle between the embeddings; hence, they are semantically similar. <h4 id="load-the-dataset">Load the dataset</h4> We will use the STSB dataset to fine-tune the model for the regression objective. STSB consists of a collection of sentence pairs that are labelled in the range [0, 5]. 0 indicates the least semantic similarity between the two sentences, and 5 indicates the most semantic similarity between the two sentences. The range of the cosine similarity is [-1, 1] and it's the output of the siamese network, but the range of the labels in the dataset is [0, 5]. We need to unify the range between the cosine similarity and the dataset labels, so while preparing the dataset, we will divide the labels by 2.5 and subtract 1. <div class="codehilite"><pre><code>TRAIN_BATCH_SIZE = 6 VALIDATION_BATCH_SIZE = 8 TRAIN_NUM_BATCHES = 300 VALIDATION_NUM_BATCHES = 40 AUTOTUNE = tf.data.experimental.AUTOTUNE def change_range(x): return (x / 2.5) - 1 def prepare_dataset(dataset, num_batches, batch_size): dataset = dataset.map( lambda z: ( [z["sentence1"], z["sentence2"]], [tf.cast(change_range(z["label"]), tf.float32)], ), num_parallel_calls=AUTOTUNE, ) dataset = dataset.batch(batch_size) dataset = dataset.take(num_batches) dataset = dataset.prefetch(AUTOTUNE) return dataset stsb_ds = tfds.load( "glue/stsb", ) stsb_train, stsb_valid = stsb_ds["train"], stsb_ds["validation"] stsb_train = prepare_dataset(stsb_train, TRAIN_NUM_BATCHES, TRAIN_BATCH_SIZE) stsb_valid = prepare_dataset(stsb_valid, VALIDATION_NUM_BATCHES, VALIDATION_BATCH_SIZE) </code></pre></div> Let's see examples from the dataset of two sentenses and their similarity. <div class="codehilite"><pre><code>for x, y in stsb_train: for i, example in enumerate(x): print(f"sentence 1 : {example[0]} ") print(f"sentence 2 : {example[1]} ") print(f"similarity : {y[i]} \n") break </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>sentence 1 : b"A young girl is sitting on Santa's lap." sentence 2 : b"A little girl is sitting on Santa's lap" similarity : [0.9200001] </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>sentence 1 : b'A women sitting at a table drinking with a basketball picture in the background.' sentence 2 : b'A woman in a sari drinks something while sitting at a table.' similarity : [0.03999996] </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>sentence 1 : b'Norway marks anniversary of massacre' sentence 2 : b"Norway Marks Anniversary of Breivik's Massacre" similarity : [0.52] </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>sentence 1 : b'US drone kills six militants in Pakistan: officials' sentence 2 : b'US missiles kill 15 in Pakistan: officials' similarity : [-0.03999996] </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>sentence 1 : b'On Tuesday, the central bank left interest rates steady, as expected, but also declared that overall risks were weighted toward weakness and warned of deflation risks.' sentence 2 : b"The central bank's policy board left rates steady for now, as widely expected, but surprised the market by declaring that overall risks were weighted toward weakness." similarity : [0.6] </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>sentence 1 : b'At one of the three sampling sites at Huntington Beach, the bacteria reading came back at 160 on June 16 and at 120 on June 23.' sentence 2 : b'The readings came back at 160 on June 16 and 120 at June 23 at one of three sampling sites at Huntington Beach.' similarity : [0.29999995] </code></pre></div> </div> <h4 id="build-the-encoder-model">Build the encoder model.</h4> Now, we'll build the encoder model that will produce the sentence embeddings. It consists of: <ul> <li>A preprocessor layer to tokenize and generate padding masks for the sentences.</li> <li>A backbone model that will generate the contextual representation of each token in the sentence.</li> <li>A mean pooling layer to produce the embeddings. We will use <a href="/api/layers/pooling_layers/global_average_pooling1d#globalaveragepooling1d-class"><code>keras.layers.GlobalAveragePooling1D</code></a> to apply the mean pooling to the backbone outputs. We will pass the padding mask to the layer to exclude padded tokens from being averaged.</li> <li>A normalization layer to normalize the embeddings as we are using the cosine similarity.</li> </ul> <div class="codehilite"><pre><code>preprocessor = keras_hub.models.RobertaPreprocessor.from_preset("roberta_base_en") backbone = keras_hub.models.RobertaBackbone.from_preset("roberta_base_en") inputs = keras.Input(shape=(1,), dtype="string", name="sentence") x = preprocessor(inputs) h = backbone(x) embedding = keras.layers.GlobalAveragePooling1D(name="pooling_layer")( h, x["padding_mask"] ) n_embedding = keras.layers.UnitNormalization(axis=1)(embedding) roberta_normal_encoder = keras.Model(inputs=inputs, outputs=n_embedding) roberta_normal_encoder.summary() </code></pre></div> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">Model: "functional_1" </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ Connected to ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ sentence │ (None, 1) │ 0 │ - │ │ (InputLayer) │ │ │ │ ├─────────────────────┼───────────────────┼─────────┼──────────────────────┤ │ roberta_preprocess… │ [(None, 512), │ 0 │ sentence[0][0] │ │ (RobertaPreprocess… │ (None, 512)] │ │ │ ├─────────────────────┼───────────────────┼─────────┼──────────────────────┤ │ roberta_backbone │ (None, 512, 768) │ 124,05… │ roberta_preprocesso… │ │ (RobertaBackbone) │ │ │ roberta_preprocesso… │ ├─────────────────────┼───────────────────┼─────────┼──────────────────────┤ │ pooling_layer │ (None, 768) │ 0 │ roberta_backbone[0]… │ │ (GlobalAveragePool… │ │ │ roberta_preprocesso… │ ├─────────────────────┼───────────────────┼─────────┼──────────────────────┤ │ unit_normalization │ (None, 768) │ 0 │ pooling_layer[0][0] │ │ (UnitNormalization) │ │ │ │ └─────────────────────┴───────────────────┴─────────┴──────────────────────┘ </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Total params: 124,052,736 (473.22 MB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Trainable params: 124,052,736 (473.22 MB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Non-trainable params: 0 (0.00 B) </pre> <h4 id="build-the-siamese-network-with-the-regression-objective-function">Build the Siamese network with the regression objective function.</h4> It's described above that the Siamese network has two or more subnetworks, and for this Siamese model, we need two encoders. But we don't have two encoders; we have only one encoder, but we will pass the two sentences through it. That way, we can have two paths to get the embeddings and also shared weights between the two paths. After passing the two sentences to the model and getting the normalized embeddings, we will multiply the two normalized embeddings to get the cosine similarity between the two sentences. <div class="codehilite"><pre><code>class RegressionSiamese(keras.Model): def __init__(self, encoder, **kwargs): inputs = keras.Input(shape=(2,), dtype="string", name="sentences") sen1, sen2 = keras.ops.split(inputs, 2, axis=1) u = encoder(sen1) v = encoder(sen2) cosine_similarity_scores = keras.ops.matmul(u, keras.ops.transpose(v)) super().__init__( inputs=inputs, outputs=cosine_similarity_scores, **kwargs, ) self.encoder = encoder def get_encoder(self): return self.encoder </code></pre></div> <h4 id="fit-the-model">Fit the model</h4> Let's try this example before training and compare it to the output after training. <div class="codehilite"><pre><code>sentences = [ "Today is a very sunny day.", "I am hungry, I will get my meal.", "The dog is eating his food.", ] query = ["The dog is enjoying his meal."] encoder = roberta_normal_encoder sentence_embeddings = encoder(tf.constant(sentences)) query_embedding = encoder(tf.constant(query)) cosine_similarity_scores = tf.matmul(query_embedding, tf.transpose(sentence_embeddings)) for i, sim in enumerate(cosine_similarity_scores[0]): print(f"cosine similarity score between sentence {i+1} and the query = {sim} ") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>cosine similarity score between sentence 1 and the query = 0.96630859375 cosine similarity score between sentence 2 and the query = 0.97607421875 cosine similarity score between sentence 3 and the query = 0.99365234375 </code></pre></div> </div> For the training we will use <code>MeanSquaredError()</code> as loss function, and <code>Adam()</code> optimizer with learning rate = 2e-5. <div class="codehilite"><pre><code>roberta_regression_siamese = RegressionSiamese(roberta_normal_encoder) roberta_regression_siamese.compile( loss=keras.losses.MeanSquaredError(), optimizer=keras.optimizers.Adam(2e-5), jit_compile=False, ) roberta_regression_siamese.fit(stsb_train, validation_data=stsb_valid, epochs=1) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code> 300/300 ━━━━━━━━━━━━━━━━━━━━ 115s 297ms/step - loss: 0.4751 - val_loss: 0.4025 <keras.src.callbacks.history.History at 0x7f5a78392140> </code></pre></div> </div> Let's try the model after training, we will notice a huge difference in the output. That means that the model after fine-tuning is capable of producing semantically meaningful embeddings. where the semantically similar sentences have a small angle between them. and semantically dissimilar sentences have a large angle between them. <div class="codehilite"><pre><code>sentences = [ "Today is a very sunny day.", "I am hungry, I will get my meal.", "The dog is eating his food.", ] query = ["The dog is enjoying his food."] encoder = roberta_regression_siamese.get_encoder() sentence_embeddings = encoder(tf.constant(sentences)) query_embedding = encoder(tf.constant(query)) cosine_simalarities = tf.matmul(query_embedding, tf.transpose(sentence_embeddings)) for i, sim in enumerate(cosine_simalarities[0]): print(f"cosine similarity between sentence {i+1} and the query = {sim} ") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>cosine similarity between sentence 1 and the query = 0.10986328125 cosine similarity between sentence 2 and the query = 0.53466796875 cosine similarity between sentence 3 and the query = 0.83544921875 </code></pre></div> </div> <h3 id="finetune-using-the-triplet-objective-function">Fine-tune Using the triplet Objective Function</h3> For the Siamese network with the triplet objective function, three sentences are passed to the Siamese network anchor, positive, and negative sentences. anchor and positive sentences are semantically similar, and anchor and negative sentences are semantically dissimilar. The objective is to minimize the distance between the anchor sentence and the positive sentence, and to maximize the distance between the anchor sentence and the negative sentence. <h4 id="load-the-dataset">Load the dataset</h4> We will use the Wikipedia-sections-triplets dataset for fine-tuning. This data set consists of sentences derived from the Wikipedia website. It has a collection of 3 sentences anchor, positive, negative. anchor and positive are derived from the same section. anchor and negative are derived from different sections. This dataset has 1.8 million training triplets and 220,000 test triplets. In this example, we will only use 1200 triplets for training and 300 for testing. <div class="codehilite"><pre><code>!wget https://sbert.net/datasets/wikipedia-sections-triplets.zip -q !unzip wikipedia-sections-triplets.zip -d wikipedia-sections-triplets </code></pre></div> <div class="codehilite"><pre><code>NUM_TRAIN_BATCHES = 200 NUM_TEST_BATCHES = 75 AUTOTUNE = tf.data.experimental.AUTOTUNE def prepare_wiki_data(dataset, num_batches): dataset = dataset.map( lambda z: ((z["Sentence1"], z["Sentence2"], z["Sentence3"]), 0) ) dataset = dataset.batch(6) dataset = dataset.take(num_batches) dataset = dataset.prefetch(AUTOTUNE) return dataset wiki_train = tf.data.experimental.make_csv_dataset( "wikipedia-sections-triplets/train.csv", batch_size=1, num_epochs=1, ) wiki_test = tf.data.experimental.make_csv_dataset( "wikipedia-sections-triplets/test.csv", batch_size=1, num_epochs=1, ) wiki_train = prepare_wiki_data(wiki_train, NUM_TRAIN_BATCHES) wiki_test = prepare_wiki_data(wiki_test, NUM_TEST_BATCHES) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Archive: wikipedia-sections-triplets.zip inflating: wikipedia-sections-triplets/validation.csv inflating: wikipedia-sections-triplets/Readme.txt inflating: wikipedia-sections-triplets/test.csv inflating: wikipedia-sections-triplets/train.csv </code></pre></div> </div> <h4 id="build-the-encoder-model">Build the encoder model</h4> For this encoder model, we will use RoBERTa with mean pooling and we will not normalize the output embeddings. The encoder model consists of: <ul> <li>A preprocessor layer to tokenize and generate padding masks for the sentences.</li> <li>A backbone model that will generate the contextual representation of each token in the sentence.</li> <li>A mean pooling layer to produce the embeddings.</li> </ul> <div class="codehilite"><pre><code>preprocessor = keras_hub.models.RobertaPreprocessor.from_preset("roberta_base_en") backbone = keras_hub.models.RobertaBackbone.from_preset("roberta_base_en") input = keras.Input(shape=(1,), dtype="string", name="sentence") x = preprocessor(input) h = backbone(x) embedding = keras.layers.GlobalAveragePooling1D(name="pooling_layer")( h, x["padding_mask"] ) roberta_encoder = keras.Model(inputs=input, outputs=embedding) roberta_encoder.summary() </code></pre></div> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">Model: "functional_3" </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ Connected to ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ sentence │ (None, 1) │ 0 │ - │ │ (InputLayer) │ │ │ │ ├─────────────────────┼───────────────────┼─────────┼──────────────────────┤ │ roberta_preprocess… │ [(None, 512), │ 0 │ sentence[0][0] │ │ (RobertaPreprocess… │ (None, 512)] │ │ │ ├─────────────────────┼───────────────────┼─────────┼──────────────────────┤ │ roberta_backbone_1 │ (None, 512, 768) │ 124,05… │ roberta_preprocesso… │ │ (RobertaBackbone) │ │ │ roberta_preprocesso… │ ├─────────────────────┼───────────────────┼─────────┼──────────────────────┤ │ pooling_layer │ (None, 768) │ 0 │ roberta_backbone_1[… │ │ (GlobalAveragePool… │ │ │ roberta_preprocesso… │ └─────────────────────┴───────────────────┴─────────┴──────────────────────┘ </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Total params: 124,052,736 (473.22 MB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Trainable params: 124,052,736 (473.22 MB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Non-trainable params: 0 (0.00 B) </pre> <h4 id="build-the-siamese-network-with-the-triplet-objective-function">Build the Siamese network with the triplet objective function</h4> For the Siamese network with the triplet objective function, we will build the model with an encoder, and we will pass the three sentences through that encoder. We will get an embedding for each sentence, and we will calculate the <code>positive_dist</code> and <code>negative_dist</code> that will be passed to the loss function described below. <div class="codehilite"><pre><code>class TripletSiamese(keras.Model): def __init__(self, encoder, **kwargs): anchor = keras.Input(shape=(1,), dtype="string") positive = keras.Input(shape=(1,), dtype="string") negative = keras.Input(shape=(1,), dtype="string") ea = encoder(anchor) ep = encoder(positive) en = encoder(negative) positive_dist = keras.ops.sum(keras.ops.square(ea - ep), axis=1) negative_dist = keras.ops.sum(keras.ops.square(ea - en), axis=1) positive_dist = keras.ops.sqrt(positive_dist) negative_dist = keras.ops.sqrt(negative_dist) output = keras.ops.stack([positive_dist, negative_dist], axis=0) super().__init__(inputs=[anchor, positive, negative], outputs=output, **kwargs) self.encoder = encoder def get_encoder(self): return self.encoder </code></pre></div> We will use a custom loss function for the triplet objective. The loss function will receive the distance between the anchor and the positive embeddings <code>positive_dist</code>, and the distance between the anchor and the negative embeddings <code>negative_dist</code>, where they are stacked together in <code>y_pred</code>. We will use <code>positive_dist</code> and <code>negative_dist</code> to compute the loss such that <code>negative_dist</code> is larger than <code>positive_dist</code> at least by a specific margin. Mathematically, we will minimize this loss function: <code>max( positive_dist - negative_dist + margin, 0)</code>. There is no <code>y_true</code> used in this loss function. Note that we set the labels in the dataset to zero, but they will not be used. <div class="codehilite"><pre><code>class TripletLoss(keras.losses.Loss): def __init__(self, margin=1, **kwargs): super().__init__(**kwargs) self.margin = margin def call(self, y_true, y_pred): positive_dist, negative_dist = tf.unstack(y_pred, axis=0) losses = keras.ops.relu(positive_dist - negative_dist + self.margin) return keras.ops.mean(losses, axis=0) </code></pre></div> <h4 id="fit-the-model">Fit the model</h4> For the training, we will use the custom <code>TripletLoss()</code> loss function, and <code>Adam()</code> optimizer with a learning rate = 2e-5. <div class="codehilite"><pre><code>roberta_triplet_siamese = TripletSiamese(roberta_encoder) roberta_triplet_siamese.compile( loss=TripletLoss(), optimizer=keras.optimizers.Adam(2e-5), jit_compile=False, ) roberta_triplet_siamese.fit(wiki_train, validation_data=wiki_test, epochs=1) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code> 200/200 ━━━━━━━━━━━━━━━━━━━━ 128s 467ms/step - loss: 0.7822 - val_loss: 0.7126 <keras.src.callbacks.history.History at 0x7f5c3636c580> </code></pre></div> </div> Let's try this model in a clustering example. Here are 6 questions. first 3 questions about learning English, and the last 3 questions about working online. Let's see if the embeddings produced by our encoder will cluster them correctly. <div class="codehilite"><pre><code>questions = [ "What should I do to improve my English writting?", "How to be good at speaking English?", "How can I improve my English?", "How to earn money online?", "How do I earn money online?", "How to work and earn money through internet?", ] encoder = roberta_triplet_siamese.get_encoder() embeddings = encoder(tf.constant(questions)) kmeans = cluster.KMeans(n_clusters=2, random_state=0, n_init="auto").fit(embeddings) for i, label in enumerate(kmeans.labels_): print(f"sentence ({questions[i]}) belongs to cluster {label}") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>sentence (What should I do to improve my English writting?) belongs to cluster 1 sentence (How to be good at speaking English?) belongs to cluster 1 sentence (How can I improve my English?) belongs to cluster 1 sentence (How to earn money online?) belongs to cluster 0 sentence (How do I earn money online?) belongs to cluster 0 sentence (How to work and earn money through internet?) belongs to cluster 0 </code></pre></div> </div> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#sentence-embeddings-using-siamese-robertanetworks'>Sentence embeddings using Siamese RoBERTa-networks</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#setup'>Setup</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#finetune-the-model-using-siamese-networks'>Fine-tune the model using siamese networks</a> </div> <div class='k-outline-depth-3'> <a href='#finetune-using-the-regression-objective-function'>Fine-tune using the regression objective function</a> </div> <div class='k-outline-depth-3'> <a href='#finetune-using-the-triplet-objective-function'>Fine-tune Using the triplet Objective Function</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>