CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="Keras documentation"> <meta name="author" content="Keras Team"> <link rel="shortcut icon" href="https://keras.io/img/favicon.ico"> <link rel="canonical" href="https://keras.io/examples/nlp/neural_machine_translation_with_keras_hub/" />  <meta property="og:title" content="Keras documentation: English-to-Spanish translation with KerasHub"> <meta property="og:image" content="https://keras.io/img/logo-k-keras-wb.png"> <meta name="twitter:title" content="Keras documentation: English-to-Spanish translation with KerasHub"> <meta name="twitter:image" content="https://keras.io/img/k-keras-social.png"> <meta name="twitter:card" content="summary"> <title>English-to-Spanish translation with KerasHub</title>  <link href="/css/bootstrap.min.css" rel="stylesheet">  <link href="https://fonts.googleapis.com/css2?family=Open+Sans:wght@400;600;700;800&display=swap" rel="stylesheet">  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/monokai.css" rel="stylesheet">  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-5DNGF4N'); </script> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-175165319-128', 'auto'); ga('send', 'pageview'); </script>  <script async defer src="https://buttons.github.io/buttons.js"></script> </head> <body>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-5DNGF4N" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  <div class='k-page'> <div class="k-nav" id="nav-menu"> <a href='/'><img src='/img/logo-small.png' class='logo-small' /></a> <div class="nav flex-column nav-pills" role="tablist" aria-orientation="vertical"> <a class="nav-link" href="/about/" role="tab" aria-selected="">About Keras</a> <a class="nav-link" href="/getting_started/" role="tab" aria-selected="">Getting started</a> <a class="nav-link" href="/guides/" role="tab" aria-selected="">Developer guides</a> <a class="nav-link active" href="/examples/" role="tab" aria-selected="">Code examples</a> <a class="nav-sublink" href="/examples/vision/">Computer Vision</a> <a class="nav-sublink active" href="/examples/nlp/">Natural Language Processing</a> <a class="nav-sublink2" href="/examples/nlp/text_classification_from_scratch/">Text classification from scratch</a> <a class="nav-sublink2" href="/examples/nlp/active_learning_review_classification/">Review Classification using Active Learning</a> <a class="nav-sublink2" href="/examples/nlp/fnet_classification_with_keras_hub/">Text Classification using FNet</a> <a class="nav-sublink2" href="/examples/nlp/multi_label_classification/">Large-scale multi-label text classification</a> <a class="nav-sublink2" href="/examples/nlp/text_classification_with_transformer/">Text classification with Transformer</a> <a class="nav-sublink2" href="/examples/nlp/text_classification_with_switch_transformer/">Text classification with Switch Transformer</a> <a class="nav-sublink2" href="/examples/nlp/tweet-classification-using-tfdf/">Text classification using Decision Forests and pretrained embeddings</a> <a class="nav-sublink2" href="/examples/nlp/pretrained_word_embeddings/">Using pre-trained word embeddings</a> <a class="nav-sublink2" href="/examples/nlp/bidirectional_lstm_imdb/">Bidirectional LSTM on IMDB</a> <a class="nav-sublink2" href="/examples/nlp/data_parallel_training_with_keras_hub/">Data Parallel Training with KerasHub and tf.distribute</a> <a class="nav-sublink2 active" href="/examples/nlp/neural_machine_translation_with_keras_hub/">English-to-Spanish translation with KerasHub</a> <a class="nav-sublink2" href="/examples/nlp/neural_machine_translation_with_transformer/">English-to-Spanish translation with a sequence-to-sequence Transformer</a> <a class="nav-sublink2" href="/examples/nlp/lstm_seq2seq/">Character-level recurrent sequence-to-sequence model</a> <a class="nav-sublink2" href="/examples/nlp/multimodal_entailment/">Multimodal entailment</a> <a class="nav-sublink2" href="/examples/nlp/ner_transformers/">Named Entity Recognition using Transformers</a> <a class="nav-sublink2" href="/examples/nlp/text_extraction_with_bert/">Text Extraction with BERT</a> <a class="nav-sublink2" href="/examples/nlp/addition_rnn/">Sequence to sequence learning for performing number addition</a> <a class="nav-sublink2" href="/examples/nlp/semantic_similarity_with_keras_hub/">Semantic Similarity with KerasHub</a> <a class="nav-sublink2" href="/examples/nlp/semantic_similarity_with_bert/">Semantic Similarity with BERT</a> <a class="nav-sublink2" href="/examples/nlp/sentence_embeddings_with_sbert/">Sentence embeddings using Siamese RoBERTa-networks</a> <a class="nav-sublink2" href="/examples/nlp/masked_language_modeling/">End-to-end Masked Language Modeling with BERT</a> <a class="nav-sublink2" href="/examples/nlp/abstractive_summarization_with_bart/">Abstractive Text Summarization with BART</a> <a class="nav-sublink2" href="/examples/nlp/pretraining_BERT/">Pretraining BERT with Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/nlp/parameter_efficient_finetuning_of_gpt2_with_lora/">Parameter-efficient fine-tuning of GPT-2 with LoRA</a> <a class="nav-sublink2" href="/examples/nlp/mlm_training_tpus/">Training a language model from scratch with 🤗 Transformers and TPUs</a> <a class="nav-sublink2" href="/examples/nlp/multiple_choice_task_with_transfer_learning/">MultipleChoice Task with Transfer Learning</a> <a class="nav-sublink2" href="/examples/nlp/question_answering/">Question Answering with Hugging Face Transformers</a> <a class="nav-sublink2" href="/examples/nlp/t5_hf_summarization/">Abstractive Summarization with Hugging Face Transformers</a> <a class="nav-sublink" href="/examples/structured_data/">Structured Data</a> <a class="nav-sublink" href="/examples/timeseries/">Timeseries</a> <a class="nav-sublink" href="/examples/generative/">Generative Deep Learning</a> <a class="nav-sublink" href="/examples/audio/">Audio Data</a> <a class="nav-sublink" href="/examples/rl/">Reinforcement Learning</a> <a class="nav-sublink" href="/examples/graph/">Graph Data</a> <a class="nav-sublink" href="/examples/keras_recipes/">Quick Keras Recipes</a> <a class="nav-link" href="/api/" role="tab" aria-selected="">Keras 3 API documentation</a> <a class="nav-link" href="/2.18/api/" role="tab" aria-selected="">Keras 2 API documentation</a> <a class="nav-link" href="/keras_tuner/" role="tab" aria-selected="">KerasTuner: Hyperparam Tuning</a> <a class="nav-link" href="/keras_hub/" role="tab" aria-selected="">KerasHub: Pretrained Models</a> </div> </div> <div class='k-main'> <div class='k-main-top'> <script> function displayDropdownMenu() { e = document.getElementById("nav-menu"); if (e.style.display == "block") { e.style.display = "none"; } else { e.style.display = "block"; document.getElementById("dropdown-nav").style.display = "block"; } } function resetMobileUI() { if (window.innerWidth <= 840) { document.getElementById("nav-menu").style.display = "none"; document.getElementById("dropdown-nav").style.display = "block"; } else { document.getElementById("nav-menu").style.display = "block"; document.getElementById("dropdown-nav").style.display = "none"; } var navmenu = document.getElementById("nav-menu"); var menuheight = navmenu.clientHeight; var kmain = document.getElementById("k-main-id"); kmain.style.minHeight = (menuheight + 100) + 'px'; } window.onresize = resetMobileUI; window.addEventListener("load", (event) => { resetMobileUI() }); </script> <div id='dropdown-nav' onclick="displayDropdownMenu();"> <svg viewBox="-20 -20 120 120" width="60" height="60"> <rect width="100" height="20"></rect> <rect y="30" width="100" height="20"></rect> <rect y="60" width="100" height="20"></rect> </svg> </div> <form class="bd-search d-flex align-items-center k-search-form" id="search-form"> <input type="search" class="k-search-input" id="search-input" placeholder="Search Keras documentation..." aria-label="Search Keras documentation..." autocomplete="off"> <button class="k-search-btn"> <svg width="13" height="13" viewBox="0 0 13 13"><title>search</title><path d="m4.8495 7.8226c0.82666 0 1.5262-0.29146 2.0985-0.87438 0.57232-0.58292 0.86378-1.2877 0.87438-2.1144 0.010599-0.82666-0.28086-1.5262-0.87438-2.0985-0.59352-0.57232-1.293-0.86378-2.0985-0.87438-0.8055-0.010599-1.5103 0.28086-2.1144 0.87438-0.60414 0.59352-0.8956 1.293-0.87438 2.0985 0.021197 0.8055 0.31266 1.5103 0.87438 2.1144 0.56172 0.60414 1.2665 0.8956 2.1144 0.87438zm4.4695 0.2115 3.681 3.6819-1.259 1.284-3.6817-3.7 0.0019784-0.69479-0.090043-0.098846c-0.87973 0.76087-1.92 1.1413-3.1207 1.1413-1.3553 0-2.5025-0.46363-3.4417-1.3909s-1.4088-2.0686-1.4088-3.4239c0-1.3553 0.4696-2.4966 1.4088-3.4239 0.9392-0.92727 2.0864-1.3969 3.4417-1.4088 1.3553-0.011889 2.4906 0.45771 3.406 1.4088 0.9154 0.95107 1.379 2.0924 1.3909 3.4239 0 1.2126-0.38043 2.2588-1.1413 3.1385l0.098834 0.090049z"></path></svg> </button> </form> <script> var form = document.getElementById('search-form'); form.onsubmit = function(e) { e.preventDefault(); var query = document.getElementById('search-input').value; window.location.href = '/search.html?query=' + query; return False } </script> </div> <div class='k-main-inner' id='k-main-id'> <div class='k-location-slug'> ► <a href='/examples/'>Code examples</a> / <a href='/examples/nlp/'>Natural Language Processing</a> / English-to-Spanish translation with KerasHub </div> <div class='k-content'> <h1 id="englishtospanish-translation-with-kerashub">English-to-Spanish translation with KerasHub</h1> Author: <a href="https://github.com/abheesht17/">Abheesht Sharma</a> Date created: 2022/05/26 Last modified: 2024/04/30 Description: Use KerasHub to train a sequence-to-sequence Transformer model on the machine translation task. <div class='example_version_banner keras_3'>ⓘ This example uses Keras 3</div> <img class="k-inline-icon" src="https://colab.research.google.com/img/colab_favicon.ico"/> <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/nlp/ipynb/neural_machine_translation_with_keras_hub.ipynb">View in Colab</a> •<img class="k-inline-icon" src="https://github.com/favicon.ico"/> <a href="https://github.com/keras-team/keras-io/blob/master/examples/nlp/neural_machine_translation_with_keras_hub.py">GitHub source</a> <hr /> <h2 id="introduction">Introduction</h2> KerasHub provides building blocks for NLP (model layers, tokenizers, metrics, etc.) and makes it convenient to construct NLP pipelines. In this example, we'll use KerasHub layers to build an encoder-decoder Transformer model, and train it on the English-to-Spanish machine translation task. This example is based on the <a href="https://keras.io/examples/nlp/neural_machine_translation_with_transformer/">English-to-Spanish NMT example</a> by <a href="https://twitter.com/fchollet">fchollet</a>. The original example is more low-level and implements layers from scratch, whereas this example uses KerasHub to show some more advanced approaches, such as subword tokenization and using metrics to compute the quality of generated translations. You'll learn how to: <ul> <li>Tokenize text using <a href="/keras_hub/api/tokenizers/word_piece_tokenizer#wordpiecetokenizer-class"><code>keras_hub.tokenizers.WordPieceTokenizer</code></a>.</li> <li>Implement a sequence-to-sequence Transformer model using KerasHub's <a href="/keras_hub/api/modeling_layers/transformer_encoder#transformerencoder-class"><code>keras_hub.layers.TransformerEncoder</code></a>, <a href="/keras_hub/api/modeling_layers/transformer_decoder#transformerdecoder-class"><code>keras_hub.layers.TransformerDecoder</code></a> and <a href="/keras_hub/api/modeling_layers/token_and_position_embedding#tokenandpositionembedding-class"><code>keras_hub.layers.TokenAndPositionEmbedding</code></a> layers, and train it.</li> <li>Use <code>keras_hub.samplers</code> to generate translations of unseen input sentences using the top-p decoding strategy!</li> </ul> Don't worry if you aren't familiar with KerasHub. This tutorial will start with the basics. Let's dive right in! <hr /> <h2 id="setup">Setup</h2> Before we start implementing the pipeline, let's import all the libraries we need. <div class="codehilite"><pre><code>!pip install -q --upgrade rouge-score !pip install -q --upgrade keras-hub !pip install -q --upgrade keras # Upgrade to Keras 3. </code></pre></div> <div class="codehilite"><pre><code>import keras_hub import pathlib import random import keras from keras import ops import tensorflow.data as tf_data from tensorflow_text.tools.wordpiece_vocab import ( bert_vocab_from_dataset as bert_vocab, ) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tensorflow 2.15.1 requires keras<2.16,>=2.15.0, but you have keras 3.3.3 which is incompatible.[31m </code></pre></div> </div> Let's also define our parameters/hyperparameters. <div class="codehilite"><pre><code>BATCH_SIZE = 64 EPOCHS = 1 # This should be at least 10 for convergence MAX_SEQUENCE_LENGTH = 40 ENG_VOCAB_SIZE = 15000 SPA_VOCAB_SIZE = 15000 EMBED_DIM = 256 INTERMEDIATE_DIM = 2048 NUM_HEADS = 8 </code></pre></div> <hr /> <h2 id="downloading-the-data">Downloading the data</h2> We'll be working with an English-to-Spanish translation dataset provided by <a href="https://www.manythings.org/anki/">Anki</a>. Let's download it: <div class="codehilite"><pre><code>text_file = keras.utils.get_file( fname="spa-eng.zip", origin="http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip", extract=True, ) text_file = pathlib.Path(text_file).parent / "spa-eng" / "spa.txt" </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Downloading data from http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip 2638744/2638744 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step </code></pre></div> </div> <hr /> <h2 id="parsing-the-data">Parsing the data</h2> Each line contains an English sentence and its corresponding Spanish sentence. The English sentence is the source sequence and Spanish one is the target sequence. Before adding the text to a list, we convert it to lowercase. <div class="codehilite"><pre><code>with open(text_file) as f: lines = f.read().split("\n")[:-1] text_pairs = [] for line in lines: eng, spa = line.split("\t") eng = eng.lower() spa = spa.lower() text_pairs.append((eng, spa)) </code></pre></div> Here's what our sentence pairs look like: <div class="codehilite"><pre><code>for _ in range(5): print(random.choice(text_pairs)) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>('tom heard that mary had bought a new computer.', 'tom oyó que mary se había comprado un computador nuevo.') ('will you stay at home?', '¿te vas a quedar en casa?') ('where is this train going?', '¿adónde va este tren?') ('tom panicked.', 'tom entró en pánico.') ("we'll help you rescue tom.", 'te ayudaremos a rescatar a tom.') </code></pre></div> </div> Now, let's split the sentence pairs into a training set, a validation set, and a test set. <div class="codehilite"><pre><code>random.shuffle(text_pairs) num_val_samples = int(0.15 * len(text_pairs)) num_train_samples = len(text_pairs) - 2 * num_val_samples train_pairs = text_pairs[:num_train_samples] val_pairs = text_pairs[num_train_samples : num_train_samples + num_val_samples] test_pairs = text_pairs[num_train_samples + num_val_samples :] print(f"{len(text_pairs)} total pairs") print(f"{len(train_pairs)} training pairs") print(f"{len(val_pairs)} validation pairs") print(f"{len(test_pairs)} test pairs") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>118964 total pairs 83276 training pairs 17844 validation pairs 17844 test pairs </code></pre></div> </div> <hr /> <h2 id="tokenizing-the-data">Tokenizing the data</h2> We'll define two tokenizers - one for the source language (English), and the other for the target language (Spanish). We'll be using <a href="/keras_hub/api/tokenizers/word_piece_tokenizer#wordpiecetokenizer-class"><code>keras_hub.tokenizers.WordPieceTokenizer</code></a> to tokenize the text. <a href="/keras_hub/api/tokenizers/word_piece_tokenizer#wordpiecetokenizer-class"><code>keras_hub.tokenizers.WordPieceTokenizer</code></a> takes a WordPiece vocabulary and has functions for tokenizing the text, and detokenizing sequences of tokens. Before we define the two tokenizers, we first need to train them on the dataset we have. The WordPiece tokenization algorithm is a subword tokenization algorithm; training it on a corpus gives us a vocabulary of subwords. A subword tokenizer is a compromise between word tokenizers (word tokenizers need very large vocabularies for good coverage of input words), and character tokenizers (characters don't really encode meaning like words do). Luckily, KerasHub makes it very simple to train WordPiece on a corpus with the <a href="/keras_hub/api/tokenizers/compute_word_piece_vocabulary#computewordpiecevocabulary-function"><code>keras_hub.tokenizers.compute_word_piece_vocabulary</code></a> utility. <div class="codehilite"><pre><code>def train_word_piece(text_samples, vocab_size, reserved_tokens): word_piece_ds = tf_data.Dataset.from_tensor_slices(text_samples) vocab = keras_hub.tokenizers.compute_word_piece_vocabulary( word_piece_ds.batch(1000).prefetch(2), vocabulary_size=vocab_size, reserved_tokens=reserved_tokens, ) return vocab </code></pre></div> Every vocabulary has a few special, reserved tokens. We have four such tokens: <ul> <li><code>"[PAD]"</code> - Padding token. Padding tokens are appended to the input sequence length when the input sequence length is shorter than the maximum sequence length.</li> <li><code>"[UNK]"</code> - Unknown token.</li> <li><code>"[START]"</code> - Token that marks the start of the input sequence.</li> <li><code>"[END]"</code> - Token that marks the end of the input sequence.</li> </ul> <div class="codehilite"><pre><code>reserved_tokens = ["[PAD]", "[UNK]", "[START]", "[END]"] eng_samples = [text_pair[0] for text_pair in train_pairs] eng_vocab = train_word_piece(eng_samples, ENG_VOCAB_SIZE, reserved_tokens) spa_samples = [text_pair[1] for text_pair in train_pairs] spa_vocab = train_word_piece(spa_samples, SPA_VOCAB_SIZE, reserved_tokens) </code></pre></div> Let's see some tokens! <div class="codehilite"><pre><code>print("English Tokens: ", eng_vocab[100:110]) print("Spanish Tokens: ", spa_vocab[100:110]) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>English Tokens: ['at', 'know', 'him', 'there', 'go', 'they', 'her', 'has', 'time', 'will'] Spanish Tokens: ['le', 'para', 'te', 'mary', 'las', 'más', 'al', 'yo', 'tu', 'estoy'] </code></pre></div> </div> Now, let's define the tokenizers. We will configure the tokenizers with the the vocabularies trained above. <div class="codehilite"><pre><code>eng_tokenizer = keras_hub.tokenizers.WordPieceTokenizer( vocabulary=eng_vocab, lowercase=False ) spa_tokenizer = keras_hub.tokenizers.WordPieceTokenizer( vocabulary=spa_vocab, lowercase=False ) </code></pre></div> Let's try and tokenize a sample from our dataset! To verify whether the text has been tokenized correctly, we can also detokenize the list of tokens back to the original text. <div class="codehilite"><pre><code>eng_input_ex = text_pairs[0][0] eng_tokens_ex = eng_tokenizer.tokenize(eng_input_ex) print("English sentence: ", eng_input_ex) print("Tokens: ", eng_tokens_ex) print( "Recovered text after detokenizing: ", eng_tokenizer.detokenize(eng_tokens_ex), ) print() spa_input_ex = text_pairs[0][1] spa_tokens_ex = spa_tokenizer.tokenize(spa_input_ex) print("Spanish sentence: ", spa_input_ex) print("Tokens: ", spa_tokens_ex) print( "Recovered text after detokenizing: ", spa_tokenizer.detokenize(spa_tokens_ex), ) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>English sentence: i am leaving the books here. Tokens: tf.Tensor([ 35 163 931 66 356 119 12], shape=(7,), dtype=int32) Recovered text after detokenizing: tf.Tensor(b'i am leaving the books here .', shape=(), dtype=string) </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>Spanish sentence: dejo los libros aquí. Tokens: tf.Tensor([2962 93 350 122 14], shape=(5,), dtype=int32) Recovered text after detokenizing: tf.Tensor(b'dejo los libros aqu\xc3\xad .', shape=(), dtype=string) </code></pre></div> </div> <hr /> <h2 id="format-datasets">Format datasets</h2> Next, we'll format our datasets. At each training step, the model will seek to predict target words N+1 (and beyond) using the source sentence and the target words 0 to N. As such, the training dataset will yield a tuple <code>(inputs, targets)</code>, where: <ul> <li><code>inputs</code> is a dictionary with the keys <code>encoder_inputs</code> and <code>decoder_inputs</code>. <code>encoder_inputs</code> is the tokenized source sentence and <code>decoder_inputs</code> is the target sentence "so far", that is to say, the words 0 to N used to predict word N+1 (and beyond) in the target sentence.</li> <li><code>target</code> is the target sentence offset by one step: it provides the next words in the target sentence – what the model will try to predict.</li> </ul> We will add special tokens, <code>"[START]"</code> and <code>"[END]"</code>, to the input Spanish sentence after tokenizing the text. We will also pad the input to a fixed length. This can be easily done using <a href="/keras_hub/api/preprocessing_layers/start_end_packer#startendpacker-class"><code>keras_hub.layers.StartEndPacker</code></a>. <div class="codehilite"><pre><code>def preprocess_batch(eng, spa): batch_size = ops.shape(spa)[0] eng = eng_tokenizer(eng) spa = spa_tokenizer(spa) # Pad `eng` to `MAX_SEQUENCE_LENGTH`. eng_start_end_packer = keras_hub.layers.StartEndPacker( sequence_length=MAX_SEQUENCE_LENGTH, pad_value=eng_tokenizer.token_to_id("[PAD]"), ) eng = eng_start_end_packer(eng) # Add special tokens (`"[START]"` and `"[END]"`) to `spa` and pad it as well. spa_start_end_packer = keras_hub.layers.StartEndPacker( sequence_length=MAX_SEQUENCE_LENGTH + 1, start_value=spa_tokenizer.token_to_id("[START]"), end_value=spa_tokenizer.token_to_id("[END]"), pad_value=spa_tokenizer.token_to_id("[PAD]"), ) spa = spa_start_end_packer(spa) return ( { "encoder_inputs": eng, "decoder_inputs": spa[:, :-1], }, spa[:, 1:], ) def make_dataset(pairs): eng_texts, spa_texts = zip(*pairs) eng_texts = list(eng_texts) spa_texts = list(spa_texts) dataset = tf_data.Dataset.from_tensor_slices((eng_texts, spa_texts)) dataset = dataset.batch(BATCH_SIZE) dataset = dataset.map(preprocess_batch, num_parallel_calls=tf_data.AUTOTUNE) return dataset.shuffle(2048).prefetch(16).cache() train_ds = make_dataset(train_pairs) val_ds = make_dataset(val_pairs) </code></pre></div> Let's take a quick look at the sequence shapes (we have batches of 64 pairs, and all sequences are 40 steps long): <div class="codehilite"><pre><code>for inputs, targets in train_ds.take(1): print(f'inputs["encoder_inputs"].shape: {inputs["encoder_inputs"].shape}') print(f'inputs["decoder_inputs"].shape: {inputs["decoder_inputs"].shape}') print(f"targets.shape: {targets.shape}") </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>inputs["encoder_inputs"].shape: (64, 40) inputs["decoder_inputs"].shape: (64, 40) targets.shape: (64, 40) </code></pre></div> </div> <hr /> <h2 id="building-the-model">Building the model</h2> Now, let's move on to the exciting part - defining our model! We first need an embedding layer, i.e., a vector for every token in our input sequence. This embedding layer can be initialised randomly. We also need a positional embedding layer which encodes the word order in the sequence. The convention is to add these two embeddings. KerasHub has a <code>keras_hub.layers.TokenAndPositionEmbedding</code> layer which does all of the above steps for us. Our sequence-to-sequence Transformer consists of a <a href="/keras_hub/api/modeling_layers/transformer_encoder#transformerencoder-class"><code>keras_hub.layers.TransformerEncoder</code></a> layer and a <a href="/keras_hub/api/modeling_layers/transformer_decoder#transformerdecoder-class"><code>keras_hub.layers.TransformerDecoder</code></a> layer chained together. The source sequence will be passed to <a href="/keras_hub/api/modeling_layers/transformer_encoder#transformerencoder-class"><code>keras_hub.layers.TransformerEncoder</code></a>, which will produce a new representation of it. This new representation will then be passed to the <a href="/keras_hub/api/modeling_layers/transformer_decoder#transformerdecoder-class"><code>keras_hub.layers.TransformerDecoder</code></a>, together with the target sequence so far (target words 0 to N). The <a href="/keras_hub/api/modeling_layers/transformer_decoder#transformerdecoder-class"><code>keras_hub.layers.TransformerDecoder</code></a> will then seek to predict the next words in the target sequence (N+1 and beyond). A key detail that makes this possible is causal masking. The <a href="/keras_hub/api/modeling_layers/transformer_decoder#transformerdecoder-class"><code>keras_hub.layers.TransformerDecoder</code></a> sees the entire sequence at once, and thus we must make sure that it only uses information from target tokens 0 to N when predicting token N+1 (otherwise, it could use information from the future, which would result in a model that cannot be used at inference time). Causal masking is enabled by default in <a href="/keras_hub/api/modeling_layers/transformer_decoder#transformerdecoder-class"><code>keras_hub.layers.TransformerDecoder</code></a>. We also need to mask the padding tokens (<code>"[PAD]"</code>). For this, we can set the <code>mask_zero</code> argument of the <a href="/keras_hub/api/modeling_layers/token_and_position_embedding#tokenandpositionembedding-class"><code>keras_hub.layers.TokenAndPositionEmbedding</code></a> layer to True. This will then be propagated to all subsequent layers. <div class="codehilite"><pre><code># Encoder encoder_inputs = keras.Input(shape=(None,), name="encoder_inputs") x = keras_hub.layers.TokenAndPositionEmbedding( vocabulary_size=ENG_VOCAB_SIZE, sequence_length=MAX_SEQUENCE_LENGTH, embedding_dim=EMBED_DIM, )(encoder_inputs) encoder_outputs = keras_hub.layers.TransformerEncoder( intermediate_dim=INTERMEDIATE_DIM, num_heads=NUM_HEADS )(inputs=x) encoder = keras.Model(encoder_inputs, encoder_outputs) # Decoder decoder_inputs = keras.Input(shape=(None,), name="decoder_inputs") encoded_seq_inputs = keras.Input(shape=(None, EMBED_DIM), name="decoder_state_inputs") x = keras_hub.layers.TokenAndPositionEmbedding( vocabulary_size=SPA_VOCAB_SIZE, sequence_length=MAX_SEQUENCE_LENGTH, embedding_dim=EMBED_DIM, )(decoder_inputs) x = keras_hub.layers.TransformerDecoder( intermediate_dim=INTERMEDIATE_DIM, num_heads=NUM_HEADS )(decoder_sequence=x, encoder_sequence=encoded_seq_inputs) x = keras.layers.Dropout(0.5)(x) decoder_outputs = keras.layers.Dense(SPA_VOCAB_SIZE, activation="softmax")(x) decoder = keras.Model( [ decoder_inputs, encoded_seq_inputs, ], decoder_outputs, ) decoder_outputs = decoder([decoder_inputs, encoder_outputs]) transformer = keras.Model( [encoder_inputs, decoder_inputs], decoder_outputs, name="transformer", ) </code></pre></div> <hr /> <h2 id="training-our-model">Training our model</h2> We'll use accuracy as a quick way to monitor training progress on the validation data. Note that machine translation typically uses BLEU scores as well as other metrics, rather than accuracy. However, in order to use metrics like ROUGE, BLEU, etc. we will have decode the probabilities and generate the text. Text generation is computationally expensive, and performing this during training is not recommended. Here we only train for 1 epoch, but to get the model to actually converge you should train for at least 10 epochs. <div class="codehilite"><pre><code>transformer.summary() transformer.compile( "rmsprop", loss="sparse_categorical_crossentropy", metrics=["accuracy"] ) transformer.fit(train_ds, epochs=EPOCHS, validation_data=val_ds) </code></pre></div> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">Model: "transformer" </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ Connected to ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ encoder_inputs │ (None, None) │ 0 │ - │ │ (InputLayer) │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ token_and_position… │ (None, None, 256) │ 3,850,240 │ encoder_inputs[0… │ │ (TokenAndPositionE… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ decoder_inputs │ (None, None) │ 0 │ - │ │ (InputLayer) │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ transformer_encoder │ (None, None, 256) │ 1,315,072 │ token_and_positi… │ │ (TransformerEncode… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ functional_3 │ (None, None, │ 9,283,992 │ decoder_inputs[0… │ │ (Functional) │ 15000) │ │ transformer_enco… │ └─────────────────────┴───────────────────┴────────────┴───────────────────┘ </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Total params: 14,449,304 (55.12 MB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Trainable params: 14,449,304 (55.12 MB) </pre> <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"> Non-trainable params: 0 (0.00 B) </pre> <div class="k-default-codeblock"> <div class="codehilite"><pre><code> 1302/1302 ━━━━━━━━━━━━━━━━━━━━ 1701s 1s/step - accuracy: 0.8168 - loss: 1.4819 - val_accuracy: 0.8650 - val_loss: 0.8129 <keras.src.callbacks.history.History at 0x7efdd7ee6a50> </code></pre></div> </div> <hr /> <h2 id="decoding-test-sentences-qualitative-analysis">Decoding test sentences (qualitative analysis)</h2> Finally, let's demonstrate how to translate brand new English sentences. We simply feed into the model the tokenized English sentence as well as the target token <code>"[START]"</code>. The model outputs probabilities of the next token. We then we repeatedly generated the next token conditioned on the tokens generated so far, until we hit the token <code>"[END]"</code>. For decoding, we will use the <code>keras_hub.samplers</code> module from KerasHub. Greedy Decoding is a text decoding method which outputs the most likely next token at each time step, i.e., the token with the highest probability. <div class="codehilite"><pre><code>def decode_sequences(input_sentences): batch_size = 1 # Tokenize the encoder input. encoder_input_tokens = ops.convert_to_tensor(eng_tokenizer(input_sentences)) if len(encoder_input_tokens[0]) < MAX_SEQUENCE_LENGTH: pads = ops.full((1, MAX_SEQUENCE_LENGTH - len(encoder_input_tokens[0])), 0) encoder_input_tokens = ops.concatenate( [encoder_input_tokens.to_tensor(), pads], 1 ) # Define a function that outputs the next token's probability given the # input sequence. def next(prompt, cache, index): logits = transformer([encoder_input_tokens, prompt])[:, index - 1, :] # Ignore hidden states for now; only needed for contrastive search. hidden_states = None return logits, hidden_states, cache # Build a prompt of length 40 with a start token and padding tokens. length = 40 start = ops.full((batch_size, 1), spa_tokenizer.token_to_id("[START]")) pad = ops.full((batch_size, length - 1), spa_tokenizer.token_to_id("[PAD]")) prompt = ops.concatenate((start, pad), axis=-1) generated_tokens = keras_hub.samplers.GreedySampler()( next, prompt, stop_token_ids=[spa_tokenizer.token_to_id("[END]")], index=1, # Start sampling after start token. ) generated_sentences = spa_tokenizer.detokenize(generated_tokens) return generated_sentences test_eng_texts = [pair[0] for pair in test_pairs] for i in range(2): input_sentence = random.choice(test_eng_texts) translated = decode_sequences([input_sentence]) translated = translated.numpy()[0].decode("utf-8") translated = ( translated.replace("[PAD]", "") .replace("[START]", "") .replace("[END]", "") .strip() ) print(f"** Example {i} **") print(input_sentence) print(translated) print() </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1714519073.816969 34774 device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process. ** Example 0 ** i got the ticket free of charge. me pregunto la comprome . </code></pre></div> </div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>** Example 1 ** i think maybe that's all you have to do. creo que tom le dije que hacer eso . </code></pre></div> </div> <hr /> <h2 id="evaluating-our-model-quantitative-analysis">Evaluating our model (quantitative analysis)</h2> There are many metrics which are used for text generation tasks. Here, to evaluate translations generated by our model, let's compute the ROUGE-1 and ROUGE-2 scores. Essentially, ROUGE-N is a score based on the number of common n-grams between the reference text and the generated text. ROUGE-1 and ROUGE-2 use the number of common unigrams and bigrams, respectively. We will calculate the score over 30 test samples (since decoding is an expensive process). <div class="codehilite"><pre><code>rouge_1 = keras_hub.metrics.RougeN(order=1) rouge_2 = keras_hub.metrics.RougeN(order=2) for test_pair in test_pairs[:30]: input_sentence = test_pair[0] reference_sentence = test_pair[1] translated_sentence = decode_sequences([input_sentence]) translated_sentence = translated_sentence.numpy()[0].decode("utf-8") translated_sentence = ( translated_sentence.replace("[PAD]", "") .replace("[START]", "") .replace("[END]", "") .strip() ) rouge_1(reference_sentence, translated_sentence) rouge_2(reference_sentence, translated_sentence) print("ROUGE-1 Score: ", rouge_1.result()) print("ROUGE-2 Score: ", rouge_2.result()) </code></pre></div> <div class="k-default-codeblock"> <div class="codehilite"><pre><code>ROUGE-1 Score: {'precision': <tf.Tensor: shape=(), dtype=float32, numpy=0.30989552>, 'recall': <tf.Tensor: shape=(), dtype=float32, numpy=0.37136248>, 'f1_score': <tf.Tensor: shape=(), dtype=float32, numpy=0.33032653>} ROUGE-2 Score: {'precision': <tf.Tensor: shape=(), dtype=float32, numpy=0.08999339>, 'recall': <tf.Tensor: shape=(), dtype=float32, numpy=0.09524643>, 'f1_score': <tf.Tensor: shape=(), dtype=float32, numpy=0.08855649>} </code></pre></div> </div> After 10 epochs, the scores are as follows: <table> <thead> <tr> <th style="text-align: center;"></th> <th style="text-align: center;">ROUGE-1</th> <th style="text-align: center;">ROUGE-2</th> </tr> </thead> <tbody> <tr> <td style="text-align: center;">Precision</td> <td style="text-align: center;">0.568</td> <td style="text-align: center;">0.374</td> </tr> <tr> <td style="text-align: center;">Recall</td> <td style="text-align: center;">0.615</td> <td style="text-align: center;">0.394</td> </tr> <tr> <td style="text-align: center;">F1 Score</td> <td style="text-align: center;">0.579</td> <td style="text-align: center;">0.381</td> </tr> </tbody> </table> </div> <div class='k-outline'> <div class='k-outline-depth-1'> <a href='#englishtospanish-translation-with-kerashub'>English-to-Spanish translation with KerasHub</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#introduction'>Introduction</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#setup'>Setup</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#downloading-the-data'>Downloading the data</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#parsing-the-data'>Parsing the data</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#tokenizing-the-data'>Tokenizing the data</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#format-datasets'>Format datasets</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#building-the-model'>Building the model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#training-our-model'>Training our model</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#decoding-test-sentences-qualitative-analysis'>Decoding test sentences (qualitative analysis)</a> </div> <div class='k-outline-depth-2'> ◆ <a href='#evaluating-our-model-quantitative-analysis'>Evaluating our model (quantitative analysis)</a> </div> </div> </div> </div> </div> </body> <footer style="float: left; width: 100%; padding: 1em; border-top: solid 1px #bbb;"> <a href="https://policies.google.com/terms">Terms</a> | <a href="https://policies.google.com/privacy">Privacy</a> </footer> </html>