Neural machine translation with attention | Text

<!doctype html> <html lang="en" dir="ltr"> <head> <meta name="google-signin-client-id" content="157101835696-ooapojlodmuabs2do2vuhhnf90bccmoi.apps.googleusercontent.com"> <meta name="google-signin-scope" content="profile email https://www.googleapis.com/auth/developerprofiles https://www.googleapis.com/auth/developerprofiles.award"> <meta property="og:site_name" content="TensorFlow"> <meta property="og:type" content="website"><meta name="theme-color" content="#ff6f00"><meta charset="utf-8"> <meta content="IE=Edge" http-equiv="X-UA-Compatible"> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="manifest" href="/_pwa/tensorflow/manifest.json" crossorigin="use-credentials"> <link rel="preconnect" href="//www.gstatic.com" crossorigin> <link rel="preconnect" href="//fonts.gstatic.com" crossorigin> <link rel="preconnect" href="//fonts.googleapis.com" crossorigin> <link rel="preconnect" href="//apis.google.com" crossorigin> <link rel="preconnect" href="//www.google-analytics.com" crossorigin><link rel="stylesheet" href="//fonts.googleapis.com/css?family=Google+Sans:400,500|Roboto:400,400italic,500,500italic,700,700italic|Roboto+Mono:400,500,700&display=swap"> <link rel="stylesheet" href="//fonts.googleapis.com/css2?family=Material+Icons&family=Material+Symbols+Outlined&display=block"><link rel="stylesheet" href="https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow/css/app.css"> <link rel="shortcut icon" href="https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow/images/favicon.png"> <link rel="apple-touch-icon" href="https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow/images/apple-touch-icon-180x180.png"><link rel="canonical" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention"><link rel="search" type="application/opensearchdescription+xml" title="TensorFlow" href="https://www.tensorflow.org/s/opensearch.xml"> <link rel="alternate" hreflang="en" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention" /><link rel="alternate" hreflang="x-default" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention" /><link rel="alternate" hreflang="ar" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=ar" /><link rel="alternate" hreflang="bn" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=bn" /><link rel="alternate" hreflang="fa" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=fa" /><link rel="alternate" hreflang="fr" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=fr" /><link rel="alternate" hreflang="he" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=he" /><link rel="alternate" hreflang="hi" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=hi" /><link rel="alternate" hreflang="id" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=id" /><link rel="alternate" hreflang="it" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=it" /><link rel="alternate" hreflang="ja" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=ja" /><link rel="alternate" hreflang="ko" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=ko" /><link rel="alternate" hreflang="pl" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=pl" /><link rel="alternate" hreflang="pt-BR" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=pt-br" /><link rel="alternate" hreflang="ru" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=ru" /><link rel="alternate" hreflang="es-419" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=es-419" /><link rel="alternate" hreflang="th" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=th" /><link rel="alternate" hreflang="tr" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=tr" /><link rel="alternate" hreflang="vi" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention?hl=vi" /><title>Neural machine translation with attention  |  Text  |  TensorFlow</title> <meta property="og:title" content="Neural machine translation with attention  |  Text  |  TensorFlow"><meta property="og:url" content="https://www.tensorflow.org/text/tutorials/nmt_with_attention"><meta property="og:image" content="https://www.tensorflow.org/static/images/tf_logo_social.png"> <meta property="og:image:width" content="1200"> <meta property="og:image:height" content="675"><meta property="og:locale" content="en"><meta name="twitter:card" content="summary_large_image"><script type="application/ld+json"> { "@context": "https://schema.org", "@type": "Article", "headline": "Neural machine translation with attention" } </script><script type="application/ld+json"> { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [{ "@type": "ListItem", "position": 1, "name": "Text", "item": "https://www.tensorflow.org/text" },{ "@type": "ListItem", "position": 2, "name": "Neural machine translation with attention", "item": "https://www.tensorflow.org/text/tutorials/nmt_with_attention" }] } </script> <link rel="stylesheet" href="/extras.css"></head> <body class="" template="page" theme="tensorflow-theme" type="article" layout="docs" display-toc pending> <devsite-progress type="indeterminate" id="app-progress"></devsite-progress> <section class="devsite-wrapper"> <devsite-cookie-notification-bar></devsite-cookie-notification-bar><devsite-header role="banner"> <div class="devsite-header--inner nocontent"> <div class="devsite-top-logo-row-wrapper-wrapper"> <div class="devsite-top-logo-row-wrapper"> <div class="devsite-top-logo-row"> <button type="button" id="devsite-hamburger-menu" class="devsite-header-icon-button button-flat material-icons gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Navigation menu button" visually-hidden aria-label="Open menu"> </button> <div class="devsite-product-name-wrapper"> <a href="/" class="devsite-site-logo-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Site logo" track-type="globalNav" track-name="tensorFlow" track-metadata-position="nav" track-metadata-eventDetail="nav"> <picture> <img src="https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow/images/lockup.svg" class="devsite-site-logo" alt="TensorFlow"> </picture> </a> <span class="devsite-product-name"> <ul class="devsite-breadcrumb-list" > <li class="devsite-breadcrumb-item "> </li> </ul> </span> </div> <div class="devsite-top-logo-row-middle"> <div class="devsite-header-upper-tabs"> <devsite-tabs class="upper-tabs"> <nav class="devsite-tabs-wrapper" aria-label="Upper tabs"> <tab > <a href="https://www.tensorflow.org/install" track-metadata-eventdetail="https://www.tensorflow.org/install" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - install" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Install" track-name="install" > Install </a> </tab> <tab class="devsite-dropdown "> <a href="https://www.tensorflow.org/learn" track-metadata-eventdetail="https://www.tensorflow.org/learn" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - learn" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Learn" track-name="learn" > Learn </a> <a href="#" role="button" aria-haspopup="true" aria-expanded="false" aria-label="Dropdown menu for Learn" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/learn" track-metadata-position="nav - learn" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Learn" track-name="learn" class="devsite-tabs-dropdown-toggle devsite-icon devsite-icon-arrow-drop-down"></a> <div class="devsite-tabs-dropdown" aria-label="submenu" hidden> <div class="devsite-tabs-dropdown-content"> <div class="devsite-tabs-dropdown-column tfo-menu-column-learn"> <ul class="devsite-tabs-dropdown-section "> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/learn" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/learn" track-metadata-position="nav - learn" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Introduction </div> <div class="devsite-nav-item-description"> New to TensorFlow? </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/tutorials" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/tutorials" track-metadata-position="nav - learn" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Tutorials </div> <div class="devsite-nav-item-description"> Learn how to use TensorFlow with end-to-end examples </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/guide" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/guide" track-metadata-position="nav - learn" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Guide </div> <div class="devsite-nav-item-description"> Learn framework concepts and components </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/resources/learn-ml" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/resources/learn-ml" track-metadata-position="nav - learn" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Learn ML </div> <div class="devsite-nav-item-description"> Educational resources to master your path with TensorFlow </div> </a> </li> </ul> </div> </div> </div> </tab> <tab class="devsite-dropdown "> <a href="https://www.tensorflow.org/api" track-metadata-eventdetail="https://www.tensorflow.org/api" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - api" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: API" track-name="api" > API </a> <a href="#" role="button" aria-haspopup="true" aria-expanded="false" aria-label="Dropdown menu for API" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/api" track-metadata-position="nav - api" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: API" track-name="api" class="devsite-tabs-dropdown-toggle devsite-icon devsite-icon-arrow-drop-down"></a> <div class="devsite-tabs-dropdown" aria-label="submenu" hidden> <div class="devsite-tabs-dropdown-content"> <div class="devsite-tabs-dropdown-column "> <ul class="devsite-tabs-dropdown-section "> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/api/stable" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/api/stable" track-metadata-position="nav - api" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> TensorFlow (v2.16.1) </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/versions" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/versions" track-metadata-position="nav - api" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Versions… </div> </a> </li> </ul> </div> <div class="devsite-tabs-dropdown-column "> <ul class="devsite-tabs-dropdown-section "> <li class="devsite-nav-item"> <a href="https://js.tensorflow.org/api/latest/" track-type="nav" track-metadata-eventdetail="https://js.tensorflow.org/api/latest/" track-metadata-position="nav - api" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> TensorFlow.js </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/lite/api_docs" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/lite/api_docs" track-metadata-position="nav - api" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> TensorFlow Lite </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/tfx/api_docs" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/tfx/api_docs" track-metadata-position="nav - api" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> TFX </div> </a> </li> </ul> </div> </div> </div> </tab> <tab class="devsite-dropdown devsite-active "> <a href="https://www.tensorflow.org/resources" track-metadata-eventdetail="https://www.tensorflow.org/resources" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - resources" track-metadata-module="primary nav" aria-label="Resources, selected" data-category="Site-Wide Custom Events" data-label="Tab: Resources" track-name="resources" > Resources </a> <a href="#" role="button" aria-haspopup="true" aria-expanded="false" aria-label="Dropdown menu for Resources" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/resources" track-metadata-position="nav - resources" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Resources" track-name="resources" class="devsite-tabs-dropdown-toggle devsite-icon devsite-icon-arrow-drop-down"></a> <div class="devsite-tabs-dropdown" aria-label="submenu" hidden> <div class="devsite-tabs-dropdown-content"> <div class="devsite-tabs-dropdown-column "> <ul class="devsite-tabs-dropdown-section "> <li class="devsite-nav-title" role="heading" tooltip>LIBRARIES</li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/js" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/js" track-metadata-position="nav - resources" track-metadata-module="tertiary nav" track-metadata-module_headline="libraries" tooltip > <div class="devsite-nav-item-title"> TensorFlow.js </div> <div class="devsite-nav-item-description"> Develop web ML applications in JavaScript </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/lite" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/lite" track-metadata-position="nav - resources" track-metadata-module="tertiary nav" track-metadata-module_headline="libraries" tooltip > <div class="devsite-nav-item-title"> TensorFlow Lite </div> <div class="devsite-nav-item-description"> Deploy ML on mobile, microcontrollers and other edge devices </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/tfx" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/tfx" track-metadata-position="nav - resources" track-metadata-module="tertiary nav" track-metadata-module_headline="libraries" tooltip > <div class="devsite-nav-item-title"> TFX </div> <div class="devsite-nav-item-description"> Build production ML pipelines </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/resources/libraries-extensions" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/resources/libraries-extensions" track-metadata-position="nav - resources" track-metadata-module="tertiary nav" track-metadata-module_headline="libraries" tooltip > <div class="devsite-nav-item-title"> All libraries </div> <div class="devsite-nav-item-description"> Create advanced models and extend TensorFlow </div> </a> </li> </ul> </div> <div class="devsite-tabs-dropdown-column "> <ul class="devsite-tabs-dropdown-section "> <li class="devsite-nav-title" role="heading" tooltip>RESOURCES</li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/resources/models-datasets" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/resources/models-datasets" track-metadata-position="nav - resources" track-metadata-module="tertiary nav" track-metadata-module_headline="resources" tooltip > <div class="devsite-nav-item-title"> Models & datasets </div> <div class="devsite-nav-item-description"> Pre-trained models and datasets built by Google and the community </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/resources/tools" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/resources/tools" track-metadata-position="nav - resources" track-metadata-module="tertiary nav" track-metadata-module_headline="resources" tooltip > <div class="devsite-nav-item-title"> Tools </div> <div class="devsite-nav-item-description"> Tools to support and accelerate TensorFlow workflows </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/responsible_ai" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/responsible_ai" track-metadata-position="nav - resources" track-metadata-module="tertiary nav" track-metadata-module_headline="resources" tooltip > <div class="devsite-nav-item-title"> Responsible AI </div> <div class="devsite-nav-item-description"> Resources for every stage of the ML workflow </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/resources/recommendation-systems" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/resources/recommendation-systems" track-metadata-position="nav - resources" track-metadata-module="tertiary nav" track-metadata-module_headline="resources" tooltip > <div class="devsite-nav-item-title"> Recommendation systems </div> <div class="devsite-nav-item-description"> Build recommendation systems with open source tools </div> </a> </li> </ul> </div> </div> </div> </tab> <tab class="devsite-dropdown "> <a href="https://www.tensorflow.org/community" track-metadata-eventdetail="https://www.tensorflow.org/community" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - community" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Community" track-name="community" > Community </a> <a href="#" role="button" aria-haspopup="true" aria-expanded="false" aria-label="Dropdown menu for Community" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/community" track-metadata-position="nav - community" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Community" track-name="community" class="devsite-tabs-dropdown-toggle devsite-icon devsite-icon-arrow-drop-down"></a> <div class="devsite-tabs-dropdown" aria-label="submenu" hidden> <div class="devsite-tabs-dropdown-content"> <div class="devsite-tabs-dropdown-column "> <ul class="devsite-tabs-dropdown-section "> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/community/groups" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/community/groups" track-metadata-position="nav - community" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Groups </div> <div class="devsite-nav-item-description"> User groups, interest groups and mailing lists </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/community/contribute" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/community/contribute" track-metadata-position="nav - community" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Contribute </div> <div class="devsite-nav-item-description"> Guide for contributing to code and documentation </div> </a> </li> <li class="devsite-nav-item"> <a href="https://blog.tensorflow.org/" track-type="nav" track-metadata-eventdetail="https://blog.tensorflow.org/" track-metadata-position="nav - community" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Blog </div> <div class="devsite-nav-item-description"> Stay up to date with all things TensorFlow </div> </a> </li> <li class="devsite-nav-item"> <a href="https://discuss.tensorflow.org" track-type="nav" track-metadata-eventdetail="https://discuss.tensorflow.org" track-metadata-position="nav - community" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Forum </div> <div class="devsite-nav-item-description"> Discussion platform for the TensorFlow community </div> </a> </li> </ul> </div> </div> </div> </tab> <tab class="devsite-dropdown "> <a href="https://www.tensorflow.org/about" track-metadata-eventdetail="https://www.tensorflow.org/about" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - why tensorflow" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Why TensorFlow" track-name="why tensorflow" > Why TensorFlow </a> <a href="#" role="button" aria-haspopup="true" aria-expanded="false" aria-label="Dropdown menu for Why TensorFlow" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/about" track-metadata-position="nav - why tensorflow" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Why TensorFlow" track-name="why tensorflow" class="devsite-tabs-dropdown-toggle devsite-icon devsite-icon-arrow-drop-down"></a> <div class="devsite-tabs-dropdown" aria-label="submenu" hidden> <div class="devsite-tabs-dropdown-content"> <div class="devsite-tabs-dropdown-column "> <ul class="devsite-tabs-dropdown-section "> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/about" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/about" track-metadata-position="nav - why tensorflow" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> About </div> </a> </li> <li class="devsite-nav-item"> <a href="https://www.tensorflow.org/about/case-studies" track-type="nav" track-metadata-eventdetail="https://www.tensorflow.org/about/case-studies" track-metadata-position="nav - why tensorflow" track-metadata-module="tertiary nav" tooltip > <div class="devsite-nav-item-title"> Case studies </div> </a> </li> </ul> </div> </div> </div> </tab> </nav> </devsite-tabs> </div> <devsite-search enable-signin enable-search enable-suggestions enable-query-completion project-name="Text" tenant-name="TensorFlow" > <form class="devsite-search-form" action="https://www.tensorflow.org/s/results" method="GET"> <div class="devsite-search-container"> <button type="button" search-open class="devsite-search-button devsite-header-icon-button button-flat material-icons" aria-label="Open search"></button> <div class="devsite-searchbox"> <input aria-activedescendant="" aria-autocomplete="list" aria-label="Search" aria-expanded="false" aria-haspopup="listbox" autocomplete="off" class="devsite-search-field devsite-search-query" name="q" placeholder="Search" role="combobox" type="text" value="" > <div class="devsite-search-image material-icons" aria-hidden="true"> </div> <div class="devsite-search-shortcut-icon-container" aria-hidden="true"> <kbd class="devsite-search-shortcut-icon">/</kbd> </div> </div> </div> </form> <button type="button" search-close class="devsite-search-button devsite-header-icon-button button-flat material-icons" aria-label="Close search"></button> </devsite-search> </div> <devsite-language-selector> <ul role="presentation"> <li role="presentation"> <a role="menuitem" lang="en" >English</a> </li> <li role="presentation"> <a role="menuitem" lang="es_419" >Español – América Latina</a> </li> <li role="presentation"> <a role="menuitem" lang="fr" >Français</a> </li> <li role="presentation"> <a role="menuitem" lang="id" >Indonesia</a> </li> <li role="presentation"> <a role="menuitem" lang="it" >Italiano</a> </li> <li role="presentation"> <a role="menuitem" lang="pl" >Polski</a> </li> <li role="presentation"> <a role="menuitem" lang="pt_br" >Português – Brasil</a> </li> <li role="presentation"> <a role="menuitem" lang="vi" >Tiếng Việt</a> </li> <li role="presentation"> <a role="menuitem" lang="tr" >Türkçe</a> </li> <li role="presentation"> <a role="menuitem" lang="ru" >Русский</a> </li> <li role="presentation"> <a role="menuitem" lang="he" >עברית</a> </li> <li role="presentation"> <a role="menuitem" lang="ar" >العربيّة</a> </li> <li role="presentation"> <a role="menuitem" lang="fa" >فارسی</a> </li> <li role="presentation"> <a role="menuitem" lang="hi" >हिंदी</a> </li> <li role="presentation"> <a role="menuitem" lang="bn" >বাংলা</a> </li> <li role="presentation"> <a role="menuitem" lang="th" >ภาษาไทย</a> </li> <li role="presentation"> <a role="menuitem" lang="zh_cn" >中文 – 简体</a> </li> <li role="presentation"> <a role="menuitem" lang="ja" >日本語</a> </li> <li role="presentation"> <a role="menuitem" lang="ko" >한국어</a> </li> </ul> </devsite-language-selector> <a class="devsite-header-link devsite-top-button button gc-analytics-event" href="//github.com/tensorflow" data-category="Site-Wide Custom Events" data-label="Site header link" > GitHub </a> <devsite-user enable-profiles id="devsite-user"> <span class="button devsite-top-button" aria-hidden="true" visually-hidden>Sign in</span> </devsite-user> </div> </div> </div> <div class="devsite-collapsible-section "> <div class="devsite-header-background"> <div class="devsite-product-id-row" > <div class="devsite-product-description-row"> <ul class="devsite-breadcrumb-list" > <li class="devsite-breadcrumb-item "> <a href="https://www.tensorflow.org/text" class="devsite-breadcrumb-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Lower Header" data-value="1" track-type="globalNav" track-name="breadcrumb" track-metadata-position="1" track-metadata-eventdetail="Text" > Text </a> </li> </ul> </div> </div> <div class="devsite-doc-set-nav-row"> <devsite-tabs class="lower-tabs"> <nav class="devsite-tabs-wrapper" aria-label="Lower tabs"> <tab > <a href="https://www.tensorflow.org/text" track-metadata-eventdetail="https://www.tensorflow.org/text" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - overview" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Overview" track-name="overview" > Overview </a> </tab> <tab class="devsite-active"> <a href="https://www.tensorflow.org/text/tutorials" track-metadata-eventdetail="https://www.tensorflow.org/text/tutorials" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - tutorials" track-metadata-module="primary nav" aria-label="Tutorials, selected" data-category="Site-Wide Custom Events" data-label="Tab: Tutorials" track-name="tutorials" > Tutorials </a> </tab> <tab > <a href="https://www.tensorflow.org/text/guide" track-metadata-eventdetail="https://www.tensorflow.org/text/guide" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - guide" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: Guide" track-name="guide" > Guide </a> </tab> <tab > <a href="https://www.tensorflow.org/text/api_overview" track-metadata-eventdetail="https://www.tensorflow.org/text/api_overview" class="devsite-tabs-content gc-analytics-event " track-type="nav" track-metadata-position="nav - api" track-metadata-module="primary nav" data-category="Site-Wide Custom Events" data-label="Tab: API" track-name="api" > API </a> </tab> </nav> </devsite-tabs> </div> </div> </div> </div> </devsite-header> <devsite-book-nav scrollbars > <div class="devsite-book-nav-filter" > <span class="filter-list-icon material-icons" aria-hidden="true"></span> <input type="text" placeholder="Filter" aria-label="Type to filter" role="searchbox"> <span class="filter-clear-button hidden" data-title="Clear filter" aria-label="Clear filter" role="button" tabindex="0"></span> </div> <nav class="devsite-book-nav devsite-nav nocontent" aria-label="Side menu"> <div class="devsite-mobile-header"> <button type="button" id="devsite-close-nav" class="devsite-header-icon-button button-flat material-icons gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Close navigation" aria-label="Close navigation"> </button> <div class="devsite-product-name-wrapper"> <a href="/" class="devsite-site-logo-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Site logo" track-type="globalNav" track-name="tensorFlow" track-metadata-position="nav" track-metadata-eventDetail="nav"> <picture> <img src="https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow/images/lockup.svg" class="devsite-site-logo" alt="TensorFlow"> </picture> </a> <span class="devsite-product-name"> <ul class="devsite-breadcrumb-list" > <li class="devsite-breadcrumb-item "> </li> </ul> </span> </div> </div> <div class="devsite-book-nav-wrapper"> <div class="devsite-mobile-nav-top"> <ul class="devsite-nav-list"> <li class="devsite-nav-item"> <a href="/install" class="devsite-nav-title gc-analytics-event devsite-nav-has-children " data-category="Site-Wide Custom Events" data-label="Tab: Install" track-name="install" data-category="Site-Wide Custom Events" data-label="Responsive Tab: Install" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Install </span> <span class="devsite-nav-icon material-icons" data-icon="forward" > </span> </a> </li> <li class="devsite-nav-item"> <a href="/learn" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Tab: Learn" track-name="learn" data-category="Site-Wide Custom Events" data-label="Responsive Tab: Learn" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Learn </span> </a> <ul class="devsite-nav-responsive-tabs devsite-nav-has-menu "> <li class="devsite-nav-item"> <span class="devsite-nav-title" tooltip data-category="Site-Wide Custom Events" data-label="Tab: Learn" track-name="learn" > <span class="devsite-nav-text" tooltip menu="Learn"> More </span> <span class="devsite-nav-icon material-icons" data-icon="forward" menu="Learn"> </span> </span> </li> </ul> </li> <li class="devsite-nav-item"> <a href="/api" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Tab: API" track-name="api" data-category="Site-Wide Custom Events" data-label="Responsive Tab: API" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > API </span> </a> <ul class="devsite-nav-responsive-tabs devsite-nav-has-menu "> <li class="devsite-nav-item"> <span class="devsite-nav-title" tooltip data-category="Site-Wide Custom Events" data-label="Tab: API" track-name="api" > <span class="devsite-nav-text" tooltip menu="API"> More </span> <span class="devsite-nav-icon material-icons" data-icon="forward" menu="API"> </span> </span> </li> </ul> </li> <li class="devsite-nav-item"> <a href="/resources" class="devsite-nav-title gc-analytics-event devsite-nav-active" data-category="Site-Wide Custom Events" data-label="Tab: Resources" track-name="resources" data-category="Site-Wide Custom Events" data-label="Responsive Tab: Resources" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Resources </span> </a> <ul class="devsite-nav-responsive-tabs devsite-nav-has-menu "> <li class="devsite-nav-item"> <span class="devsite-nav-title" tooltip data-category="Site-Wide Custom Events" data-label="Tab: Resources" track-name="resources" > <span class="devsite-nav-text" tooltip menu="Resources"> More </span> <span class="devsite-nav-icon material-icons" data-icon="forward" menu="Resources"> </span> </span> </li> </ul> <ul class="devsite-nav-responsive-tabs"> <li class="devsite-nav-item"> <a href="/text" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Tab: Overview" track-name="overview" data-category="Site-Wide Custom Events" data-label="Responsive Tab: Overview" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Overview </span> </a> </li> <li class="devsite-nav-item"> <a href="/text/tutorials" class="devsite-nav-title gc-analytics-event devsite-nav-has-children devsite-nav-active" data-category="Site-Wide Custom Events" data-label="Tab: Tutorials" track-name="tutorials" data-category="Site-Wide Custom Events" data-label="Responsive Tab: Tutorials" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip menu="_book"> Tutorials </span> <span class="devsite-nav-icon material-icons" data-icon="forward" menu="_book"> </span> </a> </li> <li class="devsite-nav-item"> <a href="/text/guide" class="devsite-nav-title gc-analytics-event devsite-nav-has-children " data-category="Site-Wide Custom Events" data-label="Tab: Guide" track-name="guide" data-category="Site-Wide Custom Events" data-label="Responsive Tab: Guide" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Guide </span> <span class="devsite-nav-icon material-icons" data-icon="forward" > </span> </a> </li> <li class="devsite-nav-item"> <a href="/text/api_overview" class="devsite-nav-title gc-analytics-event devsite-nav-has-children " data-category="Site-Wide Custom Events" data-label="Tab: API" track-name="api" data-category="Site-Wide Custom Events" data-label="Responsive Tab: API" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > API </span> <span class="devsite-nav-icon material-icons" data-icon="forward" > </span> </a> </li> </ul> </li> <li class="devsite-nav-item"> <a href="/community" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Tab: Community" track-name="community" data-category="Site-Wide Custom Events" data-label="Responsive Tab: Community" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Community </span> </a> <ul class="devsite-nav-responsive-tabs devsite-nav-has-menu "> <li class="devsite-nav-item"> <span class="devsite-nav-title" tooltip data-category="Site-Wide Custom Events" data-label="Tab: Community" track-name="community" > <span class="devsite-nav-text" tooltip menu="Community"> More </span> <span class="devsite-nav-icon material-icons" data-icon="forward" menu="Community"> </span> </span> </li> </ul> </li> <li class="devsite-nav-item"> <a href="/about" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Tab: Why TensorFlow" track-name="why tensorflow" data-category="Site-Wide Custom Events" data-label="Responsive Tab: Why TensorFlow" track-type="globalNav" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Why TensorFlow </span> </a> <ul class="devsite-nav-responsive-tabs devsite-nav-has-menu "> <li class="devsite-nav-item"> <span class="devsite-nav-title" tooltip data-category="Site-Wide Custom Events" data-label="Tab: Why TensorFlow" track-name="why tensorflow" > <span class="devsite-nav-text" tooltip menu="Why TensorFlow"> More </span> <span class="devsite-nav-icon material-icons" data-icon="forward" menu="Why TensorFlow"> </span> </span> </li> </ul> </li> <li class="devsite-nav-item"> <a href="//github.com/tensorflow" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: GitHub" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > GitHub </span> </a> </li> </ul> </div> <div class="devsite-mobile-nav-bottom"> <ul class="devsite-nav-list" menu="_book"> <li class="devsite-nav-item"><a href="/text/tutorials" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials" ><span class="devsite-nav-text" tooltip>Overview</span></a></li> <li class="devsite-nav-item devsite-nav-divider devsite-nav-heading"><div class="devsite-nav-title devsite-nav-title-no-path"> <span class="devsite-nav-text" tooltip>Keras NLP</span> </div></li> <li class="devsite-nav-item devsite-nav-external"><a href="https://keras.io/guides/keras_nlp/getting_started/" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: https://keras.io/guides/keras_nlp/getting_started/" track-type="bookNav" track-name="click" track-metadata-eventdetail="https://keras.io/guides/keras_nlp/getting_started/" ><span class="devsite-nav-text" tooltip>Get started with KerasNLP</span><span class="devsite-nav-icon material-icons" data-icon="external" data-title="External" aria-hidden="true"></span></a></li> <li class="devsite-nav-item devsite-nav-divider devsite-nav-heading"><div class="devsite-nav-title devsite-nav-title-no-path"> <span class="devsite-nav-text" tooltip>Text Generation</span> </div></li> <li class="devsite-nav-item"><a href="/text/tutorials/text_generation" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/text_generation" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/text_generation" ><span class="devsite-nav-text" tooltip>Generate Text with RNNs</span></a></li> <li class="devsite-nav-item"><a href="/text/tutorials/nmt_with_attention" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/nmt_with_attention" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/nmt_with_attention" ><span class="devsite-nav-text" tooltip>Translate text with seq2seq models</span></a></li> <li class="devsite-nav-item"><a href="/text/tutorials/transformer" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/transformer" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/transformer" ><span class="devsite-nav-text" tooltip>Translate text with transformer models</span></a></li> <li class="devsite-nav-item"><a href="/text/tutorials/image_captioning" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/image_captioning" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/image_captioning" ><span class="devsite-nav-text" tooltip>Image captioning</span></a></li> <li class="devsite-nav-item devsite-nav-divider devsite-nav-heading"><div class="devsite-nav-title devsite-nav-title-no-path"> <span class="devsite-nav-text" tooltip>Text Classification</span> </div></li> <li class="devsite-nav-item"><a href="/text/tutorials/classify_text_with_bert" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/classify_text_with_bert" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/classify_text_with_bert" ><span class="devsite-nav-text" tooltip>Text classification with BERT</span></a></li> <li class="devsite-nav-item"><a href="/text/tutorials/text_classification_rnn" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/text_classification_rnn" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/text_classification_rnn" ><span class="devsite-nav-text" tooltip>Text classification with RNNs</span></a></li> <li class="devsite-nav-item"><a href="/text/tutorials/text_similarity" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/text_similarity" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/text_similarity" ><span class="devsite-nav-text" tooltip>Compute Similarity Metrics</span></a></li> <li class="devsite-nav-item devsite-nav-divider devsite-nav-heading"><div class="devsite-nav-title devsite-nav-title-no-path"> <span class="devsite-nav-text" tooltip>NLP with BERT</span> </div></li> <li class="devsite-nav-item"><a href="/text/tutorials/bert_glue" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/bert_glue" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/bert_glue" ><span class="devsite-nav-text" tooltip>Fine Tune Bert on GLUE tasks</span></a></li> <li class="devsite-nav-item"><a href="/tfmodels/nlp/fine_tune_bert" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /tfmodels/nlp/fine_tune_bert" track-type="bookNav" track-name="click" track-metadata-eventdetail="/tfmodels/nlp/fine_tune_bert" ><span class="devsite-nav-text" tooltip>Fine tune BERT</span></a></li> <li class="devsite-nav-item"><a href="/text/tutorials/uncertainty_quantification_with_sngp_bert" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/uncertainty_quantification_with_sngp_bert" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/uncertainty_quantification_with_sngp_bert" ><span class="devsite-nav-text" tooltip>Quantify uncertainty with BERT</span></a></li> <li class="devsite-nav-item devsite-nav-divider devsite-nav-heading"><div class="devsite-nav-title devsite-nav-title-no-path"> <span class="devsite-nav-text" tooltip>Embeddings</span> </div></li> <li class="devsite-nav-item"><a href="/text/tutorials/word_embeddings" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/word_embeddings" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/word_embeddings" ><span class="devsite-nav-text" tooltip>Word embeddings</span></a></li> <li class="devsite-nav-item"><a href="/text/tutorials/warmstart_embedding_matrix" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/warmstart_embedding_matrix" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/warmstart_embedding_matrix" ><span class="devsite-nav-text" tooltip>Warmstarting embeddings</span></a></li> <li class="devsite-nav-item"><a href="/text/tutorials/word2vec" class="devsite-nav-title gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Book nav link, pathname: /text/tutorials/word2vec" track-type="bookNav" track-name="click" track-metadata-eventdetail="/text/tutorials/word2vec" ><span class="devsite-nav-text" tooltip>Word2Vec</span></a></li> </ul> <ul class="devsite-nav-list" menu="Learn" aria-label="Side menu" hidden> <li class="devsite-nav-item"> <a href="/learn" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Introduction" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Introduction </span> </a> </li> <li class="devsite-nav-item"> <a href="/tutorials" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Tutorials" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Tutorials </span> </a> </li> <li class="devsite-nav-item"> <a href="/guide" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Guide" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Guide </span> </a> </li> <li class="devsite-nav-item"> <a href="/resources/learn-ml" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Learn ML" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Learn ML </span> </a> </li> </ul> <ul class="devsite-nav-list" menu="API" aria-label="Side menu" hidden> <li class="devsite-nav-item"> <a href="/api/stable" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: TensorFlow (v2.16.1)" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > TensorFlow (v2.16.1) </span> </a> </li> <li class="devsite-nav-item"> <a href="/versions" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Versions…" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Versions… </span> </a> </li> <li class="devsite-nav-item"> <a href="https://js.tensorflow.org/api/latest/" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: TensorFlow.js" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > TensorFlow.js </span> </a> </li> <li class="devsite-nav-item"> <a href="/lite/api_docs" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: TensorFlow Lite" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > TensorFlow Lite </span> </a> </li> <li class="devsite-nav-item"> <a href="/tfx/api_docs" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: TFX" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > TFX </span> </a> </li> </ul> <ul class="devsite-nav-list" menu="Resources" aria-label="Side menu" hidden> <li class="devsite-nav-item devsite-nav-heading"> <span class="devsite-nav-title" tooltip > <span class="devsite-nav-text" tooltip > LIBRARIES </span> </span> </li> <li class="devsite-nav-item"> <a href="/js" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: TensorFlow.js" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > TensorFlow.js </span> </a> </li> <li class="devsite-nav-item"> <a href="/lite" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: TensorFlow Lite" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > TensorFlow Lite </span> </a> </li> <li class="devsite-nav-item"> <a href="/tfx" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: TFX" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > TFX </span> </a> </li> <li class="devsite-nav-item"> <a href="/resources/libraries-extensions" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: All libraries" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > All libraries </span> </a> </li> <li class="devsite-nav-item devsite-nav-heading"> <span class="devsite-nav-title" tooltip > <span class="devsite-nav-text" tooltip > RESOURCES </span> </span> </li> <li class="devsite-nav-item"> <a href="/resources/models-datasets" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Models & datasets" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Models & datasets </span> </a> </li> <li class="devsite-nav-item"> <a href="/resources/tools" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Tools" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Tools </span> </a> </li> <li class="devsite-nav-item"> <a href="/responsible_ai" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Responsible AI" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Responsible AI </span> </a> </li> <li class="devsite-nav-item"> <a href="/resources/recommendation-systems" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Recommendation systems" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Recommendation systems </span> </a> </li> </ul> <ul class="devsite-nav-list" menu="Community" aria-label="Side menu" hidden> <li class="devsite-nav-item"> <a href="/community/groups" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Groups" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Groups </span> </a> </li> <li class="devsite-nav-item"> <a href="/community/contribute" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Contribute" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Contribute </span> </a> </li> <li class="devsite-nav-item"> <a href="https://blog.tensorflow.org/" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Blog" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Blog </span> </a> </li> <li class="devsite-nav-item"> <a href="https://discuss.tensorflow.org" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Forum" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Forum </span> </a> </li> </ul> <ul class="devsite-nav-list" menu="Why TensorFlow" aria-label="Side menu" hidden> <li class="devsite-nav-item"> <a href="/about" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: About" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > About </span> </a> </li> <li class="devsite-nav-item"> <a href="/about/case-studies" class="devsite-nav-title gc-analytics-event " data-category="Site-Wide Custom Events" data-label="Responsive Tab: Case studies" track-type="navMenu" track-metadata-eventDetail="globalMenu" track-metadata-position="nav"> <span class="devsite-nav-text" tooltip > Case studies </span> </a> </li> </ul> </div> </div> </nav> </devsite-book-nav> <section id="gc-wrapper"> <main role="main" class="devsite-main-content" has-book-nav has-sidebar > <div class="devsite-sidebar"> <div class="devsite-sidebar-content"> <devsite-toc class="devsite-nav" role="navigation" aria-label="On this page" depth="2" scrollbars ></devsite-toc> <devsite-recommendations-sidebar class="nocontent devsite-nav"> </devsite-recommendations-sidebar> </div> </div> <devsite-content> <article class="devsite-article"><style> /* Styles inlined from /site-assets/css/style.css */ /* override theme */ table img { max-width: 100%; } /* .devsite-terminal virtualenv prompt */ .tfo-terminal-venv::before { content: "(venv) $ " !important; } /* .devsite-terminal root prompt */ .tfo-terminal-root::before { content: "# " !important; } /* Used in links for type annotations in function/method signatures */ .tfo-signature-link a, .tfo-signature-link a:visited, .tfo-signature-link a:hover, .tfo-signature-link a:focus, .tfo-signature-link a:hover *, .tfo-signature-link a:focus * { text-decoration: none !important; } .tfo-signature-link a, .tfo-signature-link a:visited { border-bottom: 1px dotted #1a73e8; } .tfo-signature-link a:focus { border-bottom-style: solid; } /* .devsite-terminal Windows prompt */ .tfo-terminal-windows::before { content: "C:\\> " !important; } /* .devsite-terminal Windows prompt w/ virtualenv */ .tfo-terminal-windows-venv::before { content: "(venv) C:\\> " !important; } .tfo-diff-green-one-level + * { background: rgba(175, 245, 162, .6) !important; } .tfo-diff-green + * > * { background: rgba(175, 245, 162, .6) !important; } .tfo-diff-green-list + ul > li:first-of-type { background: rgba(175, 245, 162, .6) !important; } .tfo-diff-red-one-level + * { background: rgba(255, 230, 230, .6) !important; text-decoration: line-through !important; } .tfo-diff-red + * > * { background: rgba(255, 230, 230, .6) !important; text-decoration: line-through !important; } .tfo-diff-red-list + ul > li:first-of-type { background: rgba(255, 230, 230, .6) !important; text-decoration: line-through !important; } devsite-code .tfo-notebook-code-cell-output { max-height: 300px; overflow: auto; background: rgba(255, 247, 237, 1); /* orange bg to distinguish from input code cells */ } devsite-code .tfo-notebook-code-cell-output + .devsite-code-buttons-container button { background: rgba(255, 247, 237, .7); /* orange bg to distinguish from input code cells */ } devsite-code[dark-code] .tfo-notebook-code-cell-output { background: rgba(64, 78, 103, 1); /* medium slate */ } devsite-code[dark-code] .tfo-notebook-code-cell-output + .devsite-code-buttons-container button { background: rgba(64, 78, 103, .7); /* medium slate */ } /* override default table styles for notebook buttons */ .devsite-table-wrapper .tfo-notebook-buttons { display: inline-block; margin-left: 3px; width: auto; } .tfo-notebook-buttons td { padding-left: 0; padding-right: 20px; } .tfo-notebook-buttons a, .tfo-notebook-buttons :link, .tfo-notebook-buttons :visited { border-radius: 8px; box-shadow: 0 1px 2px 0 rgba(60, 64, 67, .3), 0 1px 3px 1px rgba(60, 64, 67, .15); color: #202124; padding: 12px 17px; transition: box-shadow 0.2s; } .tfo-notebook-buttons a:hover, .tfo-notebook-buttons a:focus { box-shadow: 0 1px 2px 0 rgba(60, 64, 67, .3), 0 2px 6px 2px rgba(60, 64, 67, .15); } .tfo-notebook-buttons tr { background: 0; border: 0; } /* on rendered notebook page, remove link to webpage since we're already here */ .tfo-notebook-buttons:not(.tfo-api) td:first-child { display: none; } .tfo-notebook-buttons td > a { -webkit-box-align: center; -ms-flex-align: center; align-items: center; display: -webkit-box; display: -ms-flexbox; display: flex; } .tfo-notebook-buttons td > a > img { margin-right: 8px; } /* landing pages */ .tfo-landing-row-item-inset-white { background-color: #fff; padding: 32px; } .tfo-landing-row-item-inset-white ol, .tfo-landing-row-item-inset-white ul { padding-left: 20px; } /* colab callout button */ .colab-callout-row devsite-code { border-radius: 8px 8px 0 0; box-shadow: none; } .colab-callout-footer { background: #e3e4e7; border-radius: 0 0 8px 8px; color: #37474f; padding: 20px; } .colab-callout-row devsite-code[dark-code] + .colab-callout-footer { background: #3f4f66; } .colab-callout-footer > .button { margin-top: 4px; color: #ff5c00; } .colab-callout-footer > a > span { vertical-align: middle; color: #37474f; padding-left: 10px; font-size: 14px; } .colab-callout-row devsite-code[dark-code] + .colab-callout-footer > a > span { color: #fff; } a.colab-button { background: rgba(255, 255, 255, .75); border: solid 1px rgba(0, 0, 0, .08); border-bottom-color: rgba(0, 0, 0, .15); border-radius: 4px; color: #aaa; display: inline-block; font-size: 11px !important; font-weight: 300; line-height: 16px; padding: 4px 8px; text-decoration: none; text-transform: uppercase; } a.colab-button:hover { background: white; border-color: rgba(0, 0, 0, .2); color: #666; } a.colab-button span { background: url(/images/colab_logo_button.svg) no-repeat 1px 1px / 20px; border-radius: 4px; display: inline-block; padding-left: 24px; text-decoration: none; } @media screen and (max-width: 600px) { .tfo-notebook-buttons td { display: block; } } /* guide and tutorials landing page cards and sections */ .tfo-landing-page-card { padding: 16px; box-shadow: 0 0 36px rgba(0,0,0,0.1); border-radius: 10px; } /* Page section headings */ .tfo-landing-page-heading h2, h2.tfo-landing-page-heading { font-family: "Google Sans", sans-serif; color: #425066; font-size: 30px; font-weight: 700; line-height: 40px; } /* Item title headings */ .tfo-landing-page-heading h3, h3.tfo-landing-page-heading, .tfo-landing-page-card h3, h3.tfo-landing-page-card { font-family: "Google Sans", sans-serif; color: #425066; font-size: 20px; font-weight: 500; line-height: 26px; } /* List of tutorials notebooks for subsites */ .tfo-landing-page-resources-ul { padding-left: 15px } .tfo-landing-page-resources-ul > li { margin: 6px 0; } /* Temporary fix to hide product description in header on landing pages */ devsite-header .devsite-product-description { display: none; } </style> <div class="devsite-article-meta nocontent" role="navigation"> <ul class="devsite-breadcrumb-list" aria-label="Breadcrumb"> <li class="devsite-breadcrumb-item "> <a href="https://www.tensorflow.org/" class="devsite-breadcrumb-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Breadcrumbs" data-value="1" track-type="globalNav" track-name="breadcrumb" track-metadata-position="1" track-metadata-eventdetail="TensorFlow" > TensorFlow </a> </li> <li class="devsite-breadcrumb-item "> <div class="devsite-breadcrumb-guillemet material-icons" aria-hidden="true"></div> <a href="https://www.tensorflow.org/resources" class="devsite-breadcrumb-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Breadcrumbs" data-value="2" track-type="globalNav" track-name="breadcrumb" track-metadata-position="2" track-metadata-eventdetail="" > Resources </a> </li> <li class="devsite-breadcrumb-item "> <div class="devsite-breadcrumb-guillemet material-icons" aria-hidden="true"></div> <a href="https://www.tensorflow.org/text" class="devsite-breadcrumb-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Breadcrumbs" data-value="3" track-type="globalNav" track-name="breadcrumb" track-metadata-position="3" track-metadata-eventdetail="Text" > Text </a> </li> <li class="devsite-breadcrumb-item "> <div class="devsite-breadcrumb-guillemet material-icons" aria-hidden="true"></div> <a href="https://www.tensorflow.org/text/tutorials" class="devsite-breadcrumb-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Breadcrumbs" data-value="4" track-type="globalNav" track-name="breadcrumb" track-metadata-position="4" track-metadata-eventdetail="" > Tutorials </a> </li> </ul> <devsite-thumb-rating position="header"> </devsite-thumb-rating> </div> <h1 class="devsite-page-title" tabindex="-1"> Neural machine translation with attention </h1> <devsite-feature-tooltip ack-key="AckCollectionsBookmarkTooltipDismiss" analytics-category="Site-Wide Custom Events" analytics-action-show="Callout Profile displayed" analytics-action-close="Callout Profile dismissed" analytics-label="Create Collection Callout" class="devsite-page-bookmark-tooltip nocontent" dismiss-button="true" id="devsite-collections-dropdown" dismiss-button-text="Dismiss" close-button-text="Got it"> <devsite-bookmark></devsite-bookmark> <span slot="popout-heading"> Stay organized with collections </span> <span slot="popout-contents"> Save and categorize content based on your preferences. </span> </devsite-feature-tooltip> <div class="devsite-page-title-meta"><devsite-view-release-notes></devsite-view-release-notes></div> <devsite-toc class="devsite-nav" depth="2" devsite-toc-embedded > </devsite-toc> <div class="devsite-article-body clearfix "> <p><devsite-mathjax config="TeX-AMS-MML_SVG"></devsite-mathjax> </p>  <table class="tfo-notebook-buttons" align="left"> <td> <a target="_blank" href="https://www.tensorflow.org/text/tutorials/nmt_with_attention"> <img src="https://www.tensorflow.org/images/tf_logo_32px.png"> View on TensorFlow.org</a> </td> <td> <a target="_blank" href="https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/nmt_with_attention.ipynb"> <img src="https://www.tensorflow.org/images/colab_logo_32px.png"> Run in Google Colab</a> </td> <td> <a target="_blank" href="https://github.com/tensorflow/text/blob/master/docs/tutorials/nmt_with_attention.ipynb"> <img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png"> View source on GitHub</a> </td> <td> <a href="https://storage.googleapis.com/tensorflow_docs/text/docs/tutorials/nmt_with_attention.ipynb"><img src="https://www.tensorflow.org/images/download_logo_32px.png">Download notebook</a> </td> </table> <p>This tutorial demonstrates how to train a sequence-to-sequence (seq2seq) model for Spanish-to-English translation roughly based on <a href="https://arxiv.org/abs/1508.04025v5">Effective Approaches to Attention-based Neural Machine Translation</a> (Luong et al., 2015). </p> <table> <tr> <td> <img width="400" src="https://www.tensorflow.org/images/tutorials/transformer/RNN%2Battention-words-spa.png"> </td> </tr> <tr> <th colspan=1>This tutorial: An encoder/decoder connected by attention.</th> <tr> </table> <p>While this architecture is somewhat outdated, it is still a very useful project to work through to get a deeper understanding of sequence-to-sequence models and attention mechanisms (before going on to <a href="/text/tutorials/transformer">Transformers</a>).</p> <p>This example assumes some knowledge of TensorFlow fundamentals below the level of a Keras layer:</p> <ul> <li><a href="https://www.tensorflow.org/guide/tensor">Working with tensors</a> directly</li> <li><a href="https://www.tensorflow.org/guide/keras/custom_layers_and_models">Writing custom <code translate="no" dir="ltr">keras.Model</code>s and <code translate="no" dir="ltr">keras.layers</code></a></li> </ul> <p>After training the model in this notebook, you will be able to input a Spanish sentence, such as "<em>¿todavia estan en casa?</em>", and return the English translation: "<em>are you still at home?</em>"</p> <p>The resulting model is exportable as a <a href="https://www.tensorflow.org/api_docs/python/tf/saved_model"><code translate="no" dir="ltr">tf.saved_model</code></a>, so it can be used in other TensorFlow environments.</p> <p>The translation quality is reasonable for a toy example, but the generated attention plot is perhaps more interesting. This shows which parts of the input sentence has the model's attention while translating:</p> <p><img src="https://tensorflow.org/images/spanish-english.png" alt="spanish-english attention plot"></p> <aside class="note"><strong>Note:</strong><span> This example takes approximately 10 minutes to run.</span></aside> <h2 id="setup" data-text="Setup" tabindex="-1">Setup</h2> <pre class="prettyprint lang-bsh" translate="no" dir="ltr"> <code class='devsite-terminal' translate="no" dir="ltr">pip install "tensorflow-text>=2.11"</code> <code class='devsite-terminal' translate="no" dir="ltr">pip install einops</code> </pre> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">import numpy as np import typing from typing import Any, Tuple import einops import matplotlib.pyplot as plt import matplotlib.ticker as ticker import tensorflow as tf import tensorflow_text as tf_text </code></pre> <p>This tutorial uses a lot of low level API's where it's easy to get shapes wrong. This class is used to check shapes throughout the tutorial.</p>  <p><section class="expandable"> <button type="button" class="button-red button expand-control">Toggle code</button></p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">class ShapeChecker(): def __init__(self): # Keep a cache of every axis-name seen self.shapes = {} def __call__(self, tensor, names, broadcast=False): if not tf.executing_eagerly(): return parsed = einops.parse_shape(tensor, names) for name, new_dim in parsed.items(): old_dim = self.shapes.get(name, None) if (broadcast and new_dim == 1): continue if old_dim is None: # If the axis name is new, add its length to the cache. self.shapes[name] = new_dim continue if new_dim != old_dim: raise ValueError(f"Shape mismatch for dimension: '{name}'\n" f" found: {new_dim}\n" f" expected: {old_dim}\n") </code></pre> <p></section></p> <h2 id="the_data" data-text="The data" tabindex="-1">The data</h2> <p>The tutorial uses a language dataset provided by <a href="http://www.manythings.org/anki/">Anki</a>. This dataset contains language translation pairs in the format:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">May I borrow this book? ¿Puedo tomar prestado este libro? </code></pre> <p>They have a variety of languages available, but this example uses the English-Spanish dataset.</p> <h3 id="download_and_prepare_the_dataset" data-text="Download and prepare the dataset" tabindex="-1">Download and prepare the dataset</h3> <p>For convenience, a copy of this dataset is hosted on Google Cloud, but you can also download your own copy. After downloading the dataset, here are the steps you need to take to prepare the data:</p> <ol> <li>Add a <em>start</em> and <em>end</em> token to each sentence.</li> <li>Clean the sentences by removing special characters.</li> <li>Create a word index and reverse word index (dictionaries mapping from word → id and id → word).</li> <li>Pad each sentence to a maximum length.</li> </ol> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr"># Download the file import pathlib path_to_zip = tf.keras.utils.get_file( 'spa-eng.zip', origin='http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip', extract=True) path_to_file = pathlib.Path(path_to_zip).parent/'spa-eng/spa.txt' </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">def load_data(path): text = path.read_text(encoding='utf-8') lines = text.splitlines() pairs = [line.split('\t') for line in lines] context = np.array([context for target, context in pairs]) target = np.array([target for target, context in pairs]) return target, context </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">target_raw, context_raw = load_data(path_to_file) print(context_raw[-1]) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">print(target_raw[-1]) </code></pre> <h3 id="create_a_tfdata_dataset" data-text="Create a tf.data dataset" tabindex="-1">Create a tf.data dataset</h3> <p>From these arrays of strings you can create a <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset"><code translate="no" dir="ltr">tf.data.Dataset</code></a> of strings that shuffles and batches them efficiently:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">BUFFER_SIZE = len(context_raw) BATCH_SIZE = 64 is_train = np.random.uniform(size=(len(target_raw),)) < 0.8 train_raw = ( tf.data.Dataset .from_tensor_slices((context_raw[is_train], target_raw[is_train])) .shuffle(BUFFER_SIZE) .batch(BATCH_SIZE)) val_raw = ( tf.data.Dataset .from_tensor_slices((context_raw[~is_train], target_raw[~is_train])) .shuffle(BUFFER_SIZE) .batch(BATCH_SIZE)) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">for example_context_strings, example_target_strings in train_raw.take(1): print(example_context_strings[:5]) print() print(example_target_strings[:5]) break </code></pre> <h3 id="text_preprocessing" data-text="Text preprocessing" tabindex="-1">Text preprocessing</h3> <p>One of the goals of this tutorial is to build a model that can be exported as a <a href="https://www.tensorflow.org/api_docs/python/tf/saved_model"><code translate="no" dir="ltr">tf.saved_model</code></a>. To make that exported model useful it should take <a href="https://www.tensorflow.org/api_docs/python/tf#string"><code translate="no" dir="ltr">tf.string</code></a> inputs, and return <a href="https://www.tensorflow.org/api_docs/python/tf#string"><code translate="no" dir="ltr">tf.string</code></a> outputs: All the text processing happens inside the model. Mainly using a <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization"><code translate="no" dir="ltr">layers.TextVectorization</code></a> layer.</p> <h4 id="standardization" data-text="Standardization" tabindex="-1">Standardization</h4> <p>The model is dealing with multilingual text with a limited vocabulary. So it will be important to standardize the input text.</p> <p>The first step is Unicode normalization to split accented characters and replace compatibility characters with their ASCII equivalents.</p> <p>The <code translate="no" dir="ltr">tensorflow_text</code> package contains a unicode normalize operation:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">example_text = tf.constant('¿Todavía está en casa?') print(example_text.numpy()) print(tf_text.normalize_utf8(example_text, 'NFKD').numpy()) </code></pre> <p>Unicode normalization will be the first step in the text standardization function:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">def tf_lower_and_split_punct(text): # Split accented characters. text = tf_text.normalize_utf8(text, 'NFKD') text = tf.strings.lower(text) # Keep space, a to z, and select punctuation. text = tf.strings.regex_replace(text, '[^ a-z.?!,¿]', '') # Add spaces around punctuation. text = tf.strings.regex_replace(text, '[.?!,¿]', r' \0 ') # Strip whitespace. text = tf.strings.strip(text) text = tf.strings.join(['[START]', text, '[END]'], separator=' ') return text </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">print(example_text.numpy().decode()) print(tf_lower_and_split_punct(example_text).numpy().decode()) </code></pre> <h4 id="text_vectorization" data-text="Text Vectorization" tabindex="-1">Text Vectorization</h4> <p>This standardization function will be wrapped up in a <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization"><code translate="no" dir="ltr">tf.keras.layers.TextVectorization</code></a> layer which will handle the vocabulary extraction and conversion of input text to sequences of tokens.</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">max_vocab_size = 5000 context_text_processor = tf.keras.layers.TextVectorization( standardize=tf_lower_and_split_punct, max_tokens=max_vocab_size, ragged=True) </code></pre> <p>The <code translate="no" dir="ltr">TextVectorization</code> layer and many other <a href="https://www.tensorflow.org/guide/keras/preprocessing_layers">Keras preprocessing layers</a> have an <code translate="no" dir="ltr">adapt</code> method. This method reads one epoch of the training data, and works a lot like <a href="https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit"><code translate="no" dir="ltr">Model.fit</code></a>. This <code translate="no" dir="ltr">adapt</code> method initializes the layer based on the data. Here it determines the vocabulary:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">context_text_processor.adapt(train_raw.map(lambda context, target: context)) # Here are the first 10 words from the vocabulary: context_text_processor.get_vocabulary()[:10] </code></pre> <p>That's the Spanish <code translate="no" dir="ltr">TextVectorization</code> layer, now build and <code translate="no" dir="ltr">.adapt()</code> the English one:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">target_text_processor = tf.keras.layers.TextVectorization( standardize=tf_lower_and_split_punct, max_tokens=max_vocab_size, ragged=True) target_text_processor.adapt(train_raw.map(lambda context, target: target)) target_text_processor.get_vocabulary()[:10] </code></pre> <p>Now these layers can convert a batch of strings into a batch of token IDs:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">example_tokens = context_text_processor(example_context_strings) example_tokens[:3, :] </code></pre> <p>The <code translate="no" dir="ltr">get_vocabulary</code> method can be used to convert token IDs back to text:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">context_vocab = np.array(context_text_processor.get_vocabulary()) tokens = context_vocab[example_tokens[0].numpy()] ' '.join(tokens) </code></pre> <p>The returned token IDs are zero-padded. This can easily be turned into a mask:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">plt.subplot(1, 2, 1) plt.pcolormesh(example_tokens.to_tensor()) plt.title('Token IDs') plt.subplot(1, 2, 2) plt.pcolormesh(example_tokens.to_tensor() != 0) plt.title('Mask') </code></pre> <h3 id="process_the_dataset" data-text="Process the dataset" tabindex="-1">Process the dataset</h3> <p>The <code translate="no" dir="ltr">process_text</code> function below converts the <code translate="no" dir="ltr">Datasets</code> of strings, into 0-padded tensors of token IDs. It also converts from a <code translate="no" dir="ltr">(context, target)</code> pair to an <code translate="no" dir="ltr">((context, target_in), target_out)</code> pair for training with <a href="https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit"><code translate="no" dir="ltr">keras.Model.fit</code></a>. Keras expects <code translate="no" dir="ltr">(inputs, labels)</code> pairs, the inputs are the <code translate="no" dir="ltr">(context, target_in)</code> and the labels are <code translate="no" dir="ltr">target_out</code>. The difference between <code translate="no" dir="ltr">target_in</code> and <code translate="no" dir="ltr">target_out</code> is that they are shifted by one step relative to eachother, so that at each location the label is the next token.</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">def process_text(context, target): context = context_text_processor(context).to_tensor() target = target_text_processor(target) targ_in = target[:,:-1].to_tensor() targ_out = target[:,1:].to_tensor() return (context, targ_in), targ_out train_ds = train_raw.map(process_text, tf.data.AUTOTUNE) val_ds = val_raw.map(process_text, tf.data.AUTOTUNE) </code></pre> <p>Here is the first sequence of each, from the first batch:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">for (ex_context_tok, ex_tar_in), ex_tar_out in train_ds.take(1): print(ex_context_tok[0, :10].numpy()) print() print(ex_tar_in[0, :10].numpy()) print(ex_tar_out[0, :10].numpy()) </code></pre> <h2 id="the_encoderdecoder" data-text="The encoder/decoder" tabindex="-1">The encoder/decoder</h2> <p>The following diagrams shows an overview of the model. In both the encoder is on the left, the decoder is on the right. At each time-step the decoder's output is combined with the encoder's output, to predict the next word. </p> <p>The original [left] contains a few extra connections that are intentionally omitted from this tutorial's model [right], as they are generally unnecessary, and difficult to implement. Those missing connections are:</p> <ol> <li>Feeding the state from the encoder's RNN to the decoder's RNN</li> <li>Feeding the attention output back to the RNN's input.</li> </ol> <table> <tr> <td> <img width="500" src="https://www.tensorflow.org/images/seq2seq/attention_mechanism.jpg"> </td> <td> <img width="380" src="https://www.tensorflow.org/images/tutorials/transformer/RNN+attention.png"> </td> </tr> <tr> <th colspan=1>The original from <a href=https://arxiv.org/abs/1508.04025v5>Effective Approaches to Attention-based Neural Machine Translation</a></th> <th colspan=1>This tutorial's model</th> <tr> </table> <p>Before getting into it define constants for the model:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">UNITS = 256 </code></pre> <h3 id="the_encoder" data-text="The encoder" tabindex="-1">The encoder</h3> <p>The goal of the encoder is to process the context sequence into a sequence of vectors that are useful for the decoder as it attempts to predict the next output for each timestep. Since the context sequence is constant, there is no restriction on how information can flow in the encoder, so use a bidirectional-RNN to do the processing:</p> <table> <tr> <td> <img width="500" src="https://tensorflow.org/images/tutorials/transformer/RNN-bidirectional.png"> </td> </tr> <tr> <th>A bidirectional RNN</th> <tr> </table> <p>The encoder:</p> <ol> <li>Takes a list of token IDs (from <code translate="no" dir="ltr">context_text_processor</code>).</li> <li>Looks up an embedding vector for each token (Using a <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding"><code translate="no" dir="ltr">layers.Embedding</code></a>).</li> <li>Processes the embeddings into a new sequence (Using a bidirectional <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/GRU"><code translate="no" dir="ltr">layers.GRU</code></a>).</li> <li>Returns the processed sequence. This will be passed to the attention head.</li> </ol> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">class Encoder(tf.keras.layers.Layer): def __init__(self, text_processor, units): super(Encoder, self).__init__() self.text_processor = text_processor self.vocab_size = text_processor.vocabulary_size() self.units = units # The embedding layer converts tokens to vectors self.embedding = tf.keras.layers.Embedding(self.vocab_size, units, mask_zero=True) # The RNN layer processes those vectors sequentially. self.rnn = tf.keras.layers.Bidirectional( merge_mode='sum', layer=tf.keras.layers.GRU(units, # Return the sequence and state return_sequences=True, recurrent_initializer='glorot_uniform')) def call(self, x): shape_checker = ShapeChecker() shape_checker(x, 'batch s') # 2. The embedding layer looks up the embedding vector for each token. x = self.embedding(x) shape_checker(x, 'batch s units') # 3. The GRU processes the sequence of embeddings. x = self.rnn(x) shape_checker(x, 'batch s units') # 4. Returns the new sequence of embeddings. return x def convert_input(self, texts): texts = tf.convert_to_tensor(texts) if len(texts.shape) == 0: texts = tf.convert_to_tensor(texts)[tf.newaxis] context = self.text_processor(texts).to_tensor() context = self(context) return context </code></pre> <p>Try it out:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr"># Encode the input sequence. encoder = Encoder(context_text_processor, UNITS) ex_context = encoder(ex_context_tok) print(f'Context tokens, shape (batch, s): {ex_context_tok.shape}') print(f'Encoder output, shape (batch, s, units): {ex_context.shape}') </code></pre> <h3 id="the_attention_layer" data-text="The attention layer" tabindex="-1">The attention layer</h3> <p>The attention layer lets the decoder access the information extracted by the encoder. It computes a vector from the entire context sequence, and adds that to the decoder's output. </p> <p>The simplest way you could calculate a single vector from the entire sequence would be to take the average across the sequence (<a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalAveragePooling1D"><code translate="no" dir="ltr">layers.GlobalAveragePooling1D</code></a>). An attention layer is similar, but calculates a <strong>weighted</strong> average across the context sequence. Where the weights are calculated from the combination of context and "query" vectors.</p> <table> <tr> <td> <img width="500" src="https://www.tensorflow.org/images/tutorials/transformer/CrossAttention-new-full.png"> </td> </tr> <tr> <th colspan=1>The attention layer</th> <tr> </table> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">class CrossAttention(tf.keras.layers.Layer): def __init__(self, units, **kwargs): super().__init__() self.mha = tf.keras.layers.MultiHeadAttention(key_dim=units, num_heads=1, **kwargs) self.layernorm = tf.keras.layers.LayerNormalization() self.add = tf.keras.layers.Add() def call(self, x, context): shape_checker = ShapeChecker() shape_checker(x, 'batch t units') shape_checker(context, 'batch s units') attn_output, attn_scores = self.mha( query=x, value=context, return_attention_scores=True) shape_checker(x, 'batch t units') shape_checker(attn_scores, 'batch heads t s') # Cache the attention scores for plotting later. attn_scores = tf.reduce_mean(attn_scores, axis=1) shape_checker(attn_scores, 'batch t s') self.last_attention_weights = attn_scores x = self.add([x, attn_output]) x = self.layernorm(x) return x </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">attention_layer = CrossAttention(UNITS) # Attend to the encoded tokens embed = tf.keras.layers.Embedding(target_text_processor.vocabulary_size(), output_dim=UNITS, mask_zero=True) ex_tar_embed = embed(ex_tar_in) result = attention_layer(ex_tar_embed, ex_context) print(f'Context sequence, shape (batch, s, units): {ex_context.shape}') print(f'Target sequence, shape (batch, t, units): {ex_tar_embed.shape}') print(f'Attention result, shape (batch, t, units): {result.shape}') print(f'Attention weights, shape (batch, t, s): {attention_layer.last_attention_weights.shape}') </code></pre> <p>The attention weights will sum to <code translate="no" dir="ltr">1</code> over the context sequence, at each location in the target sequence.</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">attention_layer.last_attention_weights[0].numpy().sum(axis=-1) </code></pre> <p>Here are the attention weights across the context sequences at <code translate="no" dir="ltr">t=0</code>:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">attention_weights = attention_layer.last_attention_weights mask=(ex_context_tok != 0).numpy() plt.subplot(1, 2, 1) plt.pcolormesh(mask*attention_weights[:, 0, :]) plt.title('Attention weights') plt.subplot(1, 2, 2) plt.pcolormesh(mask) plt.title('Mask'); </code></pre> <p>Because of the small-random initialization the attention weights are initially all close to <code translate="no" dir="ltr">1/(sequence_length)</code>. The model will learn to make these less uniform as training progresses.</p> <h3 id="the_decoder" data-text="The decoder" tabindex="-1">The decoder</h3> <p>The decoder's job is to generate predictions for the next token at each location in the target sequence.</p> <ol> <li>It looks up embeddings for each token in the target sequence.</li> <li>It uses an RNN to process the target sequence, and keep track of what it has generated so far.</li> <li>It uses RNN output as the "query" to the attention layer, when attending to the encoder's output.</li> <li>At each location in the output it predicts the next token.</li> </ol> <p>When training, the model predicts the next word at each location. So it's important that the information only flows in one direction through the model. The decoder uses a unidirectional (not bidirectional) RNN to process the target sequence.</p> <p>When running inference with this model it produces one word at a time, and those are fed back into the model.</p> <table> <tr> <td> <img width="500" src="https://tensorflow.org/images/tutorials/transformer/RNN.png"> </td> </tr> <tr> <th>A unidirectional RNN</th> <tr> </table> <p>Here is the <code translate="no" dir="ltr">Decoder</code> class' initializer. The initializer creates all the necessary layers.</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">class Decoder(tf.keras.layers.Layer): @classmethod def add_method(cls, fun): setattr(cls, fun.__name__, fun) return fun def __init__(self, text_processor, units): super(Decoder, self).__init__() self.text_processor = text_processor self.vocab_size = text_processor.vocabulary_size() self.word_to_id = tf.keras.layers.StringLookup( vocabulary=text_processor.get_vocabulary(), mask_token='', oov_token='[UNK]') self.id_to_word = tf.keras.layers.StringLookup( vocabulary=text_processor.get_vocabulary(), mask_token='', oov_token='[UNK]', invert=True) self.start_token = self.word_to_id('[START]') self.end_token = self.word_to_id('[END]') self.units = units # 1. The embedding layer converts token IDs to vectors self.embedding = tf.keras.layers.Embedding(self.vocab_size, units, mask_zero=True) # 2. The RNN keeps track of what's been generated so far. self.rnn = tf.keras.layers.GRU(units, return_sequences=True, return_state=True, recurrent_initializer='glorot_uniform') # 3. The RNN output will be the query for the attention layer. self.attention = CrossAttention(units) # 4. This fully connected layer produces the logits for each # output token. self.output_layer = tf.keras.layers.Dense(self.vocab_size) </code></pre> <h4 id="training" data-text="Training" tabindex="-1">Training</h4> <p>Next, the <code translate="no" dir="ltr">call</code> method, takes 3 arguments:</p> <ul> <li><code translate="no" dir="ltr">inputs</code> - a <code translate="no" dir="ltr">context, x</code> pair where: <ul> <li><code translate="no" dir="ltr">context</code> - is the context from the encoder's output.</li> <li><code translate="no" dir="ltr">x</code> - is the target sequence input.</li> </ul></li> <li><code translate="no" dir="ltr">state</code> - Optional, the previous <code translate="no" dir="ltr">state</code> output from the decoder (the internal state of the decoder's RNN). Pass the state from a previous run to continue generating text where you left off.</li> <li><code translate="no" dir="ltr">return_state</code> - [Default: False] - Set this to <code translate="no" dir="ltr">True</code> to return the RNN state. </li> </ul> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">@Decoder.add_method def call(self, context, x, state=None, return_state=False): shape_checker = ShapeChecker() shape_checker(x, 'batch t') shape_checker(context, 'batch s units') # 1. Lookup the embeddings x = self.embedding(x) shape_checker(x, 'batch t units') # 2. Process the target sequence. x, state = self.rnn(x, initial_state=state) shape_checker(x, 'batch t units') # 3. Use the RNN output as the query for the attention over the context. x = self.attention(x, context) self.last_attention_weights = self.attention.last_attention_weights shape_checker(x, 'batch t units') shape_checker(self.last_attention_weights, 'batch t s') # Step 4. Generate logit predictions for the next token. logits = self.output_layer(x) shape_checker(logits, 'batch t target_vocab_size') if return_state: return logits, state else: return logits </code></pre> <p>That will be sufficient for training. Create an instance of the decoder to test out:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">decoder = Decoder(target_text_processor, UNITS) </code></pre> <p>In training you'll use the decoder like this:</p> <p>Given the context and target tokens, for each target token it predicts the next target token. </p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">logits = decoder(ex_context, ex_tar_in) print(f'encoder output shape: (batch, s, units) {ex_context.shape}') print(f'input target tokens shape: (batch, t) {ex_tar_in.shape}') print(f'logits shape shape: (batch, target_vocabulary_size) {logits.shape}') </code></pre> <h4 id="inference" data-text="Inference" tabindex="-1">Inference</h4> <p>To use it for inference you'll need a couple more methods:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">@Decoder.add_method def get_initial_state(self, context): batch_size = tf.shape(context)[0] start_tokens = tf.fill([batch_size, 1], self.start_token) done = tf.zeros([batch_size, 1], dtype=tf.bool) embedded = self.embedding(start_tokens) return start_tokens, done, self.rnn.get_initial_state(embedded)[0] </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">@Decoder.add_method def tokens_to_text(self, tokens): words = self.id_to_word(tokens) result = tf.strings.reduce_join(words, axis=-1, separator=' ') result = tf.strings.regex_replace(result, '^ *\[START\] *', '') result = tf.strings.regex_replace(result, ' *\[END\] *$', '') return result </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">@Decoder.add_method def get_next_token(self, context, next_token, done, state, temperature = 0.0): logits, state = self( context, next_token, state = state, return_state=True) if temperature == 0.0: next_token = tf.argmax(logits, axis=-1) else: logits = logits[:, -1, :]/temperature next_token = tf.random.categorical(logits, num_samples=1) # If a sequence produces an `end_token`, set it `done` done = done | (next_token == self.end_token) # Once a sequence is done it only produces 0-padding. next_token = tf.where(done, tf.constant(0, dtype=tf.int64), next_token) return next_token, done, state </code></pre> <p>With those extra functions, you can write a generation loop:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr"># Setup the loop variables. next_token, done, state = decoder.get_initial_state(ex_context) tokens = [] for n in range(10): # Run one step. next_token, done, state = decoder.get_next_token( ex_context, next_token, done, state, temperature=1.0) # Add the token to the output. tokens.append(next_token) # Stack all the tokens together. tokens = tf.concat(tokens, axis=-1) # (batch, t) # Convert the tokens back to a a string result = decoder.tokens_to_text(tokens) result[:3].numpy() </code></pre> <p>Since the model's untrained, it outputs items from the vocabulary almost uniformly at random.</p> <h2 id="the_model" data-text="The model" tabindex="-1">The model</h2> <p>Now that you have all the model components, combine them to build the model for training:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">class Translator(tf.keras.Model): @classmethod def add_method(cls, fun): setattr(cls, fun.__name__, fun) return fun def __init__(self, units, context_text_processor, target_text_processor): super().__init__() # Build the encoder and decoder encoder = Encoder(context_text_processor, units) decoder = Decoder(target_text_processor, units) self.encoder = encoder self.decoder = decoder def call(self, inputs): context, x = inputs context = self.encoder(context) logits = self.decoder(context, x) #TODO(b/250038731): remove this try: # Delete the keras mask, so keras doesn't scale the loss+accuracy. del logits._keras_mask except AttributeError: pass return logits </code></pre> <p>During training the model will be used like this:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">model = Translator(UNITS, context_text_processor, target_text_processor) logits = model((ex_context_tok, ex_tar_in)) print(f'Context tokens, shape: (batch, s, units) {ex_context_tok.shape}') print(f'Target tokens, shape: (batch, t) {ex_tar_in.shape}') print(f'logits, shape: (batch, t, target_vocabulary_size) {logits.shape}') </code></pre> <h3 id="train" data-text="Train" tabindex="-1">Train</h3> <p>For training, you'll want to implement your own masked loss and accuracy functions:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">def masked_loss(y_true, y_pred): # Calculate the loss for each item in the batch. loss_fn = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True, reduction='none') loss = loss_fn(y_true, y_pred) # Mask off the losses on padding. mask = tf.cast(y_true != 0, loss.dtype) loss *= mask # Return the total. return tf.reduce_sum(loss)/tf.reduce_sum(mask) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">def masked_acc(y_true, y_pred): # Calculate the loss for each item in the batch. y_pred = tf.argmax(y_pred, axis=-1) y_pred = tf.cast(y_pred, y_true.dtype) match = tf.cast(y_true == y_pred, tf.float32) mask = tf.cast(y_true != 0, tf.float32) return tf.reduce_sum(match)/tf.reduce_sum(mask) </code></pre> <p>Configure the model for training:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">model.compile(optimizer='adam', loss=masked_loss, metrics=[masked_acc, masked_loss]) </code></pre> <p>The model is randomly initialized, and should give roughly uniform output probabilities. So it's easy to predict what the initial values of the metrics should be:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">vocab_size = 1.0 * target_text_processor.vocabulary_size() {"expected_loss": tf.math.log(vocab_size).numpy(), "expected_acc": 1/vocab_size} </code></pre> <p>That should roughly match the values returned by running a few steps of evaluation:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">model.evaluate(val_ds, steps=20, return_dict=True) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">history = model.fit( train_ds.repeat(), epochs=100, steps_per_epoch = 100, validation_data=val_ds, validation_steps = 20, callbacks=[ tf.keras.callbacks.EarlyStopping(patience=3)]) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">plt.plot(history.history['loss'], label='loss') plt.plot(history.history['val_loss'], label='val_loss') plt.ylim([0, max(plt.ylim())]) plt.xlabel('Epoch #') plt.ylabel('CE/token') plt.legend() </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">plt.plot(history.history['masked_acc'], label='accuracy') plt.plot(history.history['val_masked_acc'], label='val_accuracy') plt.ylim([0, max(plt.ylim())]) plt.xlabel('Epoch #') plt.ylabel('CE/token') plt.legend() </code></pre> <h3 id="translate" data-text="Translate" tabindex="-1">Translate</h3> <p>Now that the model is trained, implement a function to execute the full <code translate="no" dir="ltr">text => text</code> translation. This code is basically identical to the <a href="#inference">inference example</a> in the <a href="#the_decoder">decoder section</a>, but this also captures the attention weights.</p>  <p><section class="expandable"> <button type="button" class="button-red button expand-control">Toggle code</button></p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">@Translator.add_method def translate(self, texts, *, max_length=50, temperature=0.0): # Process the input texts context = self.encoder.convert_input(texts) batch_size = tf.shape(texts)[0] # Setup the loop inputs tokens = [] attention_weights = [] next_token, done, state = self.decoder.get_initial_state(context) for _ in range(max_length): # Generate the next token next_token, done, state = self.decoder.get_next_token( context, next_token, done, state, temperature) # Collect the generated tokens tokens.append(next_token) attention_weights.append(self.decoder.last_attention_weights) if tf.executing_eagerly() and tf.reduce_all(done): break # Stack the lists of tokens and attention weights. tokens = tf.concat(tokens, axis=-1) # t*[(batch 1)] -> (batch, t) self.last_attention_weights = tf.concat(attention_weights, axis=1) # t*[(batch 1 s)] -> (batch, t s) result = self.decoder.tokens_to_text(tokens) return result </code></pre> <p></section></p> <p>Here are the two helper methods, used above, to convert tokens to text, and to get the next token:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">result = model.translate(['¿Todavía está en casa?']) # Are you still home result[0].numpy().decode() </code></pre> <p>Use that to generate the attention plot:</p>  <p><section class="expandable"> <button type="button" class="button-red button expand-control">Toggle code</button></p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">@Translator.add_method def plot_attention(self, text, **kwargs): assert isinstance(text, str) output = self.translate([text], **kwargs) output = output[0].numpy().decode() attention = self.last_attention_weights[0] context = tf_lower_and_split_punct(text) context = context.numpy().decode().split() output = tf_lower_and_split_punct(output) output = output.numpy().decode().split()[1:] fig = plt.figure(figsize=(10, 10)) ax = fig.add_subplot(1, 1, 1) ax.matshow(attention, cmap='viridis', vmin=0.0) fontdict = {'fontsize': 14} ax.set_xticklabels([''] + context, fontdict=fontdict, rotation=90) ax.set_yticklabels([''] + output, fontdict=fontdict) ax.xaxis.set_major_locator(ticker.MultipleLocator(1)) ax.yaxis.set_major_locator(ticker.MultipleLocator(1)) ax.set_xlabel('Input text') ax.set_ylabel('Output text') </code></pre> <p></section></p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">model.plot_attention('¿Todavía está en casa?') # Are you still home </code></pre> <p>Translate a few more sentences and plot them:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time # This is my life. model.plot_attention('Esta es mi vida.') </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time # Try to find out.' model.plot_attention('Tratar de descubrir.') </code></pre> <p>The short sentences often work well, but if the input is too long the model literally loses focus and stops providing reasonable predictions. There are two main reasons for this:</p> <ol> <li>The model was trained with teacher-forcing feeding the correct token at each step, regardless of the model's predictions. The model could be made more robust if it were sometimes fed its own predictions.</li> <li>The model only has access to its previous output through the RNN state. If the RNN state looses track of where it was in the context sequence there's no way for the model to recover. <a href="/text/tutorials/transformer">Transformers</a> improve on this by letting the decoder look at what it has output so far.</li> </ol> <p>The raw data is sorted by length, so try translating the longest sequence:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">long_text = context_raw[-1] import textwrap print('Expected output:\n', '\n'.join(textwrap.wrap(target_raw[-1]))) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">model.plot_attention(long_text) </code></pre> <p>The <code translate="no" dir="ltr">translate</code> function works on batches, so if you have multiple texts to translate you can pass them all at once, which is much more efficient than translating them one at a time:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">inputs = [ 'Hace mucho frio aqui.', # "It's really cold here." 'Esta es mi vida.', # "This is my life." 'Su cuarto es un desastre.' # "His room is a mess" ] </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time for t in inputs: print(model.translate([t])[0].numpy().decode()) print() </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time result = model.translate(inputs) print(result[0].numpy().decode()) print(result[1].numpy().decode()) print(result[2].numpy().decode()) print() </code></pre> <p>So overall this text generation function mostly gets the job done, but so you've only used it here in python with eager execution. Let's try to export it next:</p> <h3 id="export" data-text="Export" tabindex="-1">Export</h3> <p>If you want to export this model you'll need to wrap the <code translate="no" dir="ltr">translate</code> method in a <a href="https://www.tensorflow.org/api_docs/python/tf/function"><code translate="no" dir="ltr">tf.function</code></a>. That implementation will get the job done:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">class Export(tf.Module): def __init__(self, model): self.model = model @tf.function(input_signature=[tf.TensorSpec(dtype=tf.string, shape=[None])]) def translate(self, inputs): return self.model.translate(inputs) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">export = Export(model) </code></pre> <p>Run the <a href="https://www.tensorflow.org/api_docs/python/tf/function"><code translate="no" dir="ltr">tf.function</code></a> once to compile it:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time _ = export.translate(tf.constant(inputs)) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time result = export.translate(tf.constant(inputs)) print(result[0].numpy().decode()) print(result[1].numpy().decode()) print(result[2].numpy().decode()) print() </code></pre> <p>Now that the function has been traced it can be exported using <a href="https://www.tensorflow.org/api_docs/python/tf/saved_model/save"><code translate="no" dir="ltr">saved_model.save</code></a>:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time tf.saved_model.save(export, 'translator', signatures={'serving_default': export.translate}) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time reloaded = tf.saved_model.load('translator') _ = reloaded.translate(tf.constant(inputs)) #warmup </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time result = reloaded.translate(tf.constant(inputs)) print(result[0].numpy().decode()) print(result[1].numpy().decode()) print(result[2].numpy().decode()) print() </code></pre> <h4 id="optional_use_a_dynamic_loop" data-text="[Optional] Use a dynamic loop" tabindex="-1">[Optional] Use a dynamic loop</h4> <p>It's worth noting that this initial implementation is not optimal. It uses a python loop:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">for _ in range(max_length): ... if tf.executing_eagerly() and tf.reduce_all(done): break </code></pre> <p>The python loop is relatively simple but when <a href="https://www.tensorflow.org/api_docs/python/tf/function"><code translate="no" dir="ltr">tf.function</code></a> converts this to a graph, it <strong>statically unrolls</strong> that loop. Unrolling the loop has two disadvantages:</p> <ol> <li>It makes <code translate="no" dir="ltr">max_length</code> copies of the loop body. So the generated graphs take longer to build, save and load.</li> <li>You have to choose a fixed value for the <code translate="no" dir="ltr">max_length</code>. </li> <li>You can't <code translate="no" dir="ltr">break</code> from a statically unrolled loop. The <a href="https://www.tensorflow.org/api_docs/python/tf/function"><code translate="no" dir="ltr">tf.function</code></a> version will run the full <code translate="no" dir="ltr">max_length</code> iterations on every call. That's why the <code translate="no" dir="ltr">break</code> only works with eager execution. This is still marginally faster than eager execution, but not as fast as it could be.</li> </ol> <p>To fix these shortcomings, the <code translate="no" dir="ltr">translate_dynamic</code> method, below, uses a tensorflow loop:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">for t in tf.range(max_length): ... if tf.reduce_all(done): break </code></pre> <p>It looks like a python loop, but when you use a tensor as the input to a <code translate="no" dir="ltr">for</code> loop (or the condition of a <code translate="no" dir="ltr">while</code> loop) <a href="https://www.tensorflow.org/api_docs/python/tf/function"><code translate="no" dir="ltr">tf.function</code></a> converts it to a dynamic loop using operations like <a href="https://www.tensorflow.org/api_docs/python/tf/while_loop"><code translate="no" dir="ltr">tf.while_loop</code></a>. </p> <p>There's no need for a <code translate="no" dir="ltr">max_length</code> here it's just in case the model gets stuck generating a loop like: <code translate="no" dir="ltr">the united states of the united states of the united states...</code>.</p> <p>On the down side, to accumulate tokens from this dynamic loop you can't just append them to a python <code translate="no" dir="ltr">list</code>, you need to use a <a href="https://www.tensorflow.org/api_docs/python/tf/TensorArray"><code translate="no" dir="ltr">tf.TensorArray</code></a>:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">tokens = tf.TensorArray(tf.int64, size=1, dynamic_size=True) ... for t in tf.range(max_length): ... tokens = tokens.write(t, next_token) # next_token shape is (batch, 1) ... tokens = tokens.stack() tokens = einops.rearrange(tokens, 't batch 1 -> batch t') </code></pre> <p>This version of the code can be quite a bit more efficient:</p>  <p><section class="expandable"> <button type="button" class="button-red button expand-control">Toggle code</button></p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">@Translator.add_method def translate(self, texts, *, max_length=500, temperature=tf.constant(0.0)): shape_checker = ShapeChecker() context = self.encoder.convert_input(texts) batch_size = tf.shape(context)[0] shape_checker(context, 'batch s units') next_token, done, state = self.decoder.get_initial_state(context) # initialize the accumulator tokens = tf.TensorArray(tf.int64, size=1, dynamic_size=True) for t in tf.range(max_length): # Generate the next token next_token, done, state = self.decoder.get_next_token( context, next_token, done, state, temperature) shape_checker(next_token, 'batch t1') # Collect the generated tokens tokens = tokens.write(t, next_token) # if all the sequences are done, break if tf.reduce_all(done): break # Convert the list of generated token ids to a list of strings. tokens = tokens.stack() shape_checker(tokens, 't batch t1') tokens = einops.rearrange(tokens, 't batch 1 -> batch t') shape_checker(tokens, 'batch t') text = self.decoder.tokens_to_text(tokens) shape_checker(text, 'batch') return text </code></pre> <p></section></p> <p>With eager execution this implementation performs on par with the original:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time result = model.translate(inputs) print(result[0].numpy().decode()) print(result[1].numpy().decode()) print(result[2].numpy().decode()) print() </code></pre> <p>But when you wrap it in a <a href="https://www.tensorflow.org/api_docs/python/tf/function"><code translate="no" dir="ltr">tf.function</code></a> you'll notice two differences.</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">class Export(tf.Module): def __init__(self, model): self.model = model @tf.function(input_signature=[tf.TensorSpec(dtype=tf.string, shape=[None])]) def translate(self, inputs): return self.model.translate(inputs) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">export = Export(model) </code></pre> <p>First, it's much quicker to trace, since it only creates one copy of the loop body:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time _ = export.translate(inputs) </code></pre> <p>The <a href="https://www.tensorflow.org/api_docs/python/tf/function"><code translate="no" dir="ltr">tf.function</code></a> is much faster than running with eager execution, and on small inputs it's often several times faster than the unrolled version, because it can break out of the loop.</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time result = export.translate(inputs) print(result[0].numpy().decode()) print(result[1].numpy().decode()) print(result[2].numpy().decode()) print() </code></pre> <p>So save this version as well:</p> <pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time tf.saved_model.save(export, 'dynamic_translator', signatures={'serving_default': export.translate}) </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time reloaded = tf.saved_model.load('dynamic_translator') _ = reloaded.translate(tf.constant(inputs)) #warmup </code></pre><pre class="prettyprint" translate="no" dir="ltr"><code translate="no" dir="ltr">%%time result = reloaded.translate(tf.constant(inputs)) print(result[0].numpy().decode()) print(result[1].numpy().decode()) print(result[2].numpy().decode()) print() </code></pre> <h2 id="next_steps" data-text="Next steps" tabindex="-1">Next steps</h2> <ul> <li><a href="http://www.manythings.org/anki/">Download a different dataset</a> to experiment with translations, for example, English to German, or English to French.</li> <li>Experiment with training on a larger dataset, or using more epochs.</li> <li>Try the <a href="/text/tutorials/transformer">transformer tutorial</a> which implements a similar translation task but uses transformer layers instead of RNNs. This version also uses a <a href="https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer"><code translate="no" dir="ltr">text.BertTokenizer</code></a> to implement word-piece tokenization.</li> <li>Visit the <a href="https://www.tensorflow.org/addons/tutorials/networks_seq2seq_nmt"><code translate="no" dir="ltr">tensorflow_addons.seq2seq</code> tutorial</a>, which demonstrates a higher-level functionality for implementing this sort of sequence-to-sequence model, such as <a href="https://www.tensorflow.org/addons/api_docs/python/tfa/seq2seq/BeamSearchDecoder"><code translate="no" dir="ltr">seq2seq.BeamSearchDecoder</code></a>.</li> </ul> </div> <devsite-thumb-rating position="footer"> </devsite-thumb-rating> <div class="devsite-floating-action-buttons"> </div> </article> <devsite-content-footer class="nocontent"> <p>Except as otherwise noted, the content of this page is licensed under the <a href="https://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 License</a>, and code samples are licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2.0 License</a>. For details, see the <a href="https://developers.google.com/site-policies">Google Developers Site Policies</a>. Java is a registered trademark of Oracle and/or its affiliates.</p> <p>Last updated 2024-05-31 UTC.</p> </devsite-content-footer> <devsite-notification > </devsite-notification> <div class="devsite-content-data"> <template class="devsite-content-data-template"> [[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2024-05-31 UTC."],[],[]] </template> </div> </devsite-content> </main> <devsite-footer-promos class="devsite-footer"> </devsite-footer-promos> <devsite-footer-linkboxes class="devsite-footer"> <nav class="devsite-footer-linkboxes nocontent" aria-label="Footer links"> <ul class="devsite-footer-linkboxes-list"> <li class="devsite-footer-linkbox "> <h3 class="devsite-footer-linkbox-heading no-link">Stay connected</h3> <ul class="devsite-footer-linkbox-list"> <li class="devsite-footer-linkbox-item"> <a href="//blog.tensorflow.org" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 1)" > Blog </a> </li> <li class="devsite-footer-linkbox-item"> <a href="//discuss.tensorflow.org" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 2)" > Forum </a> </li> <li class="devsite-footer-linkbox-item"> <a href="//github.com/tensorflow/" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 3)" > GitHub </a> </li> <li class="devsite-footer-linkbox-item"> <a href="//twitter.com/tensorflow" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 4)" > Twitter </a> </li> <li class="devsite-footer-linkbox-item"> <a href="//youtube.com/tensorflow" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 5)" > YouTube </a> </li> </ul> </li> <li class="devsite-footer-linkbox "> <h3 class="devsite-footer-linkbox-heading no-link">Support</h3> <ul class="devsite-footer-linkbox-list"> <li class="devsite-footer-linkbox-item"> <a href="//github.com/tensorflow/tensorflow/issues" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 1)" > Issue tracker </a> </li> <li class="devsite-footer-linkbox-item"> <a href="//github.com/tensorflow/tensorflow/blob/master/RELEASE.md" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 2)" > Release notes </a> </li> <li class="devsite-footer-linkbox-item"> <a href="//stackoverflow.com/questions/tagged/tensorflow" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 3)" > Stack Overflow </a> </li> <li class="devsite-footer-linkbox-item"> <a href="/extras/tensorflow_brand_guidelines.pdf" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 4)" > Brand guidelines </a> </li> <li class="devsite-footer-linkbox-item"> <a href="/about/bib" class="devsite-footer-linkbox-link gc-analytics-event" data-category="Site-Wide Custom Events" data-label="Footer Link (index 5)" > Cite TensorFlow </a> </li> </ul> </li> </ul> </nav> </devsite-footer-linkboxes> <devsite-footer-utility class="devsite-footer"> <div class="devsite-footer-utility nocontent"> <nav class="devsite-footer-utility-links" aria-label="Utility links"> <ul class="devsite-footer-utility-list"> <li class="devsite-footer-utility-item "> <a class="devsite-footer-utility-link gc-analytics-event" href="//policies.google.com/terms" data-category="Site-Wide Custom Events" data-label="Footer Terms link" > Terms </a> </li> <li class="devsite-footer-utility-item "> <a class="devsite-footer-utility-link gc-analytics-event" href="//policies.google.com/privacy" data-category="Site-Wide Custom Events" data-label="Footer Privacy link" > Privacy </a> </li> <li class="devsite-footer-utility-item glue-cookie-notification-bar-control"> <a class="devsite-footer-utility-link gc-analytics-event" href="#" data-category="Site-Wide Custom Events" data-label="Footer Manage cookies link" aria-hidden="true" > Manage cookies </a> </li> <li class="devsite-footer-utility-item devsite-footer-utility-button"> <span class="devsite-footer-utility-description">Sign up for the TensorFlow newsletter</span> <a class="devsite-footer-utility-link gc-analytics-event" href="//www.tensorflow.org/subscribe" data-category="Site-Wide Custom Events" data-label="Footer Subscribe link" > Subscribe </a> </li> </ul> <devsite-language-selector> <ul role="presentation"> <li role="presentation"> <a role="menuitem" lang="en" >English</a> </li> <li role="presentation"> <a role="menuitem" lang="es_419" >Español – América Latina</a> </li> <li role="presentation"> <a role="menuitem" lang="fr" >Français</a> </li> <li role="presentation"> <a role="menuitem" lang="id" >Indonesia</a> </li> <li role="presentation"> <a role="menuitem" lang="it" >Italiano</a> </li> <li role="presentation"> <a role="menuitem" lang="pl" >Polski</a> </li> <li role="presentation"> <a role="menuitem" lang="pt_br" >Português – Brasil</a> </li> <li role="presentation"> <a role="menuitem" lang="vi" >Tiếng Việt</a> </li> <li role="presentation"> <a role="menuitem" lang="tr" >Türkçe</a> </li> <li role="presentation"> <a role="menuitem" lang="ru" >Русский</a> </li> <li role="presentation"> <a role="menuitem" lang="he" >עברית</a> </li> <li role="presentation"> <a role="menuitem" lang="ar" >العربيّة</a> </li> <li role="presentation"> <a role="menuitem" lang="fa" >فارسی</a> </li> <li role="presentation"> <a role="menuitem" lang="hi" >हिंदी</a> </li> <li role="presentation"> <a role="menuitem" lang="bn" >বাংলা</a> </li> <li role="presentation"> <a role="menuitem" lang="th" >ภาษาไทย</a> </li> <li role="presentation"> <a role="menuitem" lang="zh_cn" >中文 – 简体</a> </li> <li role="presentation"> <a role="menuitem" lang="ja" >日本語</a> </li> <li role="presentation"> <a role="menuitem" lang="ko" >한국어</a> </li> </ul> </devsite-language-selector> </nav> </div> </devsite-footer-utility> <devsite-panel></devsite-panel> </section></section> <devsite-sitemask></devsite-sitemask> <devsite-snackbar></devsite-snackbar> <devsite-tooltip ></devsite-tooltip> <devsite-heading-link></devsite-heading-link> <devsite-analytics> <script type="application/json" analytics>[{"dimensions": {"dimension1": "Signed out", "dimension12": false, "dimension5": "en", "dimension4": "Text", "dimension3": false, "dimension6": "en"}, "gaid": "UA-69864048-1", "metrics": {"ratings_count": "metric2", "ratings_value": "metric1"}, "purpose": 0}]</script> <script type="application/json" tag-management>{"at": "True", "ga4": [], "ga4p": [], "gtm": [{"id": "GTM-MXSL34P", "purpose": 0}], "parameters": {"internalUser": "False", "language": {"machineTranslated": "False", "requested": "en", "served": "en"}, "pageType": "article", "projectName": "Text", "signedIn": "False", "tenant": "tensorflow", "recommendations": {"sourcePage": "", "sourceType": 0, "sourceRank": 0, "sourceIdenticalDescriptions": 0, "sourceTitleWords": 0, "sourceDescriptionWords": 0, "experiment": ""}, "experiment": {"ids": ""}}}</script> </devsite-analytics> <devsite-badger></devsite-badger> <script nonce="F6fq926JCbPJpaLyc9FRFa25GEJNFJ"> (function(d,e,v,s,i,t,E){d['GoogleDevelopersObject']=i; t=e.createElement(v);t.async=1;t.src=s;E=e.getElementsByTagName(v)[0]; E.parentNode.insertBefore(t,E);})(window, document, 'script', 'https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow/js/app_loader.js', '[15,"en",null,"/js/devsite_app_module.js","https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625","https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow","https://tensorflow-dot-devsite-v2-prod-3p.appspot.com",null,null,["/_pwa/tensorflow/manifest.json","https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/images/video-placeholder.svg","https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow/images/favicon.png","https://www.gstatic.com/devrel-devsite/prod/v870e399c64f7c43c99a3043db4b3a74327bb93d0914e84a0c3dba90bbfd67625/tensorflow/images/lockup.svg","https://fonts.googleapis.com/css?family=Google+Sans:400,500|Roboto:400,400italic,500,500italic,700,700italic|Roboto+Mono:400,500,700&display=swap"],1,null,[1,6,8,12,14,17,21,25,50,52,63,70,75,76,80,87,91,92,93,97,98,100,101,102,103,104,105,107,108,109,110,112,113,117,118,120,122,124,125,126,127,129,130,131,132,133,134,135,136,138,140,141,147,148,149,151,152,156,157,158,159,161,163,164,168,169,170,179,180,182,183,186,191,193,196],"AIzaSyCNm9YxQumEXwGJgTDjxoxXK6m1F-9720Q","AIzaSyCc76DZePGtoyUjqKrLdsMGk_ry7sljLbY","www.tensorflow.org","AIzaSyB9bqgQ2t11WJsOX8qNsCQ6U-w91mmqF-I","AIzaSyAdYnStPdzjcJJtQ0mvIaeaMKj7_t6J_Fg",null,null,null,["Cloud__enable_legacy_calculator_redirect","Concierge__enable_pushui","EngEduTelemetry__enable_engedu_telemetry","Experiments__reqs_query_experiments","Cloud__enable_cloudx_experiment_ids","Profiles__enable_completecodelab_endpoint","Analytics__enable_clearcut_logging","Cloud__enable_cloud_facet_chat","TpcFeatures__enable_required_headers","MiscFeatureFlags__developers_footer_dark_image","Profiles__enable_dashboard_curated_recommendations","MiscFeatureFlags__enable_view_transitions","Cloud__enable_llm_concierge_chat","Cloud__enable_free_trial_server_call","TpcFeatures__enable_mirror_tenant_redirects","CloudShell__cloud_shell_button","Cloud__enable_cloudx_ping","DevPro__enable_developer_subscriptions","CloudShell__cloud_code_overflow_menu","Cloud__enable_cloud_dlp_service","Search__enable_suggestions_from_borg","DevPro__enable_cloud_innovators_plus","MiscFeatureFlags__enable_explain_this_code","Profiles__enable_awarding_url","Cloud__enable_cloud_shell","Search__enable_dynamic_content_confidential_banner","Profiles__enable_public_developer_profiles","Cloud__enable_cloud_shell_fte_user_flow","MiscFeatureFlags__enable_firebase_utm","Profiles__enable_page_saving","MiscFeatureFlags__developers_footer_image","Profiles__enable_complete_playlist_endpoint","Search__enable_page_map","Profiles__enable_recognition_badges","Search__enable_ai_eligibility_checks","Profiles__enable_profile_collections","MiscFeatureFlags__enable_variable_operator","Profiles__require_profile_eligibility_for_signin","Profiles__enable_developer_profiles_callout","BookNav__enable_tenant_cache_key","MiscFeatureFlags__emergency_css","Profiles__enable_release_notes_notifications","MiscFeatureFlags__enable_project_variables","SignIn__enable_refresh_access_tokens"],null,null,"AIzaSyA58TaKli1DculwmAmbpzLVGuWc8eCQgQc","https://developerscontentserving-pa.googleapis.com","AIzaSyDWBU60w0P9hEkr29kkksYs8Z7gvZ8u_wc","https://developerscontentsearch-pa.googleapis.com",2,4,null,"https://developerprofiles-pa.googleapis.com",[15,"tensorflow","TensorFlow","www.tensorflow.org",null,"tensorflow-dot-devsite-v2-prod-3p.appspot.com",null,null,[null,1,null,null,null,null,null,null,null,null,null,[1],null,null,null,null,null,null,[1],[1,null,null,[1]],null,null,null,[1,null,1],[1,1,null,1,1]],null,[25,null,null,null,null,null,"/images/lockup.svg","/images/logo.png",null,null,null,1,1,null,null,null,null,null,null,null,null,1,null,null,null,null,[]],[],null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,[6,1],null,[[],[1,1]],[[["UA-69864048-1"],["UA-69864048-4"],null,null,["UA-69864048-5"],["GTM-MXSL34P"],null,null,[["UA-69864048-1",1]],null,[["UA-69864048-5",1]],[["GTM-MXSL34P",1]],1],[[6,5],[3,2],[12,8],[1,1],[4,3],[5,4]],[[2,2],[1,1]]],null,4]]') </script> <devsite-a11y-announce></devsite-a11y-announce> </body> </html>

CINXE.COM

Neural machine translation with attention | Text | TensorFlow