CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8" /> <title>Understanding the New Language Tags</title> <meta name="description" content="As the new language tags and software based on RFC 3066bis begins to appear, it's important to review how the new standard for language tags changes them, and what remains to be done." /> <script type="text/javascript"> var f = { } // AUTHORS should fill in these assignments: f.directory = 'articles/bcp47'+'/'; // the path to this file, not including /International or the file name f.filename = 'index'; // the file name WITHOUT extensions f.authors = 'Addison P. Phillips, Yahoo! Inc'; // author(s) and affiliations f.previousauthors = ''; // as above f.modifiers = ''; // people making substantive changes, and their affiliation f.searchString = 'article-bcp47'; // blog search string - usually the filename without extensions f.firstPubDate = '2006-05-15'; // date of the first publication of the document (after review) f.lastSubstUpdate = { date:'2006-05-15', time:'08:04'} // date and time of latest substantive changes to this document f.status = 'published'; // should be one of draft, review, published, notreviewed or obsolete f.path = '../../' // what you need to prepend to a URL to get to the /International directory // AUTHORS AND TRANSLATORS should fill in these assignments: f.thisVersion = { date:'2016-01-30', time:'06:30'} // date and time of latest edits to this document/translation f.contributors = ''; // people providing useful contributions or feedback during review or at other times // also make sure that the lang attribute on the html tag is correct! // TRANSLATORS should fill in these assignments: f.translators = 'xxxNAME, ORG'; // translator(s) and their affiliation - a elements allowed, but use double quotes for attributes f.breadcrumb = 'language'; f.additionalLinks = '' </script> <script type="text/javascript" src="index-data/translations.js"> </script> <script type="text/javascript" src="../../javascript/doc-structure/article-dt.js"> </script> <script type="text/javascript" src="../../javascript/boilerplate-text/boilerplate-en.js"></script> <script type="text/javascript" src="../../javascript/doc-structure/article.js"> </script> <script type="text/javascript" src="../../javascript/articletoc-html5.js"></script> <link rel="stylesheet" href="../../style/article-2016.css" type="text/css" /> <link rel="copyright" href="#copyright"/>  <style type="text/css" media="all"> th.bytes { font-size: 20px; } </style> </head> <body> <header> <nav id="mainNavigation"></nav><script>document.getElementById('mainNavigation').innerHTML = mainNavigation</script> <h1>Understanding the New Language Tags</h1> </header> <section> <div id="audience"> <p><span id="intendedAudience" class="leadin">Intended audience:</span> users, XHTML/HTML coders (using editors or scripting), script developers (PHP, JSP, etc.), CSS coders, schema developers (DTDs, XML Schema, RelaxNG, etc.), XSLT developers, Web project managers, and anyone who is likely to use language tags. </p> <div id="updateInfo"></div><script>document.getElementById('updateInfo').innerHTML = g.updated</script> </div> </section> <p class="unlinked"><strong>Warning:</strong> This article is only of historical interest, since the proposed new approach it refers to was published as RFC 4646 and RFC 4647 (collectively known as BCP 47) in September 2006, and have since been revised. The article is now out of date.</p> <p>For up-to-date information about how to construct language tags see:</p> <ul> <li><a class="print" href="/International/articles/language-tags/">Language tags in HTML and XML</a></li> <br/> <hr/> </ul> <section> <p>As the new language tags and software based on RFC 3066bis begin to appear, Addison Phillips, co-editor of RFC3066bis with Mark Davis, reviews how the new standard changes language tags, and what remains to be done.</p> </section> <section> <h2 id="intro"><a href="#intro">Introduction</a></h2> <p>Language tags are identifiers used in protocols or document formats to indicate the natural language of the content, or to express a user's preference for a specific language or set of languages. Language tags can be used by a computer system to apply specific processing or formatting to the text in a language sensitive manner. For example, a language tag might be used to assist in default font selection or to select which dictionary to use in the spell checker.</p> <p>Language tags can also be used to identify the audience for a document or in a language negotiation mechanism to select which content should be displayed. Web-based applications often use language tags to infer the user's locale preference (affecting number or date formats in a Web site, for example).</p> <p>The standard for language tags on the Internet, which includes Web technologies, email, protocol headers, HTML, XML, IMAP, LDAP, RDF, RSS, and a potpourri of other acronyms, is a something called "BCP 47".</p> <div class="sidenoteGroup"> <p>In November of 2005 the <acronym title="Internet Engineering Task Force">IETF</acronym>, which is the standards body that defines these tags, approved a set of documents to update BCP 47, making changes in the structure of language tags and their use. Never heard of BCP 47? You might have heard of RFC 1766 (which was the original BCP 47) or its successor, RFC 3066, which was BCP 47 until this past November. These were the documents that defined the use of ISO 639 and ISO 3166 codes to form language tags like "fr", "en-US", "de-CH", or "ja", as well as a registry for other, more specialized, tags.</p> <div class="sideinfonote"><p class="info">BCP stands for Best Current Practice. RFC stands for Requests for Comments. RFCs are stable and official IETF specifications, having moved past the Internet Draft stage.</p></div> </div> </section> <section> <h2 id="overview"><a href="#overview">Overview of the new approach</a></h2> <p>RFC 3066bis places additional restrictions on the format of a language tag, that is, language tags cannot vary as widely under RFC 3066bis as they could under either predecessor. Since only a few tags were actually registered, this doesn't impose a new burden on users or software. Any tag that was valid under RFC 3066 is still a valid RFC 3066bis tag. It is usually the right tag too, that is, users usually will not want to choose a different tag today in place of the RFC 3066 tag of yesterday.</p> <p>Any implementation that could handle the registered tags of RFC 3066 should be able to handle the tags generated by RFC 3066bis, since all of the new tags were valid to register under RFC 3066. Indeed, more than half the registry consisted of tags that anticipated the addition of script subtags using the RFC 3066bis structure by the time the new rules were adopted.</p> <p>Language tags still consist of a sequence of "subtags" separated by hyphens. A subtag can be between one and eight characters in length and is restricted to the ASCII letters and numbers (that is, a-z, A-Z, and 0-9). Upper and lowercase letters are not distinguished, so the tag "EN" is considered to be the same as the tag "en" or "eN". </p> <p>RFC 3066bis defines each type of subtag according to its position and size in the tag. The complete syntax for these tags is shown here:</p> <div class="example-wide" id="fig_1" title="Figure 1"> <pre>A Language-Tag consists of: langtag ; generated tag -or- private-use ; a private use tag -or- grandfathered ; grandfathered registrations langtag = (language ["-" script] ["-" region] *("-" variant) *("-" extension) ["-" privateuse]) language = "en", "ale", or a registered value script = "Latn", "Cyrl", "Hant" ISO 15924 codes region = "US", "CS", "FR" ISO 3166 codes "419", "019", or UN M.49 codes variant = "rozaj", "nedis", "1996", multiple subtags can be used in a tag extension = single letter followed by additional subtags; more than one extension may be used in a language tag private-use = "x-" followed by additional subtags, as many as are required Note that these can start a tag or appear at the end (but not in the middle) grandfathered = tags listed in the old registry that are not otherwise redundant (a closed list) </pre> </div> <p>As noted above, any valid RFC 3066 language tag is also valid under the new scheme. Most tags are now composed of a sequence of subtags using the generative syntax. </p> <p>The few exceptions are "grandfathered" tags: these are tags registered under RFC 3066 that don't fit the syntax above (or were obsolete before its adoption). Any existing content or software can thus continue to use these tags. There are 34 of these, of which eight are obsolete and ten more will be made obsolete in the near future. Four grandfathered tags fit the RFC 3066bis pattern, but were not made redundant initially. </p>  <p>The big difference with RFC 3066bis is that, excepting a few grandfathered registrations, all tags are now generative. Because ISO code lists were not always free and because they change over time, a key idea was to create a permanent, stable registry for <em>all</em> of the subtags valid in a language tag. This means that instead of five separate lists of codes, there is also just one table containing the values located in a single place. The IANA Language Subtag Registry still tracks the ISO standards, except that subtags are never withdrawn and there are clear rules for dealing with conflicting assignments if or when these arise.</p> <p>Each type of subtag has unique length and content restrictions. The tag always begins with a language subtag - either one of the ISO 639 codes or a registered value. It can then, optionally, be followed by various subtags. Today there are five kinds of subtags that follow the language identifier: scripts, regions, variants, extensions, and private use. The order, length, and content of each subtag type is fixed, so a tag processor can always identify exactly which type of subtag it has, even if the processor doesn't have that subtag in its copy of the registry (or has no copy of the registry at all!)</p> <p><strong class="leadin">Script.</strong> Script subtags we've already met. These are based on ISO 15924 and indicate the writing system. The script subtag can occur at most once (it may be omitted) and must appear directly after the language. Some languages have a field in the registry indicating that a particular script code should be "suppressed". For example, "zh-Hant" and "zh-Hans" represent Chinese written in Traditional and Simplified scripts respectively, while the language subtag "en" has a "Suppress-Script" field in the registry indicating that most English texts are written in the Latin script, discouraging a tag such as "en-Latn-US".</p> <p><strong class="leadin">Region.</strong> Region subtags we've also met: these are based mostly on ISO 3166-1 code and indicate the country or regional variation. The region code can also include selected UN M.49 region codes. UN M.49 codes cover larger areas of the earth or provide for conflict resolution should ISO 3166 reassign a code already in the registry. In fact, ISO 3166 depends on UN M.49 to define what is or is not a "country" or region worthy of a code. The region code can occur at most once (it may be omitted) and must follow any language and script codes. For example, the tag "es-419" represents "Spanish as used in Latin America and the Caribbean" while "es-CO" represents "Spanish as used in Columbia".</p> <p><strong class="leadin">Variants.</strong> Variant subtags are not based on an external standard. They are all individually registered values, mostly indicating particular dialects or other language variations not covered by scripts or regions. Multiple variant subtags can be included in a tag. Each variant has fields in the registry, though, indicating which subtags it is intended for use with. For example, the "nedis" subtag has a prefix of "sl" (Slovenian) since it represents a dialect of Slovenian. Variants shouldn't be used together unless one variant lists the other in its "prefix". For example, the tag "sl-IT-nedis" identifies the Nadiza dialect of Slovenian as used in Italy.</p> <p><strong class="leadin">Extensions.</strong> Extensions are a mechanism whereby future additions to language tags can be standardized. Each extension has a single character subtag (a "singleton") that identifies it. Various restrictions apply to extensions and how they are formed, used, and administered. Extensions form the basis for future addition of features to language tags.</p> <p><strong class="leadin">Private use.</strong> Private use subtags are not based on any standard at all. They are for use by individuals or groups that need to identify something language related that might not rise to the level of standardization. RFC 3066 included private use tags, but the whole tag was private use (this is still valid, of course). Now private use and generative subtags can be used together. The single-letter subtag "x" identifies where the private use subtags begin. For example: en-US-x-twain might identify writing by Mark Twain between two colleagues studying American literature. One benefit of this ability to mix the two is that vendors who extend language tags for proprietary reasons in the future can do so while preserving the maximum amount of interoperability between their system and others.</p> <p>The new tag syntax uses length and content to distinguish each type of subtag, making it easier than ever to validate the contents of a tag, even without a copy of the registry. The following table shows a number of examples of the new tags:</p> <table width="65%" border="1" cellpadding="1" cellspacing="0"> <tr> <td>Tag</td> <td>Form</td> <td>Meaning</td> </tr> <tr> <td>en</td> <td>language</td> <td>English</td> </tr> <tr> <td>de-AT</td> <td>language-region</td> <td>German as used in Austria</td> </tr> <tr> <td>es-419</td> <td>language-region (UN M.49)</td> <td>Spanish as used in Central and South America</td> </tr> <tr> <td>de-CH-1901</td> <td>language-region-variant</td> <td>German as used in Switzerland, orthography of 1901</td> </tr> <tr> <td>sr-Cyrl</td> <td>language-script</td> <td>Serbian as written in Cyrillic</td> </tr> <tr> <td>sr-Cyrl-CS</td> <td>language-script-region</td> <td>Serbian as written in Cyrillic as used in Serbia and Montenegro</td> </tr> <tr> <td>sl-Latn-IT-rozaj</td> <td>language-script-region-variant</td> <td>Slovenian as written in Latin as used in Italy, Resian dialect</td> </tr> </table> </section> <section> <h2 id="script"><a href="#script">The great script debate</a></h2> <p>A critical point of debate during the development of the new language tags was the positioning of the script subtag after the language subtag but before the region subtag. Nothing in RFC 1766 or RFC 3066 guaranteed that the region subtag would appear in the second position and, prior to 2003, when this effort started, no registered tags existed that would clarify whether it was valid to assume that the region code, if it existed, would always appear second (it was quite clear that other values, such as script, could appear second).</p> <p>Some people felt that putting scripts into the second position presented some problems. In particular, some feared that the script subtag would interfere with common language selection or language negotiation mechanisms. These mechanisms, such as the one described in RFC 2616 (HTTP 1.1) use a prefix called a "language range" which is specified by the user in order to select content.</p> <p>This form of matching assumes that the user's preference "matches" a piece of content if the user's language tag is a prefix for that of the content. This selection mechanism relies on the assumption that languages which share a prefix are usually "mutually intelligible". (Note that this assumption is often wrong.) Here are some examples of prefix matching:</p> <table width="65%" border="1" cellpadding="1" cellspacing="0"> <thead> <tr> <td>Language Range</td> <td>... matches</td> <td>... does not match</td> </tr> </thead> <tbody> <tr> <td>de</td> <td>de, de-CH, de-AT, de-DE, de-1901, de-AT-1901</td> <td>en, fr-CH</td> </tr> <tr> <td>de-CH</td> <td>de-CH, de-CH-1901, de-CH-1996</td> <td>de, de-DE, de-1901, de-AT, etc.</td> </tr> <tr> <td>zh-TW</td> <td>zh-TW</td> <td>zh-Hant-TW, zh-Hans-TW</td> </tr> <tr> <td>zh-Hant</td> <td>zh-Hant, zh-Hant-TW, zh-Hant-HK</td> <td>zh, zh-Hans, zh-TW</td> </tr> </tbody> </table> <p>Inserting the script subtag between language and region might have a negative effect on existing user requests or on content that doesn't use a script subtag. Instead of the expected match, the user might receive no content or a less accurate match. This is shown by the last example above. </p> <p>On the other hand, script is usually more closely associated with language than regional variations are. Prefix matching produces more sensible results when the script subtag is closer to the language subtag than the region is. In order to work, users who require script subtags must use (or omit) them in a consistent fashion, in both their requests and their content.</p> <p>Another problem was the ambiguity of RFC 3066 regarding the generative syntax. The idea of "language-dash-region" language tags was easy enough to grasp; most users didn't read RFC 3066 directly or consider the unstated-but-realized implication that other subtags might sometimes occur in the second position.</p> <p>Ultimately it was decided that the closer relationship between script and language made the second position a better choice than artificially placing it last. This decision was partially guided by recognition that another type of subtag might be necessary in the future (which we'll get to later).</p> <p>In addition to the subtags themselves, the new subtag registry contains information to help users select the best combination to identify a particular language. Critical to the acceptance of the position of the script subtag was the inclusion of information in the registry to make clear the need to avoid script subtags except where they add useful distinguishing information. Thus, the registry entry for the language subtag "en" (English) has a field called "Suppress-Script" indicating that the script subtag "Latn" should be avoided with that language, since virtually all English documents use the Latin script.</p> <p>Note that this doesn't mean that "en-Latn" tags will never be used. There are cases where the script will provide information that distinguishes content. For example, a document that contains both Latin script and Braille might need to distinguish the two forms. However, these are unusual cases and the exception will be sensible (and even obvious) in those cases.</p> <p>In any case, for virtually any content that does not use a script subtag today, it remains the best practice not to use one in the future. Languages that do use more than one script or are undergoing a script transition - such as those listed above - can and should benefit from identifying content using script subtags. Just over a year from its registration, a quick look at a search engine shows over 8000 pages in Simplified Chinese mentioning the tag "zh-Hans" alone. The generative syntax will greatly assist the use and acceptance of script subtags for languages that need them.</p> </section> <section> <h2 id="registry"><a href="#registry">The IANA Language Subtag Registry</a></h2> <p>The new IANA Language Subtag Registry contains the information about each subtag which is valid for use in a language tag. The registry is a text file in a special, machine-readable, format called "record-jar". Each subtag has its own record, consisting of several lines of text, which identifies the subtags, their use, and some information useful in selecting which subtags are right for specific circumstances.</p> <p>Here are some examples of some "language" subtag records:</p> <div class="example"> <pre> %% Type: language Subtag: cs Description: Czech Added: 2005-10-16 Suppress-Script: Latn %% Type: language Subtag: cu Description: Church Slavic Description: Old Slavonic Description: Church Slavonic Description: Old Bulgarian Description: Old Church Slavonic Added: 2005-10-16 %% Type: language Subtag: cv Description: Chuvash Added: 2005-10-16 %%</pre> </div> <p>Each record contains the subtag itself, its type ("language", in this case), a description (or set of descriptions), and the date that the record was added to the registry. All of the initial records have the date "2005-10-16" as shown above. </p> <p>Additional information is sometimes available. For example, in the record for the Czech language (cs) above, you'll notice a field called "Suppress-Script". This field indicates that most texts in Czech are written in the Latin script and that the "Latn" script code is inappropriate for most language tags identifying content in Czech. That is, a tag like "cs-CZ" is recommended, while a tag such as "cs-Latn-CZ" is strongly discouraged.</p> <p>Other fields that can appear include a "Deprecated" field that shows a date on which a particular code was deprecated. This almost always appears with another field called "Preferred-Value", which indicates a more appropriate subtag to use for that value. For example, the code "TP" was deprecated by ISO 3166 when that country changed its administration and name in 2002:</p> <div class="example"> <pre> %% Type: region Subtag: TP Description: East Timor Added: 2005-10-16 Preferred-Value: TL Deprecated: 2002-11-15 %%</pre> </div> <p>The registration process can still be used to add information to or update information about specific records, as well as adding entire new subtags. Records cannot be removed and there are rules to prevent the meaning of a subtag from being "mutated" to mean something completely different.</p> <p>The file itself contains a "File-Date" record, showing the last time the registry was updated. Combined with the various date fields in the records themselves, it is possible to validate any particular tag or its subtags for any given date, past or present.</p> </section> <section> <h2 id="remaining"><a href="#remaining">Current status & remaining work</a></h2> <section> <h3 id="current"><a href="#current">Current status</a></h3> <p>RFC 3066bis actually consists of three parts. First, there is the document that describes the syntax of language tags and the registry, as well as how language tags are maintained and so forth. This document is an Internet-Draft called "draft-ietf-ltru-registry-14" and is about 62 pages long. Then there is the initial contents of the IANA Language Subtag Registry, which, confusingly, is contained in an Internet-Draft called "draft-ietf-ltru-initial-05". This document was edited and maintained by Doug Ewell.</p> <p> The <a class="print" href="http://www.iana.org/assignments/language-subtag-registry">IANA Language Subtag Registry</a> is now up and running, and has even been receiving registrations.</p> <p>The last piece of the puzzle is an Internet-Draft on matching of language tags. This document was being worked on at the time this was written and its current name is "draft-ietf-ltru-matching-12".</p> <p>The IETF website hosts all of these documents, or you can find the latest versions of them all listed on my <a class="print" href="http://www.inter-locale.com">personal website</a> and on the <a class="print" href="/International/core/langtags/rfc3066bis.html">W3C site</a>. </p> </section> <section> <h3 id="matching"><a href="#matching">Matching</a></h3> <p>Matching, as noted earlier, is fairly well understood in its simplest, "prefix matching" form, which is described above in the section on scripts. However, there are some intriguing applications for RFC 3066bis style tags in matching, as well as some well-known matching schemes that were not well documented in RFC 3066. This work is, at the time of writing, awaiting completion of the Last Call process.</p> </section> <section> <h3 id="macro"><a href="#macro">ISO 639-3 and Macro Languages</a></h3> <p>Despite the changes in how language tags are formed and maintained, a few cases remain which the new design does not fully address.</p> <p>A notable problem is that of identifying variations of a language or within family of languages. While variant or region subtags are often useful for this purpose, some languages exhibit long-lived, stable, well-described variations that are not particularly well-described by national boundaries. In addition, ISO 639 has occasionally assigned codes to "macro-languages", which are language families that contain a number of recognizably related (but not necessarily mutually intelligible) languages.</p> <p>An excellent example is once again Chinese. The ISO 639-1 code 'zh' identifies "Chinese", but the concept of Chinese encloses a number of distinct languages or dialects that share certain traits. While these languages are written very similarly (making tags such as "zh-Hant" and "zh-Hans" useful), spoken content is very different indeed. And, again, the available regional options are poor proxies for the spoken dialects (many of which are confined to mainland China).</p> <p>RFC 3066bis provides part of the solution to this conundrum by reserving space for yet another kind of specialized subtag, called an "extended language subtag". These are three-letter codes that follow the primary language subtag but occur before the script subtag. There are very clear rules for when one of these subtags can be used (they must be used only with the specified prefix), and it is anticipated that a very small revision to RFC 3066bis will take place in mid-2006 to make these available. </p> <p>"If the requirements for these codes exists now and we know what they are, why weren't the codes just incorporated into RFC 3066bis?" one might ask. The reason for the delay is that the basis for defining the extended language subtags is expected to be ISO 639-3. Like ISO 639-2 is a superset of ISO 639-1, ISO 639-3 defines an even larger set of language codes, based originally on the codes in the SIL Ethnologue (SIL, in fact, is the "Registration Authority" for ISO 639-3, that is, the folks who will maintain the code list in the future). ISO 639-3 also defines which languages are enclosed by which Macro Languages. Thus Mandarin Chinese (a spoken variation) will be identified by the ISO 639-3 code 'cmn' and rules will require that code, when used as a subtag, to always appear with its macrolanguage "zh" (Chinese). This will finally make is possible to tag Chinese content accurately in all dimensions:</p> <table width="65%" border="1" cellpadding="1" cellspacing="0"> <tr> <td>Tag</td> <td>Meaning</td> </tr> <tr> <td>zh-cmn</td> <td>Mandarin Chinese</td> </tr> <tr> <td>zh-cmn-Hant</td> <td>Mandarin Chinese as written in Traditional script</td> </tr> <tr> <td>zh-cmn-Hans</td> <td>Mandarin Chinese, Simplified script</td> </tr> <tr> <td>zh-cmn-Hant-HK</td> <td>Mandarin Chinese, Traditional script, as used in Hong Kong SAR</td> </tr> <tr> <td>zh-cmn-Hans-CN</td> <td>Mandarin Chinese, Simplified script, as used in China</td> </tr> <tr> <td>zh-gan</td> <td>Gan Chinese</td> </tr> <tr> <td>zh-hak</td> <td>Hakka Chinese</td> </tr> <tr> <td>zh-yue</td> <td>Yue Chinese (Cantonese)</td> </tr> <tr> <td>zh-hsn</td> <td>Xiang Chinese</td> </tr> <tr> <td>zh-yue-Hant-HK</td> <td>Cantonese, Traditional script, Hong Kong SAR</td> </tr> </table> <p>There are about forty different languages other than Chinese that are defined as Macro Languages in the prototype for ISO 639-3. Most of these are minority languages and it is possible that the ability to accurately identify these language variations in content may have an impact on their preservation amongst the living languages.</p> <p>In any case, extended language subtags are already fully specified and are merely waiting for ISO 639-3 to finally be official and complete before being included in the list of language subtags. Note that implementers merely need to update their copy of the registry when ISO 639-3 is added, as long as they have followed the implementation requirements already in RFC 3066bis.</p> </section> </section> <section> <h2 id="conclusion"><a href="#conclusion">Conclusion</a></h2> <p>The new version of BCP 47 provides the ability to accurately tag or request content using stable, well-defined tags. These tags address a number of long standing problems with language identification, leading, hopefully, to richer language-aware features in our software and better support for language in our documents. Understanding these tags and their format will help users adopt them and use them wisely and consistently.</p> </section> <section> <h2 id="endlinks"><a href="#endlinks">Further reading</a></h2> <aside class="section" id="survey"> </aside><script>document.getElementById('survey').innerHTML = g.survey</script> <ul id="full-links"> <li> <p><a href="/International/core/langtags/rfc3066bis">Internet-Draft: BCP 47 (Latest versions of the specifications)</a></p> </li> <li> <p><a href="/International/articles/language-tags/">Language tags in HTML and XML (Overview of RFC 3066)</a></p> </li> </ul> </section> <footer id="thefooter"></footer><script type="text/javascript">document.getElementById('thefooter').innerHTML = g.bottomOfPage</script> <script type="text/javascript">completePage()</script> </body> </html>