CINXE.COM
perlunitut - Perl Unicode Tutorial - Perldoc Browser
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>perlunitut - Perl Unicode Tutorial - Perldoc Browser</title> <link rel="search" href="/opensearch.xml" type="application/opensearchdescription+xml" title="Perldoc Browser"> <link rel="canonical" href="https://perldoc.perl.org/perlunitut"> <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css" integrity="sha384-JcKb8q3iqJ61gNV9KGb8thSsNjpSL0n8PARn9HuZOnIxN0hoP+VmmDGMN5t9UJ0Z" crossorigin="anonymous"> <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.5.0/styles/stackoverflow-light.min.css" integrity="sha512-cG1IdFxqipi3gqLmksLtuk13C+hBa57a6zpWxMeoY3Q9O6ooFxq50DayCdm0QrDgZjMUn23z/0PMZlgft7Yp5Q==" crossorigin="anonymous" /> <style> body { background: #f4f4f5; color: #020202; } .navbar-dark { background-image: -webkit-linear-gradient(top, #005f85 0, #002e49 100%); background-image: -o-linear-gradient(top, #005f85 0, #002e49 100%); background-image: linear-gradient(to bottom, #005f85 0, #002e49 100%); filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff005f85', endColorstr='#ff002e49', GradientType=0); background-repeat: repeat-x; } .navbar-dark .navbar-nav .nav-link, .navbar-dark .navbar-nav .nav-link:focus { color: #fff } .navbar-dark .navbar-nav .nav-link:hover { color: #ffef68 } #wrapperlicious { margin: 0 auto; font: 0.9em 'Helvetica Neue', Helvetica, sans-serif; font-weight: normal; line-height: 1.5em; margin: 0; padding: 0; } #wrapperlicious h1 { font-size: 1.5em } #wrapperlicious h2 { font-size: 1.3em } #wrapperlicious h3 { font-size: 1.1em } #wrapperlicious h4 { font-size: 0.9em } #wrapperlicious h1, #wrapperlicious h2, #wrapperlicious h3, #wrapperlicious h4, #wrapperlicious dt { color: #020202; margin-top: 1em; margin-bottom: 1em; position: relative; font-weight: bold; } #wrapperlicious a { color: inherit; text-decoration: underline } #wrapperlicious #toc { text-decoration: none } #wrapperlicious a:hover { color: #2a2a2a } #wrapperlicious a img { border: 0 } #wrapperlicious :not(pre) > code { color: inherit; background-color: rgba(0, 0, 0, 0.04); border-radius: 3px; font: 0.9em Consolas, Menlo, Monaco, monospace; padding: 0.3em; } #wrapperlicious dd { margin: 0; margin-left: 2em; } #wrapperlicious dt { color: #2a2a2a; font-weight: bold; margin-left: 0.9em; } #wrapperlicious p { margin-bottom: 1em; margin-top: 1em; } #wrapperlicious li > p { margin-bottom: 0; margin-top: 0; } #wrapperlicious pre { border: 1px solid #c1c1c1; border-radius: 3px; font: 100% Consolas, Menlo, Monaco, monospace; margin-bottom: 1em; margin-top: 1em; } #wrapperlicious pre > code { display: block; background-color: #f6f6f6; font: 0.9em Consolas, Menlo, Monaco, monospace; line-height: 1.5em; text-align: left; white-space: pre; padding: 1em; } #wrapperlicious dl, #wrapperlicious ol, #wrapperlicious ul { margin-bottom: 1em; margin-top: 1em; } #wrapperlicious ul { list-style-type: square; } #wrapperlicious ul ul { margin-bottom: 0px; margin-top: 0px; } #footer { font-size: 0.8em; padding-top: 0.5em; text-align: center; } #more { display: inline; font-size: 0.8em; } #perldocdiv { background-color: #fff; border: 1px solid #c1c1c1; border-bottom-left-radius: 5px; border-bottom-right-radius: 5px; margin-left: auto; margin-right: auto; padding: 3em; padding-top: 1em; max-width: 960px; } #moduleversion { float: right } #wrapperlicious .leading-notice { font-style: italic; padding-left: 1em; margin-top: 1em; margin-bottom: 1em; } #wrapperlicious .permalink { display: none; left: -0.75em; position: absolute; padding-right: 0.25em; text-decoration: none; } #wrapperlicious h1:hover .permalink, #wrapperlicious h2:hover .permalink, #wrapperlicious h3:hover .permalink, #wrapperlicious h4:hover .permalink, #wrapperlicious dt:hover .permalink { display: block; } </style> <!-- Global site tag (gtag.js) - Google Analytics --> <script async src="https://www.googletagmanager.com/gtag/js?id=G-KVNWBNT5FB"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-KVNWBNT5FB'); gtag('config', 'UA-50555-3'); </script> </head> <body> <nav class="navbar navbar-expand-md navbar-dark bg-dark justify-content-between"> <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarNav" aria-controls="navbarNav" aria-expanded="false" aria-label="Toggle navigation"> <span class="navbar-toggler-icon"></span> </button> <a class="navbar-brand" href="/"><img src="/images/perl_camel_30.png" width="30" height="30" class="d-inline-block align-top" alt="Perl Camel Logo"> Perldoc Browser</a> <div class="collapse navbar-collapse" id="navbarNav"> <ul class="navbar-nav mr-auto"> <li class="nav-item dropdown text-nowrap"> <a class="nav-link dropdown-toggle" href="#" id="dropdownlink-stable" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">5.26.3</a> <div class="dropdown-menu" aria-labelledby="dropdownlink-stable"> <a class="dropdown-item" href="/perlunitut">Latest</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.40.1/perlunitut">5.40.1</a> <a class="dropdown-item" href="/5.40.0/perlunitut">5.40.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.38.3/perlunitut">5.38.3</a> <a class="dropdown-item" href="/5.38.2/perlunitut">5.38.2</a> <a class="dropdown-item" href="/5.38.1/perlunitut">5.38.1</a> <a class="dropdown-item" href="/5.38.0/perlunitut">5.38.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.36.3/perlunitut">5.36.3</a> <a class="dropdown-item" href="/5.36.2/perlunitut">5.36.2</a> <a class="dropdown-item" href="/5.36.1/perlunitut">5.36.1</a> <a class="dropdown-item" href="/5.36.0/perlunitut">5.36.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.34.3/perlunitut">5.34.3</a> <a class="dropdown-item" href="/5.34.2/perlunitut">5.34.2</a> <a class="dropdown-item" href="/5.34.1/perlunitut">5.34.1</a> <a class="dropdown-item" href="/5.34.0/perlunitut">5.34.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.32.1/perlunitut">5.32.1</a> <a class="dropdown-item" href="/5.32.0/perlunitut">5.32.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.30.3/perlunitut">5.30.3</a> <a class="dropdown-item" href="/5.30.2/perlunitut">5.30.2</a> <a class="dropdown-item" href="/5.30.1/perlunitut">5.30.1</a> <a class="dropdown-item" href="/5.30.0/perlunitut">5.30.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.28.3/perlunitut">5.28.3</a> <a class="dropdown-item" href="/5.28.2/perlunitut">5.28.2</a> <a class="dropdown-item" href="/5.28.1/perlunitut">5.28.1</a> <a class="dropdown-item" href="/5.28.0/perlunitut">5.28.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item active" href="/5.26.3/perlunitut">5.26.3</a> <a class="dropdown-item" href="/5.26.2/perlunitut">5.26.2</a> <a class="dropdown-item" href="/5.26.1/perlunitut">5.26.1</a> <a class="dropdown-item" href="/5.26.0/perlunitut">5.26.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.24.4/perlunitut">5.24.4</a> <a class="dropdown-item" href="/5.24.3/perlunitut">5.24.3</a> <a class="dropdown-item" href="/5.24.2/perlunitut">5.24.2</a> <a class="dropdown-item" href="/5.24.1/perlunitut">5.24.1</a> <a class="dropdown-item" href="/5.24.0/perlunitut">5.24.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.22.4/perlunitut">5.22.4</a> <a class="dropdown-item" href="/5.22.3/perlunitut">5.22.3</a> <a class="dropdown-item" href="/5.22.2/perlunitut">5.22.2</a> <a class="dropdown-item" href="/5.22.1/perlunitut">5.22.1</a> <a class="dropdown-item" href="/5.22.0/perlunitut">5.22.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.20.3/perlunitut">5.20.3</a> <a class="dropdown-item" href="/5.20.2/perlunitut">5.20.2</a> <a class="dropdown-item" href="/5.20.1/perlunitut">5.20.1</a> <a class="dropdown-item" href="/5.20.0/perlunitut">5.20.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.18.4/perlunitut">5.18.4</a> <a class="dropdown-item" href="/5.18.3/perlunitut">5.18.3</a> <a class="dropdown-item" href="/5.18.2/perlunitut">5.18.2</a> <a class="dropdown-item" href="/5.18.1/perlunitut">5.18.1</a> <a class="dropdown-item" href="/5.18.0/perlunitut">5.18.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.16.3/perlunitut">5.16.3</a> <a class="dropdown-item" href="/5.16.2/perlunitut">5.16.2</a> <a class="dropdown-item" href="/5.16.1/perlunitut">5.16.1</a> <a class="dropdown-item" href="/5.16.0/perlunitut">5.16.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.14.4/perlunitut">5.14.4</a> <a class="dropdown-item" href="/5.14.3/perlunitut">5.14.3</a> <a class="dropdown-item" href="/5.14.2/perlunitut">5.14.2</a> <a class="dropdown-item" href="/5.14.1/perlunitut">5.14.1</a> <a class="dropdown-item" href="/5.14.0/perlunitut">5.14.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.12.5/perlunitut">5.12.5</a> <a class="dropdown-item" href="/5.12.4/perlunitut">5.12.4</a> <a class="dropdown-item" href="/5.12.3/perlunitut">5.12.3</a> <a class="dropdown-item" href="/5.12.2/perlunitut">5.12.2</a> <a class="dropdown-item" href="/5.12.1/perlunitut">5.12.1</a> <a class="dropdown-item" href="/5.12.0/perlunitut">5.12.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.10.1/perlunitut">5.10.1</a> <a class="dropdown-item" href="/5.10.0/perlunitut">5.10.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.8.9/perlunitut">5.8.9</a> <a class="dropdown-item" href="/5.8.8/perlunitut">5.8.8</a> <a class="dropdown-item" href="/5.8.7/perlunitut">5.8.7</a> <a class="dropdown-item" href="/5.8.6/perlunitut">5.8.6</a> <a class="dropdown-item" href="/5.8.5/perlunitut">5.8.5</a> <a class="dropdown-item" href="/5.8.4/perlunitut">5.8.4</a> <a class="dropdown-item" href="/5.8.3/perlunitut">5.8.3</a> <a class="dropdown-item" href="/5.8.2/perlunitut">5.8.2</a> <a class="dropdown-item" href="/5.8.1/perlunitut">5.8.1</a> <a class="dropdown-item" href="/5.8.0/perlunitut">5.8.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.6.2/perlunitut">5.6.2</a> <a class="dropdown-item" href="/5.6.1/perlunitut">5.6.1</a> <a class="dropdown-item" href="/5.6.0/perlunitut">5.6.0</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.005_04/perlunitut">5.005_04</a> <a class="dropdown-item" href="/5.005_03/perlunitut">5.005_03</a> <a class="dropdown-item" href="/5.005_02/perlunitut">5.005_02</a> <a class="dropdown-item" href="/5.005_01/perlunitut">5.005_01</a> <a class="dropdown-item" href="/5.005/perlunitut">5.005</a> </div> </li> <li class="nav-item dropdown text-nowrap"> <a class="nav-link dropdown-toggle" href="#" id="dropdownlink-dev" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Dev</a> <div class="dropdown-menu" aria-labelledby="dropdownlink-dev"> <a class="dropdown-item" href="/blead/perlunitut">blead</a> <a class="dropdown-item" href="/5.41.10/perlunitut">5.41.10</a> <a class="dropdown-item" href="/5.41.9/perlunitut">5.41.9</a> <a class="dropdown-item" href="/5.41.8/perlunitut">5.41.8</a> <a class="dropdown-item" href="/5.41.7/perlunitut">5.41.7</a> <a class="dropdown-item" href="/5.41.6/perlunitut">5.41.6</a> <a class="dropdown-item" href="/5.41.5/perlunitut">5.41.5</a> <a class="dropdown-item" href="/5.41.4/perlunitut">5.41.4</a> <a class="dropdown-item" href="/5.41.3/perlunitut">5.41.3</a> <a class="dropdown-item" href="/5.41.2/perlunitut">5.41.2</a> <a class="dropdown-item" href="/5.41.1/perlunitut">5.41.1</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.40.1-RC1/perlunitut">5.40.1-RC1</a> <a class="dropdown-item" href="/5.40.0-RC2/perlunitut">5.40.0-RC2</a> <a class="dropdown-item" href="/5.40.0-RC1/perlunitut">5.40.0-RC1</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.39.10/perlunitut">5.39.10</a> <a class="dropdown-item" href="/5.39.9/perlunitut">5.39.9</a> <a class="dropdown-item" href="/5.39.8/perlunitut">5.39.8</a> <a class="dropdown-item" href="/5.39.7/perlunitut">5.39.7</a> <a class="dropdown-item" href="/5.39.6/perlunitut">5.39.6</a> <a class="dropdown-item" href="/5.39.5/perlunitut">5.39.5</a> <a class="dropdown-item" href="/5.39.4/perlunitut">5.39.4</a> <a class="dropdown-item" href="/5.39.3/perlunitut">5.39.3</a> <a class="dropdown-item" href="/5.39.2/perlunitut">5.39.2</a> <a class="dropdown-item" href="/5.39.1/perlunitut">5.39.1</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.38.3-RC1/perlunitut">5.38.3-RC1</a> </div> </li> <li class="nav-item dropdown text-nowrap"> <a class="nav-link dropdown-toggle" href="#" id="dropdownlink-nav" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Documentation</a> <div class="dropdown-menu" aria-labelledby="dropdownlink-nav"> <a class="dropdown-item" href="/5.26.3/perl">Perl</a> <a class="dropdown-item" href="/5.26.3/perlintro">Intro</a> <a class="dropdown-item" href="/5.26.3/perl#Tutorials">Tutorials</a> <a class="dropdown-item" href="/5.26.3/perlfaq">FAQs</a> <a class="dropdown-item" href="/5.26.3/perl#Reference-Manual">Reference</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.26.3/perlop">Operators</a> <a class="dropdown-item" href="/5.26.3/functions">Functions</a> <a class="dropdown-item" href="/5.26.3/variables">Variables</a> <a class="dropdown-item" href="/5.26.3/modules">Modules</a> <a class="dropdown-item" href="/5.26.3/perlutil">Utilities</a> <div class="dropdown-divider"></div> <a class="dropdown-item" href="/5.26.3/perldelta">Release Notes</a> <a class="dropdown-item" href="/5.26.3/perlcommunity">Community</a> <a class="dropdown-item" href="/5.26.3/perlhist">History</a> </div> </li> </ul> <ul class="navbar-nav"> <script> function set_expand (expand) { var perldocdiv = document.getElementById('perldocdiv'); var width = window.getComputedStyle(perldocdiv).getPropertyValue('max-width'); var expanded = (width == '' || width == 'none') ? true : false; if (expand === null) { expand = !expanded; } if ((expand && !expanded) || (!expand && expanded)) { perldocdiv.style.setProperty('max-width', expand ? 'none' : '960px'); var button_classlist = document.getElementById('content-expand-button').classList; if (expand) { button_classlist.add('btn-light'); button_classlist.remove('btn-outline-light'); } else { button_classlist.add('btn-outline-light'); button_classlist.remove('btn-light'); } } return expand; } function toggle_expand () { var expand = set_expand(null); document.cookie = 'perldoc_expand=' + (expand ? 1 : 0) + '; path=/; expires=Tue, 19 Jan 2038 03:14:07 UTC'; } function read_expand () { return document.cookie.split(';').some(function (item) { return item.indexOf('perldoc_expand=1') >= 0 }); } if (document.readyState === 'loading') { document.addEventListener('DOMContentLoaded', function () { if (read_expand()) { set_expand(true); } }); } else if (read_expand()) { set_expand(true); } </script> <button id="content-expand-button" type="button" class="btn btn-outline-light d-none d-lg-inline-block mr-4" onclick="toggle_expand()">Expand</button> </ul> <form class="form-inline" method="get" action="/5.26.3/search"> <input class="form-control mr-3" type="search" name="q" placeholder="Search" aria-label="Search" value=""> </form> </div> </nav> <div id="wrapperlicious" class="container-fluid"> <div id="perldocdiv"> <div id="links"> <a href="/5.26.3/perlunitut">perlunitut</a> <div id="more"> (<a href="/5.26.3/perlunitut.txt">source</a>, <a href="https://metacpan.org/pod/perlunitut">CPAN</a>) </div> </div> <div class="leading-notice"> You are viewing the version of this documentation from Perl 5.26.3. <a href="/perlunitut">View the latest version</a> </div> <h1><a id="toc">CONTENTS</a></h1> <ul> <li> <a class="text-decoration-none" href="#NAME">NAME</a> </li> <li> <a class="text-decoration-none" href="#DESCRIPTION">DESCRIPTION</a> <ul> <li> <a class="text-decoration-none" href="#Definitions">Definitions</a> <ul> <li> <a class="text-decoration-none" href="#Unicode">Unicode</a> </li> <li> <a class="text-decoration-none" href="#UTF-8">UTF-8</a> </li> <li> <a class="text-decoration-none" href="#Text-strings-(character-strings)">Text strings (character strings)</a> </li> <li> <a class="text-decoration-none" href="#Binary-strings-(byte-strings)">Binary strings (byte strings)</a> </li> <li> <a class="text-decoration-none" href="#Encoding">Encoding</a> </li> <li> <a class="text-decoration-none" href="#Decoding">Decoding</a> </li> <li> <a class="text-decoration-none" href="#Internal-format">Internal format</a> </li> </ul> </li> <li> <a class="text-decoration-none" href="#Your-new-toolkit">Your new toolkit</a> </li> <li> <a class="text-decoration-none" href="#I/O-flow-(the-actual-5-minute-tutorial)">I/O flow (the actual 5 minute tutorial)</a> </li> </ul> </li> <li> <a class="text-decoration-none" href="#SUMMARY">SUMMARY</a> </li> <li> <a class="text-decoration-none" href="#Q-and-A-(or-FAQ)">Q and A (or FAQ)</a> </li> <li> <a class="text-decoration-none" href="#ACKNOWLEDGEMENTS">ACKNOWLEDGEMENTS</a> </li> <li> <a class="text-decoration-none" href="#AUTHOR">AUTHOR</a> </li> <li> <a class="text-decoration-none" href="#SEE-ALSO">SEE ALSO</a> </li> </ul> <h1 id="NAME"><a class="permalink" href="#NAME">#</a>NAME</h1> <p>perlunitut - Perl Unicode Tutorial</p> <h1 id="DESCRIPTION"><a class="permalink" href="#DESCRIPTION">#</a>DESCRIPTION</h1> <p>The days of just flinging strings around are over. It's well established that modern programs need to be capable of communicating funny accented letters, and things like euro symbols. This means that programmers need new habits. It's easy to program Unicode capable software, but it does require discipline to do it right.</p> <p>There's a lot to know about character sets, and text encodings. It's probably best to spend a full day learning all this, but the basics can be learned in minutes.</p> <p>These are not the very basics, though. It is assumed that you already know the difference between bytes and characters, and realise (and accept!) that there are many different character sets and encodings, and that your program has to be explicit about them. Recommended reading is "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" by Joel Spolsky, at <a href="http://joelonsoftware.com/articles/Unicode.html">http://joelonsoftware.com/articles/Unicode.html</a>.</p> <p>This tutorial speaks in rather absolute terms, and provides only a limited view of the wealth of character string related features that Perl has to offer. For most projects, this information will probably suffice.</p> <h2 id="Definitions"><a class="permalink" href="#Definitions">#</a>Definitions</h2> <p>It's important to set a few things straight first. This is the most important part of this tutorial. This view may conflict with other information that you may have found on the web, but that's mostly because many sources are wrong.</p> <p>You may have to re-read this entire section a few times...</p> <h3 id="Unicode"><a class="permalink" href="#Unicode">#</a>Unicode</h3> <p><b>Unicode</b> is a character set with room for lots of characters. The ordinal value of a character is called a <b>code point</b>. (But in practice, the distinction between code point and character is blurred, so the terms often are used interchangeably.)</p> <p>There are many, many code points, but computers work with bytes, and a byte has room for only 256 values. Unicode has many more characters than that, so you need a method to make these accessible.</p> <p>Unicode is encoded using several competing encodings, of which UTF-8 is the most used. In a Unicode encoding, multiple subsequent bytes can be used to store a single code point, or simply: character.</p> <h3 id="UTF-8"><a class="permalink" href="#UTF-8">#</a><a id="UTF"></a>UTF-8</h3> <p><b>UTF-8</b> is a Unicode encoding. Many people think that Unicode and UTF-8 are the same thing, but they're not. There are more Unicode encodings, but much of the world has standardized on UTF-8.</p> <p>UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take only one byte per character. All other characters are encoded as two to four bytes using a complex scheme. Fortunately, Perl handles this for us, so we don't have to worry about this.</p> <h3 id="Text-strings-(character-strings)"><a class="permalink" href="#Text-strings-(character-strings)">#</a><a id="Text"></a><a id="Text-strings-character-strings"></a>Text strings (character strings)</h3> <p><b>Text strings</b>, or <b>character strings</b> are made of characters. Bytes are irrelevant here, and so are encodings. Each character is just that: the character.</p> <p>On a text string, you would do things like:</p> <pre><code>$text =~ s/foo/bar/; if ($string =~ /^\d+$/) { ... } $text = ucfirst $text; my $character_count = length $text;</code></pre> <p>The value of a character (<code>ord</code>, <code>chr</code>) is the corresponding Unicode code point.</p> <h3 id="Binary-strings-(byte-strings)"><a class="permalink" href="#Binary-strings-(byte-strings)">#</a><a id="Binary"></a><a id="Binary-strings-byte-strings"></a>Binary strings (byte strings)</h3> <p><b>Binary strings</b>, or <b>byte strings</b> are made of bytes. Here, you don't have characters, just bytes. All communication with the outside world (anything outside of your current Perl process) is done in binary.</p> <p>On a binary string, you would do things like:</p> <pre><code>my (@length_content) = unpack "(V/a)*", $binary; $binary =~ s/\x00\x0F/\xFF\xF0/; # for the brave :) print {$fh} $binary; my $byte_count = length $binary;</code></pre> <h3 id="Encoding"><a class="permalink" href="#Encoding">#</a>Encoding</h3> <p><b>Encoding</b> (as a verb) is the conversion from <i>text</i> to <i>binary</i>. To encode, you have to supply the target encoding, for example <code>iso-8859-1</code> or <code>UTF-8</code>. Some encodings, like the <code>iso-8859</code> ("latin") range, do not support the full Unicode standard; characters that can't be represented are lost in the conversion.</p> <h3 id="Decoding"><a class="permalink" href="#Decoding">#</a>Decoding</h3> <p><b>Decoding</b> is the conversion from <i>binary</i> to <i>text</i>. To decode, you have to know what encoding was used during the encoding phase. And most of all, it must be something decodable. It doesn't make much sense to decode a PNG image into a text string.</p> <h3 id="Internal-format"><a class="permalink" href="#Internal-format">#</a><a id="Internal"></a>Internal format</h3> <p>Perl has an <b>internal format</b>, an encoding that it uses to encode text strings so it can store them in memory. All text strings are in this internal format. In fact, text strings are never in any other format!</p> <p>You shouldn't worry about what this format is, because conversion is automatically done when you decode or encode.</p> <h2 id="Your-new-toolkit"><a class="permalink" href="#Your-new-toolkit">#</a><a id="Your"></a>Your new toolkit</h2> <p>Add to your standard heading the following line:</p> <pre><code>use Encode qw(encode decode);</code></pre> <p>Or, if you're lazy, just:</p> <pre><code>use Encode;</code></pre> <h2 id="I/O-flow-(the-actual-5-minute-tutorial)"><a class="permalink" href="#I/O-flow-(the-actual-5-minute-tutorial)">#</a><a id="I"></a><a id="I-O-flow-the-actual-5-minute-tutorial"></a>I/O flow (the actual 5 minute tutorial)</h2> <p>The typical input/output flow of a program is:</p> <pre><code class="plaintext">1. Receive and decode 2. Process 3. Encode and output</code></pre> <p>If your input is binary, and is supposed to remain binary, you shouldn't decode it to a text string, of course. But in all other cases, you should decode it.</p> <p>Decoding can't happen reliably if you don't know how the data was encoded. If you get to choose, it's a good idea to standardize on UTF-8.</p> <pre><code>my $foo = decode('UTF-8', get 'http://example.com/'); my $bar = decode('ISO-8859-1', readline STDIN); my $xyzzy = decode('Windows-1251', $cgi->param('foo'));</code></pre> <p>Processing happens as you knew before. The only difference is that you're now using characters instead of bytes. That's very useful if you use things like <code>substr</code>, or <code>length</code>.</p> <p>It's important to realize that there are no bytes in a text string. Of course, Perl has its internal encoding to store the string in memory, but ignore that. If you have to do anything with the number of bytes, it's probably best to move that part to step 3, just after you've encoded the string. Then you know exactly how many bytes it will be in the destination string.</p> <p>The syntax for encoding text strings to binary strings is as simple as decoding:</p> <pre><code>$body = encode('UTF-8', $body);</code></pre> <p>If you needed to know the length of the string in bytes, now's the perfect time for that. Because <code>$body</code> is now a byte string, <code>length</code> will report the number of bytes, instead of the number of characters. The number of characters is no longer known, because characters only exist in text strings.</p> <pre><code>my $byte_count = length $body;</code></pre> <p>And if the protocol you're using supports a way of letting the recipient know which character encoding you used, please help the receiving end by using that feature! For example, E-mail and HTTP support MIME headers, so you can use the <code>Content-Type</code> header. They can also have <code>Content-Length</code> to indicate the number of <i>bytes</i>, which is always a good idea to supply if the number is known.</p> <pre><code>"Content-Type: text/plain; charset=UTF-8", "Content-Length: $byte_count"</code></pre> <h1 id="SUMMARY"><a class="permalink" href="#SUMMARY">#</a>SUMMARY</h1> <p>Decode everything you receive, encode everything you send out. (If it's text data.)</p> <h1 id="Q-and-A-(or-FAQ)"><a class="permalink" href="#Q-and-A-(or-FAQ)">#</a><a id="Q"></a><a id="Q-and-A-or-FAQ"></a>Q and A (or FAQ)</h1> <p>After reading this document, you ought to read <a href="/5.26.3/perlunifaq">perlunifaq</a> too, then <a href="/5.26.3/perluniintro">perluniintro</a>.</p> <h1 id="ACKNOWLEDGEMENTS"><a class="permalink" href="#ACKNOWLEDGEMENTS">#</a>ACKNOWLEDGEMENTS</h1> <p>Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the Amsterdam Perl Mongers meetings got me interested and determined to find out how to use character encodings in Perl in ways that don't break easily.</p> <p>Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild" (Dutch Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.</p> <p>Thanks to the people who asked about this kind of stuff in several Perl IRC channels, and have constantly reminded me that a simpler explanation was needed.</p> <p>Thanks to the people who reviewed this document for me, before it went public. They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan Gray.</p> <h1 id="AUTHOR"><a class="permalink" href="#AUTHOR">#</a>AUTHOR</h1> <p>Juerd Waalboer <#####@juerd.nl></p> <h1 id="SEE-ALSO"><a class="permalink" href="#SEE-ALSO">#</a><a id="SEE"></a>SEE ALSO</h1> <p><a href="/5.26.3/perlunifaq">perlunifaq</a>, <a href="/5.26.3/perlunicode">perlunicode</a>, <a href="/5.26.3/perluniintro">perluniintro</a>, <a href="/5.26.3/Encode">Encode</a></p> </div> <div id="footer"> <p>Perldoc Browser is maintained by Dan Book (<a href="https://metacpan.org/author/DBOOK">DBOOK</a>). Please contact him via the <a href="https://github.com/Grinnz/perldoc-browser/issues">GitHub issue tracker</a> or <a href="mailto:dbook@cpan.org">email</a> regarding any issues with the site itself, search, or rendering of documentation.</p> <p>The Perl documentation is maintained by the Perl 5 Porters in the development of Perl. Please contact them via the <a href="https://github.com/Perl/perl5/issues">Perl issue tracker</a>, the <a href="https://lists.perl.org/list/perl5-porters.html">mailing list</a>, or <a href="https://kiwiirc.com/client/irc.perl.org/p5p">IRC</a> to report any issues with the contents or format of the documentation.</p> </div> </div> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.slim.min.js" integrity="sha512-/DXTXr6nQodMUiq+IUJYCt2PPOUjrHJ9wFrqpJ3XkgPNOZVfMok7cRw6CSxyCQxXn6ozlESsSh1/sMCTF1rL/g==" crossorigin="anonymous"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.16.1/umd/popper.min.js" integrity="sha512-ubuT8Z88WxezgSqf3RLuNi5lmjstiJcyezx34yIU2gAHonIi27Na7atqzUZCOoY4CExaoFumzOsFQ2Ch+I/HCw==" crossorigin="anonymous"></script> <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/js/bootstrap.min.js" integrity="sha384-B4gt1jrGC7Jh4AgTPSdUtOBvfO8shuf57BaghqFfPlYxofvL8/KUEfYiJOMMV+rV" crossorigin="anonymous"></script> <script src="/js/highlight.pack.js"></script> <script>hljs.highlightAll();</script> </body> </html>