Note (b) for Data Compression: A New Kind of Science | Online by Stephen Wolfram [Page 1069]

<!doctype html> <html lang="en" class="notes text-version"> <head>  <meta http-equiv="x-ua-compatible" content="ie=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta charset="utf-8"> <title>Note (b) for Data Compression: A New Kind of Science | Online by Stephen Wolfram [Page 1069]</title> <meta name="description" content="History [of data compression] Morse code, invented in 1838 for use in telegraphy, is an early example of data compression... – from A New Kind of Science"> <link rel="alternate" hreflang="en" href="https://www.wolframscience.com/nks/index.en.php"> <meta property="og:image" content="http://www.wolframscience.com/share.png"> <meta property="og:url" content="https://www.wolframscience.com/nks/notes-10-5--history-of-data-compression/"> <meta property="og:type" content="website"> <meta property="og:title" content="Note (b) for Data Compression: A New Kind of Science | Online by Stephen Wolfram [Page 1069]"> <meta property="og:description" content="History [of data compression] Morse code, invented in 1838 for use in telegraphy, is an early example of data compression... – from A New Kind of Science"> <meta property="og:locale" content="en"> <meta name="twitter:card" content="summary_large_image"> <meta name="twitter:site" content="@WolframResearch"> <meta name="twitter:title" content="Note (b) for Data Compression: A New Kind of Science | Online by Stephen Wolfram [Page 1069]"> <meta name="twitter:description" content="History [of data compression] Morse code, invented in 1838 for use in telegraphy, is an early example of data compression... – from A New Kind of Science"> <meta name="twitter:image:src" content="http://www.wolframscience.com/share.png"> <link rel="canonical" href="https://www.wolframscience.com/nks/notes-10-5--history-of-data-compression/"> <link rel="stylesheet" href="/common/framework/css/framework.en.css"> <script src="//www.wolframcdn.com/consent/cookie-consent.js"></script> <script src="/common/javascript/wal/latest/walLoad.js"></script> <script defer src="/common/js/announcements/script.js"></script> <script src="/common/framework/js/head.en.js"></script> <script src="/common/js/jquery/3.7.1/jquery.min.js"></script> <script src="/common/js/jquery/plugins/migrate/3.5.2/jquery.migrate.min.js"></script> <script src="/common/javascript/analytics.js"></script>  <link rel="stylesheet" href="//www.wolframcdn.com/fonts/source-sans-pro/1.0/global.css"> <link rel="stylesheet" href="//www.wolframcdn.com/fonts/pt-serif/1.0/all.css"> <link rel="stylesheet" href="//www.wolframcdn.com/fonts/mathematica/1.0/all.css">  <link rel="preconnect" href="https://fonts.googleapis.com"> <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> <link href="https://fonts.googleapis.com/css2?family=Crimson+Text:ital@0;1&display=swap" rel="stylesheet"> <link rel="stylesheet" href="https://use.typekit.net/hod7dyh.css">  <link rel="stylesheet" href="/common/js/clipboard/2.0/clipboard.css"> <link rel="stylesheet" href="/nks/css/c2c.en.css"> <link rel="stylesheet" href="/nks/css/shared.en.css"> <link rel="stylesheet" href="/nks/css/notes.en.css"> </head> <body>    <header id="header"> <div class="table width-full height-full"> <div class="heirs-vertical-align-m"> <div class="padding-lr-1"> <h1><a href="/nks/"><img src="/nks/img/a-new-kind-of-science-logo.png" alt="Stephen Wolfram's A New Kind of Science | Online" width="372" height="43" class="fill"></a></h1> </div> <div class="padding-r-1"> <nav id="nav" class="table"> <div class="nav-main heirs-vertical-align-t"> <div class="size-picker"> <i class="icon"></i> <ul> <li class="size-auto checked"><span>auto</span></li> <li class="size-small"><span>small</span></li> <li class="size-medium"><span>medium</span></li> <li class="size-large"><span>large</span></li> <li class="size-x-large"><span>x-large</span></li> </ul> </div> <div class="search"> <div><i class="icon"></i></div> </div> <div class="toc"> <div><a href="/nks/"><i class="icon"></i></a></div> </div> <div class="breadcrumbs"> <div> <span class="notes"><a href="/nks/notes-for-chapters/">Notes</a></span> <span class="chapter"><a href="/nks/chap-10--processes-of-perception-and-analysis--notes/">Chapter 10</a></span> <span class="section"><a href="/nks/sect-10-5--data-compression--notes/">Section 5</a></span> <span class="page"><a href="/nks/p1069">Page 1069</a></span> </div> </div> </div> <div class="nav-burger"> <div> <div class="hamburger"><i class="icon"></i></div> </div> <div class="burger-nav-l1"> <div id="top-stuff"> <div class="crumb-wrap-supreme"> <div> <span class="toc"><a href="/nks/"><i class="icon"></i></a></span> </div> </div> <div class="size-picker"> <div><i class="icon"></i></div> </div> <ul class="size-pickings"> <li class="size-auto checked"><span>auto</span></li> <li class="size-small"><span>small</span></li> <li class="size-medium"><span>medium</span></li> <li class="size-large"><span>large</span></li> <li class="size-x-large"><span>x-large</span></li> </ul> </div> <ul class="chapters"> <li class="chapter preface"> <a href="/nks/pix--preface/"> <span></span> <span>Preface</span> </a> </li> <li class="chapter chapter-1"> <a href="/nks/chap-1--the-foundations-for-a-new-kind-of-science/"> <span>1</span> <span>The Foundations for a New Kind of Science</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p1--an-outline-of-basic-ideas/"> <span>1</span> <span>An Outline of Basic Ideas</span> </a> </li> <li class="section section-2"> <a href="/nks/p7--relations-to-other-areas/"> <span>2</span> <span>Relations to Other Areas</span> </a> </li> <li class="section section-3"> <a href="/nks/p12--some-past-initiatives/"> <span>3</span> <span>Some Past Initiatives</span> </a> </li> <li class="section section-4"> <a href="/nks/p17--the-personal-story-of-the-science-in-this-book/"> <span>4</span> <span>The Personal Story of the Science in This Book</span> </a> </li> </ul> </li> <li class="chapter chapter-2"> <a href="/nks/chap-2--the-crucial-experiment/"> <span>2</span> <span>The Crucial Experiment</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p23--how-do-simple-programs-behave/"> <span>1</span> <span>How Do Simple Programs Behave?</span> </a> </li> <li class="section section-2"> <a href="/nks/p39--the-need-for-a-new-intuition/"> <span>2</span> <span>The Need for a New Intuition</span> </a> </li> <li class="section section-3"> <a href="/nks/p42--why-these-discoveries-were-not-made-before/"> <span>3</span> <span>Why These Discoveries Were Not Made Before</span> </a> </li> </ul> </li> <li class="chapter chapter-3"> <a href="/nks/chap-3--the-world-of-simple-programs/"> <span>3</span> <span>The World of Simple Programs</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p51--the-search-for-general-features/"> <span>1</span> <span>The Search for General Features</span> </a> </li> <li class="section section-2"> <a href="/nks/p53--more-cellular-automata/"> <span>2</span> <span>More Cellular Automata</span> </a> </li> <li class="section section-3"> <a href="/nks/p71--mobile-automata/"> <span>3</span> <span>Mobile Automata</span> </a> </li> <li class="section section-4"> <a href="/nks/p78--turing-machines/"> <span>4</span> <span>Turing Machines</span> </a> </li> <li class="section section-5"> <a href="/nks/p82--substitution-systems/"> <span>5</span> <span>Substitution Systems</span> </a> </li> <li class="section section-6"> <a href="/nks/p88--sequential-substitution-systems/"> <span>6</span> <span>Sequential Substitution Systems</span> </a> </li> <li class="section section-7"> <a href="/nks/p93--tag-systems/"> <span>7</span> <span>Tag Systems</span> </a> </li> <li class="section section-8"> <a href="/nks/p95--cyclic-tag-systems/"> <span>8</span> <span>Cyclic Tag Systems</span> </a> </li> <li class="section section-9"> <a href="/nks/p97--register-machines/"> <span>9</span> <span>Register Machines</span> </a> </li> <li class="section section-10"> <a href="/nks/p102--symbolic-systems/"> <span>10</span> <span>Symbolic Systems</span> </a> </li> <li class="section section-11"> <a href="/nks/p105--some-conclusions/"> <span>11</span> <span>Some Conclusions</span> </a> </li> <li class="section section-12"> <a href="/nks/p108--how-the-discoveries-in-this-chapter-were-made/"> <span>12</span> <span>How the Discoveries in This Chapter Were Made</span> </a> </li> </ul> </li> <li class="chapter chapter-4"> <a href="/nks/chap-4--systems-based-on-numbers/"> <span>4</span> <span>Systems Based on Numbers</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p115--the-notion-of-numbers/"> <span>1</span> <span>The Notion of Numbers</span> </a> </li> <li class="section section-2"> <a href="/nks/p117--elementary-arithmetic/"> <span>2</span> <span>Elementary Arithmetic</span> </a> </li> <li class="section section-3"> <a href="/nks/p128--recursive-sequences/"> <span>3</span> <span>Recursive Sequences</span> </a> </li> <li class="section section-4"> <a href="/nks/p132--the-sequence-of-primes/"> <span>4</span> <span>The Sequence of Primes</span> </a> </li> <li class="section section-5"> <a href="/nks/p136--mathematical-constants/"> <span>5</span> <span>Mathematical Constants</span> </a> </li> <li class="section section-6"> <a href="/nks/p145--mathematical-functions/"> <span>6</span> <span>Mathematical Functions</span> </a> </li> <li class="section section-7"> <a href="/nks/p149--iterated-maps-and-the-chaos-phenomenon/"> <span>7</span> <span>Iterated Maps and the Chaos Phenomenon</span> </a> </li> <li class="section section-8"> <a href="/nks/p155--continuous-cellular-automata/"> <span>8</span> <span>Continuous Cellular Automata</span> </a> </li> <li class="section section-9"> <a href="/nks/p161--partial-differential-equations/"> <span>9</span> <span>Partial Differential Equations</span> </a> </li> <li class="section section-10"> <a href="/nks/p167--continuous-versus-discrete-systems/"> <span>10</span> <span>Continuous Versus Discrete Systems</span> </a> </li> </ul> </li> <li class="chapter chapter-5"> <a href="/nks/chap-5--two-dimensions-and-beyond/"> <span>5</span> <span>Two Dimensions and Beyond</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p169--introduction/"> <span>1</span> <span>Introduction</span> </a> </li> <li class="section section-2"> <a href="/nks/p170--cellular-automata/"> <span>2</span> <span>Cellular Automata</span> </a> </li> <li class="section section-3"> <a href="/nks/p184--turing-machines/"> <span>3</span> <span>Turing Machines</span> </a> </li> <li class="section section-4"> <a href="/nks/p187--substitution-systems-and-fractals/"> <span>4</span> <span>Substitution Systems and Fractals</span> </a> </li> <li class="section section-5"> <a href="/nks/p193--network-systems/"> <span>5</span> <span>Network Systems</span> </a> </li> <li class="section section-6"> <a href="/nks/p204--multiway-systems/"> <span>6</span> <span>Multiway Systems</span> </a> </li> <li class="section section-7"> <a href="/nks/p210--systems-based-on-constraints/"> <span>7</span> <span>Systems Based on Constraints</span> </a> </li> </ul> </li> <li class="chapter chapter-6"> <a href="/nks/chap-6--starting-from-randomness/"> <span>6</span> <span>Starting from Randomness</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p223--the-emergence-of-order/"> <span>1</span> <span>The Emergence of Order</span> </a> </li> <li class="section section-2"> <a href="/nks/p231--four-classes-of-behavior/"> <span>2</span> <span>Four Classes of Behavior</span> </a> </li> <li class="section section-3"> <a href="/nks/p250--sensitivity-to-initial-conditions/"> <span>3</span> <span>Sensitivity to Initial Conditions</span> </a> </li> <li class="section section-4"> <a href="/nks/p255--systems-of-limited-size-and-class-2-behavior/"> <span>4</span> <span>Systems of Limited Size and Class 2 Behavior</span> </a> </li> <li class="section section-5"> <a href="/nks/p261--randomness-in-class-3-systems/"> <span>5</span> <span>Randomness in Class 3 Systems</span> </a> </li> <li class="section section-6"> <a href="/nks/p266--special-initial-conditions/"> <span>6</span> <span>Special Initial Conditions</span> </a> </li> <li class="section section-7"> <a href="/nks/p275--the-notion-of-attractors/"> <span>7</span> <span>The Notion of Attractors</span> </a> </li> <li class="section section-8"> <a href="/nks/p281--structures-in-class-4-systems/"> <span>8</span> <span>Structures in Class 4 Systems</span> </a> </li> </ul> </li> <li class="chapter chapter-7"> <a href="/nks/chap-7--mechanisms-in-programs-and-nature/"> <span>7</span> <span>Mechanisms in Programs and Nature</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p297--universality-of-behavior/"> <span>1</span> <span>Universality of Behavior</span> </a> </li> <li class="section section-2"> <a href="/nks/p299--three-mechanisms-for-randomness/"> <span>2</span> <span>Three Mechanisms for Randomness</span> </a> </li> <li class="section section-3"> <a href="/nks/p301--randomness-from-the-environment/"> <span>3</span> <span>Randomness from the Environment</span> </a> </li> <li class="section section-4"> <a href="/nks/p304--chaos-theory-and-randomness-from-initial-conditions/"> <span>4</span> <span>Chaos Theory and Randomness from Initial Conditions</span> </a> </li> <li class="section section-5"> <a href="/nks/p315--the-intrinsic-generation-of-randomness/"> <span>5</span> <span>The Intrinsic Generation of Randomness</span> </a> </li> <li class="section section-6"> <a href="/nks/p327--the-phenomenon-of-continuity/"> <span>6</span> <span>The Phenomenon of Continuity</span> </a> </li> <li class="section section-7"> <a href="/nks/p337--origins-of-discreteness/"> <span>7</span> <span>Origins of Discreteness</span> </a> </li> <li class="section section-8"> <a href="/nks/p342--the-problem-of-satisfying-constraints/"> <span>8</span> <span>The Problem of Satisfying Constraints</span> </a> </li> <li class="section section-9"> <a href="/nks/p351--origins-of-simple-behavior/"> <span>9</span> <span>Origins of Simple Behavior</span> </a> </li> </ul> </li> <li class="chapter chapter-8"> <a href="/nks/chap-8--implications-for-everyday-systems/"> <span>8</span> <span>Implications for Everyday Systems</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p363--issues-of-modelling/"> <span>1</span> <span>Issues of Modelling</span> </a> </li> <li class="section section-2"> <a href="/nks/p369--the-growth-of-crystals/"> <span>2</span> <span>The Growth of Crystals</span> </a> </li> <li class="section section-3"> <a href="/nks/p374--the-breaking-of-materials/"> <span>3</span> <span>The Breaking of Materials</span> </a> </li> <li class="section section-4"> <a href="/nks/p376--fluid-flow/"> <span>4</span> <span>Fluid Flow</span> </a> </li> <li class="section section-5"> <a href="/nks/p383--fundamental-issues-in-biology/"> <span>5</span> <span>Fundamental Issues in Biology</span> </a> </li> <li class="section section-6"> <a href="/nks/p400--growth-of-plants-and-animals/"> <span>6</span> <span>Growth of Plants and Animals</span> </a> </li> <li class="section section-7"> <a href="/nks/p422--biological-pigmentation-patterns/"> <span>7</span> <span>Biological Pigmentation Patterns</span> </a> </li> <li class="section section-8"> <a href="/nks/p429--financial-systems/"> <span>8</span> <span>Financial Systems</span> </a> </li> </ul> </li> <li class="chapter chapter-9"> <a href="/nks/chap-9--fundamental-physics/"> <span>9</span> <span>Fundamental Physics</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p433--the-problems-of-physics/"> <span>1</span> <span>The Problems of Physics</span> </a> </li> <li class="section section-2"> <a href="/nks/p435--the-notion-of-reversibility/"> <span>2</span> <span>The Notion of Reversibility</span> </a> </li> <li class="section section-3"> <a href="/nks/p441--irreversibility-and-the-second-law-of-thermodynamics/"> <span>3</span> <span>Irreversibility and the Second Law of Thermodynamics</span> </a> </li> <li class="section section-4"> <a href="/nks/p458--conserved-quantities-and-continuum-phenomena/"> <span>4</span> <span>Conserved Quantities and Continuum Phenomena</span> </a> </li> <li class="section section-5"> <a href="/nks/p465--ultimate-models-for-the-universe/"> <span>5</span> <span>Ultimate Models for the Universe</span> </a> </li> <li class="section section-6"> <a href="/nks/p472--the-nature-of-space/"> <span>6</span> <span>The Nature of Space</span> </a> </li> <li class="section section-7"> <a href="/nks/p475--space-as-a-network/"> <span>7</span> <span>Space as a Network</span> </a> </li> <li class="section section-8"> <a href="/nks/p481--the-relationship-of-space-and-time/"> <span>8</span> <span>The Relationship of Space and Time</span> </a> </li> <li class="section section-9"> <a href="/nks/p486--time-and-causal-networks/"> <span>9</span> <span>Time and Causal Networks</span> </a> </li> <li class="section section-10"> <a href="/nks/p497--the-sequencing-of-events-in-the-universe/"> <span>10</span> <span>The Sequencing of Events in the Universe</span> </a> </li> <li class="section section-11"> <a href="/nks/p504--uniqueness-and-branching-in-time/"> <span>11</span> <span>Uniqueness and Branching in Time</span> </a> </li> <li class="section section-12"> <a href="/nks/p508--evolution-of-networks/"> <span>12</span> <span>Evolution of Networks</span> </a> </li> <li class="section section-13"> <a href="/nks/p516--space-time-and-relativity/"> <span>13</span> <span>Space, Time and Relativity</span> </a> </li> <li class="section section-14"> <a href="/nks/p525--elementary-particles/"> <span>14</span> <span>Elementary Particles</span> </a> </li> <li class="section section-15"> <a href="/nks/p530--the-phenomenon-of-gravity/"> <span>15</span> <span>The Phenomenon of Gravity</span> </a> </li> <li class="section section-16"> <a href="/nks/p537--quantum-phenomena/"> <span>16</span> <span>Quantum Phenomena</span> </a> </li> </ul> </li> <li class="chapter chapter-10"> <a href="/nks/chap-10--processes-of-perception-and-analysis/"> <span>10</span> <span>Processes of Perception and Analysis</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p547--introduction/"> <span>1</span> <span>Introduction</span> </a> </li> <li class="section section-2"> <a href="/nks/p548--what-perception-and-analysis-do/"> <span>2</span> <span>What Perception and Analysis Do</span> </a> </li> <li class="section section-3"> <a href="/nks/p552--defining-the-notion-of-randomness/"> <span>3</span> <span>Defining the Notion of Randomness</span> </a> </li> <li class="section section-4"> <a href="/nks/p557--defining-complexity/"> <span>4</span> <span>Defining Complexity</span> </a> </li> <li class="section section-5"> <a href="/nks/p560--data-compression/"> <span>5</span> <span>Data Compression</span> </a> </li> <li class="section section-6"> <a href="/nks/p572--irreversible-data-compression/"> <span>6</span> <span>Irreversible Data Compression</span> </a> </li> <li class="section section-7"> <a href="/nks/p577--visual-perception/"> <span>7</span> <span>Visual Perception</span> </a> </li> <li class="section section-8"> <a href="/nks/p585--auditory-perception/"> <span>8</span> <span>Auditory Perception</span> </a> </li> <li class="section section-9"> <a href="/nks/p588--statistical-analysis/"> <span>9</span> <span>Statistical Analysis</span> </a> </li> <li class="section section-10"> <a href="/nks/p598--cryptography-and-cryptanalysis/"> <span>10</span> <span>Cryptography and Cryptanalysis</span> </a> </li> <li class="section section-11"> <a href="/nks/p606--traditional-mathematics-and-mathematical-formulas/"> <span>11</span> <span>Traditional Mathematics and Mathematical Formulas</span> </a> </li> <li class="section section-12"> <a href="/nks/p620--human-thinking/"> <span>12</span> <span>Human Thinking</span> </a> </li> <li class="section section-13"> <a href="/nks/p632--higher-forms-of-perception-and-analysis/"> <span>13</span> <span>Higher Forms of Perception and Analysis</span> </a> </li> </ul> </li> <li class="chapter chapter-11"> <a href="/nks/chap-11--the-notion-of-computation/"> <span>11</span> <span>The Notion of Computation</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p637--computation-as-a-framework/"> <span>1</span> <span>Computation as a Framework</span> </a> </li> <li class="section section-2"> <a href="/nks/p638--computations-in-cellular-automata/"> <span>2</span> <span>Computations in Cellular Automata</span> </a> </li> <li class="section section-3"> <a href="/nks/p642--the-phenomenon-of-universality/"> <span>3</span> <span>The Phenomenon of Universality</span> </a> </li> <li class="section section-4"> <a href="/nks/p644--a-universal-cellular-automaton/"> <span>4</span> <span>A Universal Cellular Automaton</span> </a> </li> <li class="section section-5"> <a href="/nks/p656--emulating-other-systems-with-cellular-automata/"> <span>5</span> <span>Emulating Other Systems with Cellular Automata</span> </a> </li> <li class="section section-6"> <a href="/nks/p664--emulating-cellular-automata-with-other-systems/"> <span>6</span> <span>Emulating Cellular Automata with Other Systems</span> </a> </li> <li class="section section-7"> <a href="/nks/p674--implications-of-universality/"> <span>7</span> <span>Implications of Universality</span> </a> </li> <li class="section section-8"> <a href="/nks/p675--the-rule-110-cellular-automaton/"> <span>8</span> <span>The Rule 110 Cellular Automaton</span> </a> </li> <li class="section section-9"> <a href="/nks/p690--the-significance-of-universality-in-rule-110/"> <span>9</span> <span>The Significance of Universality in Rule 110</span> </a> </li> <li class="section section-10"> <a href="/nks/p691--class-4-behavior-and-universality/"> <span>10</span> <span>Class 4 Behavior and Universality</span> </a> </li> <li class="section section-11"> <a href="/nks/p694--the-threshold-of-universality-in-cellular-automata/"> <span>11</span> <span>The Threshold of Universality in Cellular Automata</span> </a> </li> <li class="section section-12"> <a href="/nks/p706--universality-in-turing-machines-and-other-systems/"> <span>12</span> <span>Universality in Turing Machines and Other Systems</span> </a> </li> </ul> </li> <li class="chapter chapter-12"> <a href="/nks/chap-12--the-principle-of-computational-equivalence/"> <span>12</span> <span>The Principle of Computational Equivalence</span> </a> <ul class="sections"> <li class="section section-1"> <a href="/nks/p715--basic-framework/"> <span>1</span> <span>Basic Framework</span> </a> </li> <li class="section section-2"> <a href="/nks/p716--outline-of-the-principle/"> <span>2</span> <span>Outline of the Principle</span> </a> </li> <li class="section section-3"> <a href="/nks/p719--the-content-of-the-principle/"> <span>3</span> <span>The Content of the Principle</span> </a> </li> <li class="section section-4"> <a href="/nks/p726--the-validity-of-the-principle/"> <span>4</span> <span>The Validity of the Principle</span> </a> </li> <li class="section section-5"> <a href="/nks/p735--explaining-the-phenomenon-of-complexity/"> <span>5</span> <span>Explaining the Phenomenon of Complexity</span> </a> </li> <li class="section section-6"> <a href="/nks/p737--computational-irreducibility/"> <span>6</span> <span>Computational Irreducibility</span> </a> </li> <li class="section section-7"> <a href="/nks/p750--the-phenomenon-of-free-will/"> <span>7</span> <span>The Phenomenon of Free Will</span> </a> </li> <li class="section section-8"> <a href="/nks/p753--undecidability-and-intractability/"> <span>8</span> <span>Undecidability and Intractability</span> </a> </li> <li class="section section-9"> <a href="/nks/p772--implications-for-mathematics-and-its-foundations/"> <span>9</span> <span>Implications for Mathematics and Its Foundations</span> </a> </li> <li class="section section-10"> <a href="/nks/p822--intelligence-in-the-universe/"> <span>10</span> <span>Intelligence in the Universe</span> </a> </li> <li class="section section-11"> <a href="/nks/p840--implications-for-technology/"> <span>11</span> <span>Implications for Technology</span> </a> </li> <li class="section section-12"> <a href="/nks/p844--historical-perspectives/"> <span>12</span> <span>Historical Perspectives</span> </a> </li> </ul> </li> <li class="chapter notes selected"> <a href="/nks/notes-for-chapters/"> <span></span> <span>Notes</span> </a> </li> </ul> <div class="search"> </div> </div> </div> </nav> </div> </div> </div> <div id="search"> <div class="text-align-c__600"> <div class="inline-grid cols-3 cols-1__600"> <form id="search-index-form" action="/nks/index/"><div><label for="search-index-input">Look up in index</label><input type="search" id="search-index-input" name="search-index-input"></div></form> <form id="search-all-form" action="/nks/search/"><div><label for="search-all-input">Text search</label><input type="search" id="search-all-input" name="search-all-input"></div></form> <form id="search-jump-form"><div><label for="search-jump-input">Jump to page</label><input type="search" id="search-jump-input" name="search-jump-input"></div></form> </div> </div> </div> </header>  <main id="main"> <div class="wide-wrapper"> <div id="previous"><a href="/nks/notes-10-5--practicalities-of-data-compression/"><span>‹</span></a></div> <div id="next"><a href="/nks/notes-10-5--number-representations/"><span>›</span></a></div> <div id="content" class="wide-padding"> <div class="ribbon-wrapper top"> </div> <h1 class="main-heading">Notes</h1> <h2 class="chapter-heading"><a href="/nks/p547--introduction/">Chapter 10: <span>Processes of Perception and Analysis</span></a></h2> <h2 class="section-heading"><a href="/nks/p560--data-compression">Section 5: <span>Data Compression</span></a></h2> <hr> <div class="text-content"><p class="notehead">History [of data compression]</p><p class="notetext">Morse code, invented in 1838 for use in telegraphy, is an early example of data compression based on using shorter codewords for letters such as "e" and "t" that are more common in English. Modern work on data compression began in the late 1940s with the development of information theory. In 1949 <a href="/nks/index/names/p-s/?search=Shannon,+Claude+E." class="names-index-link">Claude Shannon</a> and <a href="/nks/index/names/f-j/?search=Fano,+Robert+M." class="names-index-link">Robert Fano</a> devised a systematic way to assign codewords based on probabilities of blocks. An optimal method for doing this was then found by <a href="/nks/index/names/f-j/?search=Huffman,+David+A." class="names-index-link">David Huffman</a> in 1951. Early implementations were typically done in hardware, with specific choices of codewords being made as compromises between compression and error correction. In the mid-1970s, the idea emerged of dynamically updating codewords for Huffman encoding, based on the actual data encountered. And in the late 1970s, with online storage of text files becoming common, software compression programs began to be developed, almost all based on adaptive Huffman coding. In 1977 <a href="/nks/index/names/k-o/?search=Lempel,+Abraham" class="names-index-link">Abraham Lempel</a> and <a href="/nks/index/names/t-z/?search=Ziv,+Jacob" class="names-index-link">Jacob Ziv</a> suggested the basic idea of pointer-based encoding. In the mid-1980s, following work by <a href="/nks/index/names/t-z/?search=Welch,+Terry+A." class="names-index-link">Terry Welch</a>, the so-called LZW algorithm rapidly became the method of choice for most general-purpose compression systems. It was used in programs such as PKZIP, as well as in hardware devices such as modems. In the late 1980s, digital images became more common, and standards for compressing them emerged. In the early 1990s, lossy compression methods (to be discussed in the next section) also began to be widely used. Current image compression standards include: FAX CCITT 3 (run-length encoding, with codewords determined by Huffman coding from a definite distribution of run lengths); GIF (LZW); JPEG (lossy discrete cosine transform, then Huffman or arithmetic coding); BMP (run-length encoding, etc.); TIFF (FAX, JPEG, GIF, etc.). Typical compression ratios currently achieved for text are around 3:1, for line diagrams and text images around 3:1, and for photographic images around 2:1 lossless, and 20:1 lossy. (For sound compression see page <a href="/nks/notes-10-8--sound-compression/">1080</a>.)</p></div> </div> <div class="ribbon-wrapper stripe"> </div> <footer id="footer" class="wide-padding"> <section id="related-pages" class="width-full max-width-900 center"> <h2 class="display-ib">Pages Related to this Note:</h2> <span> <a href="/nks/p560--data-compression/">560</a>, <a href="/nks/p576--irreversible-data-compression/">576</a> </span> <div> <div class="thumbnails cf"> <a href="/nks/p560--data-compression/"><img src="/nks/pages/p560--data-compression/image-tiny.png" alt="Page 560" title="Page 560" width="70" height="86"></a> <a href="/nks/p576--irreversible-data-compression/"><img src="/nks/pages/p576--irreversible-data-compression/image-tiny.png" alt="Page 576" title="Page 576" width="70" height="86"></a> </div> </div> </section><br> <section id="related-notes" class="width-full max-width-900 center"> <h2>Other Notes From Page <a href="/nks/p1069">1069</a>:</h2> <div> <div class="thumbnails cf"> </div> <ul class="cols-2 cols-1__600 heirs-width-full__600 show-thumbnails"> <li><span><a href="/nks/notes-10-4--history-of-complexity-definitions/"><span><img src="/nks/img/thumbnails/notes-10-4--history-of-complexity-definitions--textonly.png" alt="" width="76" height="62"></span><span>History [of complexity definitions]</span></a></span></li> <li><span><a href="/nks/notes-10-5--practicalities-of-data-compression/"><span><img src="/nks/img/thumbnails/notes-10-5--practicalities-of-data-compression--textonly.png" alt="" width="76" height="62"></span><span>Practicalities [of data compression]</span></a></span></li> </ul> </div> </section> <section id="image-sources" class="width-full max-width-900 center hide"> <hr class="even"> <h2>Image Source Notebooks:</h2> <div> <ul class="grid cols-2 width-full cols-1__600 heirs-width-full__600"> </section> <section id="related-resources" class="width-full max-width-900" style="display: none;"> <hr class="even"> <h2>External Resources Related to this Page:</h2> <div> <ul class="grid cols-2 width-full heirs-width-1-2 cols-1__600 heirs-width-full__600"> </ul> </div> </section>  <footer id="inline-footer"> <div class="heirs-display-ib heirs-margin-bt-1-2__600 heirs-display-b__600"> <div> <a href="/nks/sect-10-5--data-compression--notes/" class="view-notes mini-t"> <span><img src="/nks/img/icon-notes.png" alt="" width="8" height="8"></span> <span>All notes for this section</span> </a> </div> <div> <a href="/nks/programs/" class="programs-link mini-t"> <span><img src="/nks/img/icon-download.png" alt="" width="11" height="13"></span> <span>Download programs</span> </a> </div> <div> <span class="pdf-link mini-t"> <span><img src="/nks/img/icon-pdf.png" alt="" width="10" height="13"></span> <span>Show PDF</span> <span class="tooltip"> <span class="close">✖</span> <a href="https://files.wolframcdn.com/pub/www.wolframscience.com/nks/nks-nts-ch10.pdf" class="chapter">Show Chapter Notes</a><br> <a href="https://files.wolframcdn.com/pub/www.wolframscience.com/nks/nks-notes.pdf" class="all">Show All Notes</a> </span> </span> </div> </div> </footer>  <script type="application/ld+json"> { "@context": "http://schema.org", "@type": "Book", "name": "A New Kind of Science", "description": "Starting from a collection of simple computer experiments Stephen Wolfram shows how their unexpected results force a whole new way of looking at the operation of our universe. Wolfram uses his approach to tackle a remarkable array of fundamental problems in science, from the origins of apparent randomness in physical systems, to the development of complexity in biology, the ultimate scope and limitations of mathematics, the possibility of a truly fundamental theory of physics, the interplay between free will and determinism, and the character of intelligence in the universe.", "author": { "@type": "Person", "name": "Stephen Wolfram" }, "publisher": { "@type": "Corporation", "name": "Wolfram Media, Inc.", "url": "http://www.wolfram-media.com/" }, "isbn": "1-57955-008-8", "datePublished": "2002", "numberOfPages": "1197", "bookFormat": "print, online, iPad", "genre": "Science, Cellular Automata, Computation, Biology, Physics", "audience": "Scientists, Researchers, Science Educators, Biologists, Physicists, Mathematicians, Programmers", "discussionUrl": "http://community.wolfram.com/content?curTag=wolfram%20science", "inLanguage": "en" } </script>   </footer>  <footer id="citation"> <p>From Stephen Wolfram: <em>A New Kind of Science</em> [<a href="/nks/citation/">citation</a>]   <a href="https://creativecommons.org/licenses/by-sa/4.0/" target="_blank"> <svg id="icon-cc-by-sa" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 110 32"> <g transform="translate(76 0)"> <path d="M15.7,0.9c4.2,0,7.7,1.5,10.7,4.4c2.9,2.9,4.4,6.4,4.4,10.6c0,4.2-1.4,7.7-4.3,10.6c-3,3-6.6,4.4-10.7,4.4 c-4.1,0-7.6-1.5-10.6-4.4C2.2,23.5,0.7,20,0.7,15.9c0-4.1,1.5-7.6,4.4-10.6C8.1,2.3,11.6,0.9,15.7,0.9z M15.8,3.6 c-3.4,0-6.3,1.2-8.7,3.6c-2.4,2.5-3.7,5.4-3.7,8.7c0,3.3,1.2,6.2,3.6,8.6c2.4,2.4,5.3,3.6,8.7,3.6c3.3,0,6.2-1.2,8.7-3.7 c2.4-2.3,3.5-5.2,3.5-8.6c0-3.4-1.2-6.3-3.6-8.7C22.1,4.8,19.2,3.6,15.8,3.6z M9.1,13.8c0.3-1.8,1-3.3,2.2-4.3 c1.2-1,2.7-1.5,4.4-1.5c2.4,0,4.2,0.8,5.6,2.3s2.1,3.5,2.1,5.8c0,2.3-0.7,4.2-2.2,5.7c-1.4,1.5-3.3,2.3-5.6,2.3 c-1.7,0-3.2-0.5-4.4-1.5c-1.2-1-2-2.5-2.2-4.3h3.8c0.1,1.8,1.2,2.7,3.3,2.7c1.1,0,1.9-0.5,2.5-1.4c0.6-0.9,1-2.1,1-3.6 c0-1.6-0.3-2.8-0.9-3.6c-0.6-0.8-1.4-1.2-2.5-1.2c-2,0-3.1,0.9-3.4,2.7h1.1l-3,3l-3-3L9.1,13.8L9.1,13.8z" fill="currentColor"></path> </g> <g transform="translate(5 0)"> <path d="M59.4,5.2c-2.9-2.9-6.4-4.3-10.7-4.3c-4.1,0-7.7,1.4-10.6,4.3c-2.9,3-4.4,6.6-4.4,10.7s1.5,7.6,4.4,10.6 c3,2.9,6.5,4.4,10.6,4.4c4.1,0,7.7-1.5,10.7-4.5c2.9-2.8,4.3-6.3,4.3-10.5C63.7,11.7,62.3,8.1,59.4,5.2z M57.5,24.5 c-2.5,2.4-5.4,3.7-8.7,3.7c-3.4,0-6.2-1.2-8.7-3.6c-2.4-2.4-3.6-5.3-3.6-8.6c0-3.3,1.2-6.2,3.7-8.7c2.4-2.4,5.2-3.6,8.7-3.6 c3.4,0,6.3,1.2,8.7,3.6c2.4,2.4,3.6,5.3,3.6,8.7C61.1,19.3,59.9,22.2,57.5,24.5z" fill="currentColor"></path> <path d="M46.7,8.3c0-1.4,0.7-2.1,2.1-2.1s2.1,0.7,2.1,2.1c0,1.4-0.7,2.1-2.1,2.1C47.3,10.4,46.7,9.7,46.7,8.3z" fill="currentColor"></path> <path d="M52.8,18.3h-1.7v7.3h-4.7v-7.3h-1.7c0,0,0-4.1,0-6.1c0-0.6,0.4-1,1-1c2,0,4,0,6,0c0.6,0,1,0.4,1,1 C52.8,14.2,52.8,18.3,52.8,18.3z" fill="currentColor"></path> </g> <g id="cc-logo"> <path d="M15.7,0.9c4.2,0,7.8,1.5,10.7,4.4c1.4,1.4,2.5,3,3.2,4.8c0.7,1.8,1.1,3.7,1.1,5.8c0,2.1-0.4,4-1.1,5.8 c-0.7,1.8-1.8,3.4-3.2,4.8c-1.4,1.4-3.1,2.6-5,3.3c-1.8,0.8-3.8,1.2-5.8,1.2c-2,0-3.9-0.4-5.7-1.1c-1.8-0.8-3.5-1.9-4.9-3.3 c-1.4-1.4-2.5-3.1-3.3-4.9c-0.8-1.8-1.1-3.7-1.1-5.7c0-2,0.4-3.9,1.1-5.7c0.8-1.8,1.9-3.5,3.3-4.9C8,2.3,11.6,0.9,15.7,0.9z M15.8,3.6c-3.4,0-6.3,1.2-8.7,3.6C6,8.3,5,9.7,4.4,11.2c-0.6,1.5-1,3.1-0.9,4.7c0,1.6,0.3,3.2,0.9,4.6C5,22,6,23.4,7.1,24.5 c1.1,1.1,2.5,2.1,4,2.7c1.5,0.6,3,0.9,4.7,0.9c1.6,0,3.2-0.3,4.7-0.9c1.5-0.6,2.9-1.6,4.1-2.7c2.3-2.3,3.5-5.2,3.5-8.6 c0-1.7-0.3-3.2-0.9-4.7c-0.6-1.5-1.5-2.8-2.7-4C22.1,4.8,19.2,3.6,15.8,3.6z M15.6,13.4l-2,1c-0.2-0.4-0.5-0.8-0.8-0.9 c-0.3-0.2-0.6-0.3-0.9-0.3c-1.3,0-2,0.9-2,2.7c0,0.8,0.2,1.4,0.5,1.9c0.3,0.5,0.8,0.7,1.5,0.7c0.9,0,1.5-0.4,1.9-1.3l1.8,0.9 c-0.4,0.7-0.9,1.3-1.6,1.7c-0.7,0.4-1.5,0.6-2.3,0.6c-1.3,0-2.4-0.4-3.2-1.2c-0.8-0.8-1.2-2-1.2-3.4c0-1.4,0.4-2.6,1.2-3.4 c0.8-0.8,1.9-1.3,3.1-1.3C13.4,11.2,14.8,11.9,15.6,13.4L15.6,13.4z M24.2,13.4l-2,1c-0.2-0.4-0.5-0.8-0.8-0.9 c-0.3-0.2-0.6-0.3-0.9-0.3c-1.3,0-2,0.9-2,2.7c0,0.8,0.2,1.4,0.5,1.9c0.3,0.5,0.8,0.7,1.5,0.7c0.9,0,1.5-0.4,1.8-1.3l1.9,0.9 c-0.4,0.7-1,1.3-1.7,1.7c-0.7,0.4-1.5,0.6-2.3,0.6c-1.4,0-2.4-0.4-3.3-1.2c-0.8-0.8-1.2-2-1.2-3.4c0-1.4,0.4-2.6,1.2-3.4 c0.8-0.8,1.9-1.3,3.1-1.3C22.1,11.2,23.4,11.9,24.2,13.4L24.2,13.4z" fill="currentColor"></path> </g> </svg></a> <a href="https://en.wikipedia.org/wiki/Wikipedia:Citing_sources" target="_blank"> <svg id="icon-wiki" xmlns="http://www.w3.org/2000/svg" viewBox="101 0 44 32"> <g transform="translate(0 0)"> <path id="wikipedia" d="M129.6,4.5l0.3,0h0.5l0.2,0h0.4l0.2,0l0.5,0h0.2l0.1,0.1v0.7c0,0.2-0.1,0.3-0.3,0.3c-1.1,0-1.3,0.2-1.7,0.7 c-0.2,0.3-0.6,1-1.1,1.7l-3.7,7l-0.1,0.2l4.6,9.4l0.3,0.1l7.2-17.1c0.3-0.7,0.2-1.2-0.1-1.5c-0.3-0.3-0.6-0.4-1.4-0.5l-0.7,0 c-0.1,0-0.2,0-0.2-0.1c-0.1,0-0.1-0.1-0.1-0.2V4.6l0.1-0.1h8.1l0.1,0.1v0.7c0,0.2-0.1,0.3-0.3,0.3c-1.1,0-1.9,0.3-2.4,0.7 c-0.5,0.4-0.9,1-1.2,1.7c0,0-2.2,5-4.4,10.1l-0.4,0.9c-1.6,3.6-3.2,7.2-4.1,9.2c-0.9,1.7-1.7,1.5-2.5,0c-0.6-1.2-1.6-3.3-2.5-5.5 l-0.7-1.4c-0.4-0.8-0.8-1.7-1.1-2.4c-1.5,3.2-3.6,7.4-4.7,9.4c-1,1.8-1.8,1.5-2.5,0c-2.3-5.4-7-15-9.3-20.3c-0.4-1-0.7-1.6-1-1.9 c-0.3-0.2-0.9-0.4-1.8-0.4c-0.3,0-0.5-0.1-0.5-0.3V4.6l0.1-0.1l0.8,0h1.4l1.4,0h0.4l2.5,0h0.3l1.3,0h0.8l0.1,0.1v0.7 c0,0.2-0.1,0.3-0.4,0.3l-0.9,0c-0.8,0-1.2,0.3-1.2,0.7c0,0.2,0.1,0.5,0.3,1c1.8,4.3,7.9,17.3,7.9,17.2l0.2,0.1l3.9-7.9 c-0.5-1.1-0.8-1.7-0.8-1.7L119,9.6c0,0-0.5-1.1-0.7-1.4c-1.2-2.4-1.2-2.5-2.4-2.6c-0.3,0-0.5-0.1-0.5-0.3V4.6l0.1-0.1h1l2.1,0h3.9 l0.2,0.1v0.7c0,0.2-0.1,0.3-0.4,0.3l-0.5,0.1c-1.3,0.1-1.1,0.6-0.2,2.3l2.6,5.3l2.9-5.7c0.5-1,0.4-1.3,0.2-1.5l0,0 c-0.1-0.1-0.5-0.4-1.3-0.4l-0.3,0c-0.1,0-0.2,0-0.2-0.1c-0.1,0-0.1-0.1-0.1-0.2V4.6l0.1-0.1l1.2,0H129.6z" fill="currentColor"></path> </g> </svg></a> </p> </footer>  </div> </main> <script src="/common/js/c2c/1.0/raw.js"></script> <script src="/common/js/clipboard/2.0/clipboard.js"></script> <script src="/nks/js/shared.en.js"></script> <script src="/nks/js/notes.en.js"></script>  <div id ="IPstripe-wrap"></div> <script src="/common/stripe/stripe.en.js"></script> <script type="application/ld+json"> { "@context": "http://schema.org", "@type": "WebSite", "name": "Stephen Wolfram: A New Kind of Science | Online", "description": "The latest on exploring the computational universe, with free online access to Stephen Wolfram's classic 1,200-page breakthrough book, \"A New Kind of Science\".", "keywords": "A New Kind of Science Online, NKS Online, NKSO", "url": "https://www.wolframscience.com/nks/", "image": "https://www.wolframscience.com/nks/img/a-new-kind-of-science-cover.png", "inLanguage": "en", "creator": { "@type": "Organization", "name": "Wolfram" } } </script> <script type="application/ld+json"> { "@context": "http://schema.org", "@type": "WebPage", "name": "Note (b) for Data Compression: A New Kind of Science | Online by Stephen Wolfram [Page 1069]", "description": "History [of data compression] Morse code, invented in 1838 for use in telegraphy, is an early example of data compression... – from A New Kind of Science", "url": "https://www.wolframscience.com/nks/notes-10-5--history-of-data-compression/", "image": "http://www.wolframscience.com/share.png", "inLanguage": "en" } </script>  </body> </html>

CINXE.COM

Note (b) for Data Compression: A New Kind of Science | Online by Stephen Wolfram [Page 1069]