CINXE.COM

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1 plus MathML 2.0 plus SVG 1.1//EN" "http://www.w3.org/2002/04/xhtml-math-svg/xhtml-math-svg-flat.dtd" > <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title> entropy in nLab </title> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta name="robots" content="index,follow" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <link href="/stylesheets/instiki.css?1676280126" media="all" rel="stylesheet" type="text/css" /> <link href="/stylesheets/mathematics.css?1660229990" media="all" rel="stylesheet" type="text/css" /> <link href="/stylesheets/syntax.css?1660229990" media="all" rel="stylesheet" type="text/css" /> <link href="/stylesheets/nlab.css?1676280126" media="all" rel="stylesheet" type="text/css" /> <link rel="stylesheet" type="text/css" href="https://cdn.jsdelivr.net/gh/dreampulse/computer-modern-web-font@master/fonts.css"/> <style type="text/css"> h1#pageName, div.info, .newWikiWord a, a.existingWikiWord, .newWikiWord a:hover, [actiontype="toggle"]:hover, #TextileHelp h3 { color: #226622; } a:visited.existingWikiWord { color: #164416; } </style> <style type="text/css"><![CDATA[/*></style> <script src="/javascripts/prototype.js?1660229990" type="text/javascript"></script> <script src="/javascripts/effects.js?1660229990" type="text/javascript"></script> <script src="/javascripts/dragdrop.js?1660229990" type="text/javascript"></script> <script src="/javascripts/controls.js?1660229990" type="text/javascript"></script> <script src="/javascripts/application.js?1660229990" type="text/javascript"></script> <script src="/javascripts/page_helper.js?1660229990" type="text/javascript"></script> <script src="/javascripts/thm_numbering.js?1660229990" type="text/javascript"></script> <script type="text/x-mathjax-config"> <![CDATA[//><!]]> </script> <script type="text/javascript"> <![CDATA[//><!]]> </script> <link href="https://ncatlab.org/nlab/atom_with_headlines" rel="alternate" title="Atom with headlines" type="application/atom+xml" /> <link href="https://ncatlab.org/nlab/atom_with_content" rel="alternate" title="Atom with full content" type="application/atom+xml" /> <script type="text/javascript"> document.observe("dom:loaded", function() { generateThmNumbers(); }); </script> </head> <body> <div id="Container"> <div id="Content"> <h1 id="pageName"> <span style="float: left; margin: 0.5em 0.25em -0.25em 0"> <svg xmlns="http://www.w3.org/2000/svg" width="1.872em" height="1.8em" viewBox="0 0 190 181"> <path fill="#226622" d="M72.8 145c-1.6 17.3-15.7 10-23.6 20.2-5.6 7.3 4.8 15 11.4 15 11.5-.2 19-13.4 26.4-20.3 3.3-3 8.2-4 11.2-7.2a14 14 0 0 0 2.9-11.1c-1.4-9.6-12.4-18.6-16.9-27.2-5-9.6-10.7-27.4-24.1-27.7-17.4-.3-.4 26 4.7 30.7 2.4 2.3 5.4 4.1 7.3 6.9 1.6 2.3 2.1 5.8-1 7.2-5.9 2.6-12.4-6.3-15.5-10-8.8-10.6-15.5-23-26.2-31.8-5.2-4.3-11.8-8-18-3.7-7.3 4.9-4.2 12.9.2 18.5a81 81 0 0 0 30.7 23c3.3 1.5 12.8 5.6 10 10.7-2.5 5.2-11.7 3-15.6 1.1-8.4-3.8-24.3-21.3-34.4-13.7-3.5 2.6-2.3 7.6-1.2 11.1 2.8 9 12.2 17.2 20.9 20.5 17.3 6.7 34.3-8 50.8-12.1z"/> <path fill="#a41e32" d="M145.9 121.3c-.2-7.5 0-19.6-4.5-26-5.4-7.5-12.9-1-14.1 5.8-1.4 7.8 2.7 14.1 4.8 21.3 3.4 12 5.8 29-.8 40.1-3.6-6.7-5.2-13-7-20.4-2.1-8.2-12.8-13.2-15.1-1.9-2 9.7 9 21.2 12 30.1 1.2 4 2 8.8 6.4 10.3 6.9 2.3 13.3-4.7 17.7-8.8 12.2-11.5 36.6-20.7 43.4-36.4 6.7-15.7-13.7-14-21.3-7.2-9.1 8-11.9 20.5-23.6 25.1 7.5-23.7 31.8-37.6 38.4-61.4 2-7.3-.8-29.6-13-19.8-14.5 11.6-6.6 37.6-23.3 49.2z"/> <path fill="#193c78" d="M86.3 47.5c0-13-10.2-27.6-5.8-40.4 2.8-8.4 14.1-10.1 17-1 3.8 11.6-.3 26.3-1.8 38 11.7-.7 10.5-16 14.8-24.3 2.1-4.2 5.7-9.1 11-6.7 6 2.7 7.4 9.2 6.6 15.1-2.2 14-12.2 18.8-22.4 27-3.4 2.7-8 6.6-5.9 11.6 2 4.4 7 4.5 10.7 2.8 7.4-3.3 13.4-16.5 21.7-16 14.6.7 12 21.9.9 26.2-5 1.9-10.2 2.3-15.2 3.9-5.8 1.8-9.4 8.7-15.7 8.9-6.1.1-9-6.9-14.3-9-14.4-6-33.3-2-44.7-14.7-3.7-4.2-9.6-12-4.9-17.4 9.3-10.7 28 7.2 35.7 12 2 1.1 11 6.9 11.4 1.1.4-5.2-10-8.2-13.5-10-11.1-5.2-30-15.3-35-27.3-2.5-6 2.8-13.8 9.4-13.6 6.9.2 13.4 7 17.5 12C70.9 34 75 43.8 86.3 47.4z"/> </svg> </span> <span class="webName">nLab</span> entropy </h1> <div class="navigation"> <span class="skipNav"><a href='#navEnd'>Skip the Navigation Links</a> | </span> <span style="display:inline-block; width: 0.3em;"></span> <a href="/nlab/show/HomePage" accesskey="H" title="Home page">Home Page</a> | <a href="/nlab/all_pages" accesskey="A" title="List of all pages">All Pages</a> | <a href="/nlab/latest_revisions" accesskey="U" title="Latest edits and page creations">Latest Revisions</a> | <a href="https://nforum.ncatlab.org/discussion/12828/#Item_9" title="Discuss this page in its dedicated thread on the nForum" style="color: black">Discuss this page</a> | <form accept-charset="utf-8" action="/nlab/search" id="navigationSearchForm" method="get"> <fieldset class="search"><input type="text" id="searchField" name="query" value="Search" style="display:inline-block; float: left;" onfocus="this.value == 'Search' ? this.value = '' : true" onblur="this.value == '' ? this.value = 'Search' : true" /></fieldset> </form> <span id='navEnd'></span> </div> <div id="revision"> <html xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xml:lang="en" lang="en"> <head><meta http-equiv="Content-type" content="application/xhtml+xml;charset=utf-8" /><title>Entropy</title></head> <body> <div class="rightHandSide"> <div class="toc clickDown" tabindex="0"> <h3 id="context">Context</h3> <h4 id="measure_and_probability_theory">Measure and probability theory</h4> <div class="hide"><div> <p><strong><a class="existingWikiWord" href="/nlab/show/measure+theory">measure theory</a></strong></p> <p><strong><a class="existingWikiWord" href="/nlab/show/probability+theory">probability theory</a></strong></p> <p>(<a class="existingWikiWord" href="/nlab/show/quantum+probability">quantum probability</a>)</p> <h2 id="measure_theory">Measure theory</h2> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/measurable+space">measurable space</a>, <a class="existingWikiWord" href="/nlab/show/measurable+locale">measurable locale</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/measure">measure</a>, <a class="existingWikiWord" href="/nlab/show/measure+space">measure space</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/von+Neumann+algebra">von Neumann algebra</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/geometric+measure+theory">geometric measure theory</a></p> </li> </ul> <h2 id="probability_theory">Probability theory</h2> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/probability+space">probability space</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/probability+distribution">probability distribution</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/state">state</a></p> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/states+in+AQFT+and+operator+algebra">in AQFT and operator algebra</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/GNS+construction">GNS construction</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/Fell%27s+theorem">Fell's theorem</a></p> </li> </ul> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/entropy">entropy</a>, <a class="existingWikiWord" href="/nlab/show/relative+entropy">relative entropy</a></p> </li> </ul> <h2 id="information_geometry">Information geometry</h2> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/information+geometry">information geometry</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/information+metric">information metric</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/Wasserstein+metric">Wasserstein metric</a></p> </li> </ul> <h2 id="thermodynamics">Thermodynamics</h2> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/thermodynamics">thermodynamics</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/second+law+of+thermodynamics">second law of thermodynamics</a>, <a class="existingWikiWord" href="/nlab/show/generalized+second+law+of+theormodynamics">generalized second law</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/ergodic+theory">ergodic theory</a></p> </li> </ul> <h2 id="theorems">Theorems</h2> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/Riesz+representation+theorem">Riesz representation theorem</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/de+Finetti%27s+theorem">de Finetti's theorem</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/law+of+large+numbers">law of large numbers</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/Kolmogorov+extension+theorem">Kolmogorov extension theorem</a></p> </li> </ul> <h2 id="applications">Applications</h2> <ul> <li><a class="existingWikiWord" href="/nlab/show/machine+learning">machine learning</a>, <a class="existingWikiWord" href="/nlab/show/neural+networks">neural networks</a></li> </ul> </div></div> </div> </div> <h1 id="entropy">Entropy</h1> <div class='maruku_toc'> <ul> <li><a href='#idea'>Idea</a></li> <li><a href='#mathematical_definitions'>Mathematical definitions</a></li> <ul> <li><a href='#preliminary_definitions'>Preliminary definitions</a></li> <ul> <li><a href='#surprisal'>Surprisal</a></li> <li><a href='#almost_partitions'>Almost partitions</a></li> </ul> <li><a href='#entropy_of_a_algebra_on_a_probability_space'>Entropy of a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra on a probability space</a></li> <li><a href='#entropy_of_a_probability_space'>Entropy of a probability space</a></li> <li><a href='#entropy_of_a_partition_of_a_probability_space'>Entropy of a partition of a probability space</a></li> <li><a href='#entropy_of_a_partition_of_a_discrete_probability_space'>Entropy of (a partition of) a discrete probability space</a></li> <li><a href='#entropy_with_respect_to_an_absolutely_continuous_probability_measure_on_the_real_line'>Entropy with respect to an absolutely continuous probability measure on the real line</a></li> <li><a href='#entropy_of_a_density_matrix'>Entropy of a density matrix</a></li> <li><a href='#RelativeEntropy'>Relative entropy</a></li> </ul> <li><a href='#physical'>Physical entropy</a></li> <ul> <li><a href='#gravitational_entropy'>Gravitational entropy</a></li> </ul> <li><a href='#related_concepts'>Related concepts</a></li> <li><a href='#References'>References</a></li> <ul> <li><a href='#ReferencesGeneral'>General</a></li> <li><a href='#ReferencesInQuantumProbabilityTheory'>In quantum probability theory</a></li> <li><a href='#category_theoretic_and_cohomological_interpretations'>Category theoretic and cohomological interpretations</a></li> <li><a href='#ReferencesAxiomaticCharacterization'>Axiomatic characterizations</a></li> </ul> </ul> </div> <h2 id="idea">Idea</h2> <p>Entropy is a measure of disorder, given by the amount of <a class="existingWikiWord" href="/nlab/show/information">information</a> necessary to precisely specify the <a class="existingWikiWord" href="/nlab/show/state">state</a> of a system.</p> <p>Entropy is important in <a class="existingWikiWord" href="/nlab/show/information+theory">information theory</a> and <a class="existingWikiWord" href="/nlab/show/statistical+physics">statistical physics</a>.</p> <h2 id="mathematical_definitions">Mathematical definitions</h2> <p>We can give a precise <a class="existingWikiWord" href="/nlab/show/mathematics">mathematical</a> definition of the entropy in <a class="existingWikiWord" href="/nlab/show/probability+theory">probability theory</a>.</p> <h3 id="preliminary_definitions">Preliminary definitions</h3> <p>We will want a couple of preliminary definitions. Fix a <a class="existingWikiWord" href="/nlab/show/probability+space">probability space</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>μ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(X,\mu)</annotation></semantics></math>; that is, <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> is a <a class="existingWikiWord" href="/nlab/show/set">set</a>, and <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math> is a <a class="existingWikiWord" href="/nlab/show/probability+measure">probability measure</a> on <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math>.</p> <h4 id="surprisal">Surprisal</h4> <p>If <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math> is a <a class="existingWikiWord" href="/nlab/show/measurable+subset">measurable subset</a> of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math>, then the <strong>surprisal</strong> or <strong>self-<a class="existingWikiWord" href="/nlab/show/information">information</a></strong> of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math> (with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math>) is</p> <div class="maruku-equation"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>σ</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>≔</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mi>log</mi><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mspace width="thinmathspace"></mspace><mo>.</mo></mrow><annotation encoding="application/x-tex"> \sigma_\mu(A) \coloneqq -\log \mu(A) \,. </annotation></semantics></math></div> <p>Notice that, despite the minus sign in this formula, <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math> is a nonnegative function (since <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>log</mi><mi>p</mi><mo>≤</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\log p \leq 0</annotation></semantics></math> for <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>p</mi><mo>≤</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">p \leq 1</annotation></semantics></math>); more precisely, <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math> takes values in <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo stretchy="false">[</mo><mn>0</mn><mo>,</mo><mn>∞</mn><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">[0,\infty]</annotation></semantics></math>. The term ‘surprisal’ is intended to suggest how surprised one ought to be upon learning that the event modelled by <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math> is true: from no surprise for an event with probability <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>1</mn></mrow><annotation encoding="application/x-tex">1</annotation></semantics></math> to infinite surprise for an event with probability <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>0</mn></mrow><annotation encoding="application/x-tex">0</annotation></semantics></math>.</p> <p>The <strong>expected surprisal</strong> of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math> is then</p> <div class="maruku-equation" id="eq:EqA"><span class="maruku-eq-number">(1)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>≔</mo><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><msub><mi>σ</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mi>log</mi><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mi>log</mi><mo stretchy="false">(</mo><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><msup><mo stretchy="false">)</mo> <mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo></mrow></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex"> h_\mu(A) \coloneqq \mu(A) \sigma_\mu(A) = -\mu(A) \log \mu(A) = -\log(\mu(A)^{\mu(A)}) </annotation></semantics></math></div> <p>(with <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">h_\mu(A) = 0</annotation></semantics></math> when <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\mu(A) = 0</annotation></semantics></math>). Like <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>, <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math> is a nonnegative function; it is also important that <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msub><mi>h</mi> <mi>μ</mi></msub></mrow><annotation encoding="application/x-tex">h_\mu</annotation></semantics></math> is <a class="existingWikiWord" href="/nlab/show/convex+function">concave</a>. Both <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mo lspace="0em" rspace="thinmathspace">nothing</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">h_\mu(\nothing)</annotation></semantics></math> and <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>X</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">h_\mu(X)</annotation></semantics></math> are <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>0</mn></mrow><annotation encoding="application/x-tex">0</annotation></semantics></math>, but for different reasons; <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">h_\mu(A) = 0</annotation></semantics></math> when <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\mu(A) = 1</annotation></semantics></math> because, upon observing an event with probability <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>1</mn></mrow><annotation encoding="application/x-tex">1</annotation></semantics></math>, one gains no information; while <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">h_\mu(A) = 0</annotation></semantics></math> when <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\mu(A) = 0</annotation></semantics></math> because one expects never to observe an event with probability <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>0</mn></mrow><annotation encoding="application/x-tex">0</annotation></semantics></math>. The maximum possible value of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>h</mi></mrow><annotation encoding="application/x-tex">h</annotation></semantics></math> is <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi mathvariant="normal">e</mi> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup><mi>log</mi><mspace width="thinmathspace"></mspace><mi mathvariant="normal">e</mi></mrow><annotation encoding="application/x-tex">\mathrm{e}^{-1} \log \,\mathrm{e}</annotation></semantics></math> (so <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi mathvariant="normal">e</mi> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup></mrow><annotation encoding="application/x-tex">\mathrm{e}^{-1}</annotation></semantics></math> if we use <a class="existingWikiWord" href="/nlab/show/natural+logarithms">natural logarithms</a>), which occurs when <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><msup><mi mathvariant="normal">e</mi> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup></mrow><annotation encoding="application/x-tex">\mu(A) = \mathrm{e}^{-1}</annotation></semantics></math>.</p> <p>We have not specified the base of the <a class="existingWikiWord" href="/nlab/show/logarithm">logarithm</a>, which amounts to a constant factor (proportional to the logarithm of the base), which we think of as specifying the <a class="existingWikiWord" href="/nlab/show/unit+of+measurement">unit of measurement</a> of entropy. Common choices for the base are <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>2</mn></mrow><annotation encoding="application/x-tex">2</annotation></semantics></math> (whose unit is the <a class="existingWikiWord" href="/nlab/show/bit">bit</a>, originally a unit of memory in computer science), <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>256</mn></mrow><annotation encoding="application/x-tex">256</annotation></semantics></math> (for the byte, which is <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>8</mn></mrow><annotation encoding="application/x-tex">8</annotation></semantics></math> bits), <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>3</mn></mrow><annotation encoding="application/x-tex">3</annotation></semantics></math> (for the trit), <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi mathvariant="normal">e</mi></mrow><annotation encoding="application/x-tex">\mathrm{e}</annotation></semantics></math> (for the nat or neper), <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>10</mn></mrow><annotation encoding="application/x-tex">10</annotation></semantics></math> (for the bel, originally a unit of relative power intensity in telegraphy, or ban, dit, or hartley), and <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mroot><mn>10</mn><mn>10</mn></mroot></mrow><annotation encoding="application/x-tex">\root{10}{10}</annotation></semantics></math> (for the decibel, <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>1</mn><mo stretchy="false">/</mo><mn>10</mn></mrow><annotation encoding="application/x-tex">1/10</annotation></semantics></math> of a bel). In applications to <a class="existingWikiWord" href="/nlab/show/statistical+physics">statistical physics</a>, common bases are exactly (since 2019)</p> <div class="maruku-equation" id="eq:EqB"><span class="maruku-eq-number">(2)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msup><mi mathvariant="normal">e</mi> <mrow><msup><mn>10</mn> <mn>29</mn></msup><mo stretchy="false">/</mo><mn>1</mn><mspace width="thinmathspace"></mspace><mn>380</mn><mspace width="thinmathspace"></mspace><mn>649</mn></mrow></msup><mo>≈</mo><msup><mn>10</mn> <mrow><mn>3.145</mn><mspace width="thinmathspace"></mspace><mn>582</mn><mspace width="thinmathspace"></mspace><mn>127</mn><mspace width="thinmathspace"></mspace><mn>704</mn><mspace width="thinmathspace"></mspace><mn>085</mn><mspace width="thinmathspace"></mspace><mn>743</mn><mo>×</mo><msup><mn>10</mn> <mn>22</mn></msup></mrow></msup><mspace width="2em"></mspace><mtext>(for the joule per kelvin)</mtext><mo>,</mo></mrow><annotation encoding="application/x-tex"> \mathrm{e}^{10^{29}/1\,380\,649} \approx 10^{3.145\,582\,127\,704\,085\,743 \times 10^{22}} \qquad \text{(for the joule per kelvin)}, </annotation></semantics></math></div> <p>or</p> <div class="maruku-equation" id="eq:EqC"><span class="maruku-eq-number">(3)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msup><mi mathvariant="normal">e</mi> <mrow><mn>104</mn><mspace width="thinmathspace"></mspace><mn>600</mn><mspace width="thinmathspace"></mspace><mn>000</mn><mspace width="thinmathspace"></mspace><mn>000</mn><mspace width="thinmathspace"></mspace><mn>000</mn><mo stretchy="false">/</mo><mn>207</mn><mspace width="thinmathspace"></mspace><mn>861</mn><mspace width="thinmathspace"></mspace><mn>565</mn><mspace width="thinmathspace"></mspace><mn>453</mn><mspace width="thinmathspace"></mspace><mn>831</mn></mrow></msup><mo>≈</mo><mn>1.654</mn><mspace width="thinmathspace"></mspace><mn>037</mn><mspace width="thinmathspace"></mspace><mn>938</mn><mspace width="thinmathspace"></mspace><mn>063</mn><mspace width="thinmathspace"></mspace><mn>167</mn><mspace width="thinmathspace"></mspace><mn>336</mn><mspace width="2em"></mspace><mtext>(for the calorie per mole-kelvin)</mtext><mo>,</mo></mrow><annotation encoding="application/x-tex"> \mathrm{e}^{104\,600\,000\,000\,000/207\,861\,565\,453\,831} \approx 1.654\,037\,938\,063\,167\,336 \qquad \text{(for the calorie per mole-kelvin)}, </annotation></semantics></math></div> <p>and so on; although <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi mathvariant="normal">e</mi></mrow><annotation encoding="application/x-tex">\mathrm{e}</annotation></semantics></math> is common in theoretical work (and then the unit of measurement is said to be <a class="existingWikiWord" href="/nlab/show/Boltzmann%27s+constant">Boltzmann's constant</a> rather than the nat or neper).</p> <h4 id="almost_partitions">Almost partitions</h4> <p>Recall that a <strong><a class="existingWikiWord" href="/nlab/show/partition">partition</a></strong> of a set <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> is a <a class="existingWikiWord" href="/nlab/show/family+of+subsets">family</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math> of subsets of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> (the <em>parts</em> of the partition) such that <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> is the <a class="existingWikiWord" href="/nlab/show/union">union</a> of the parts and any two distinct parts are <a class="existingWikiWord" href="/nlab/show/disjoint+sets">disjoint</a> (or better, for <a class="existingWikiWord" href="/nlab/show/constructive+mathematics">constructive mathematics</a>, two parts are equal if their intersection is <a class="existingWikiWord" href="/nlab/show/inhabited+subset">inhabited</a>).</p> <p>When <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> is a probability space, we may relax both conditions: for the union of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math>, we require only that it be a <a class="existingWikiWord" href="/nlab/show/full+set">full set</a>; for the intersections of pairs of elements of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math>, we require only that they be <a class="existingWikiWord" href="/nlab/show/null+sets">null sets</a> (or better, for constructive mathematics, that <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>A</mi><mo>=</mo><mi>B</mi></mrow><annotation encoding="application/x-tex">A = B</annotation></semantics></math> when <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>μ</mi> <mo>*</mo></msup><mo stretchy="false">(</mo><mi>A</mi><mo>∩</mo><mi>B</mi><mo stretchy="false">)</mo><mo>></mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\mu^*(A \cap B) \gt 0</annotation></semantics></math>, where <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>μ</mi> <mo>*</mo></msup></mrow><annotation encoding="application/x-tex">\mu^*</annotation></semantics></math> is the <span class="newWikiWord">outer measure<a href="/nlab/new/outer+measure">?</a></span> corresponding to <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math>).</p> <p>For definiteness, call such a collection of subsets a <strong><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math>-almost partition</strong>; a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math>-almost partition is <em>measurable</em> if each of its part is measurable (in which case we can use <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math> instead of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>μ</mi> <mo>*</mo></msup></mrow><annotation encoding="application/x-tex">\mu^*</annotation></semantics></math>).</p> <h3 id="entropy_of_a_algebra_on_a_probability_space">Entropy of a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra on a probability space</h3> <p>This is a general mathematical definition of entropy.</p> <p>Given a probability <a class="existingWikiWord" href="/nlab/show/measure+space">measure space</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>μ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(X,\mu)</annotation></semantics></math> and a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-<a class="existingWikiWord" href="/nlab/show/sigma-algebra">algebra</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>ℳ</mi></mrow><annotation encoding="application/x-tex">\mathcal{M}</annotation></semantics></math> of <a class="existingWikiWord" href="/nlab/show/measurable+sets">measurable sets</a> in <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math>, the <strong>entropy</strong> of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>ℳ</mi></mrow><annotation encoding="application/x-tex">\mathcal{M}</annotation></semantics></math> with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math> is</p> <div class="maruku-equation" id="eq:general"><span class="maruku-eq-number">(4)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>H</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>ℳ</mi><mo stretchy="false">)</mo><mo>≔</mo><mi>sup</mi><mo stretchy="false">{</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>A</mi><mo>∈</mo><mi>ℱ</mi></mrow></munder><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mspace width="thickmathspace"></mspace><mo stretchy="false">|</mo><mspace width="thickmathspace"></mspace><mi>ℱ</mi><mo>⊆</mo><mi>ℳ</mi><mo>,</mo><mspace width="thickmathspace"></mspace><mrow><mo stretchy="false">|</mo><mi>ℱ</mi><mo stretchy="false">|</mo></mrow><mo><</mo><msub><mi>ℵ</mi> <mn>0</mn></msub><mo>,</mo><mspace width="thickmathspace"></mspace><mi>X</mi><mo>=</mo><mo lspace="thinmathspace" rspace="thinmathspace">⨄</mo><mi>ℱ</mi><mo stretchy="false">}</mo><mspace width="thinmathspace"></mspace><mo>.</mo></mrow><annotation encoding="application/x-tex"> H_\mu(\mathcal{M}) \coloneqq \sup \{ \sum_{A \in \mathcal{F}} h_\mu(A) \;|\; \mathcal{F} \subseteq \mathcal{M},\; {|\mathcal{F}|} \lt \aleph_0,\; X = \biguplus \mathcal{F} \} \,. </annotation></semantics></math></div> <p>In words, the entropy is the <a class="existingWikiWord" href="/nlab/show/supremum">supremum</a>, over all ways of expressing <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> as an internal <a class="existingWikiWord" href="/nlab/show/disjoint+union">disjoint union</a> of <a class="existingWikiWord" href="/nlab/show/finite+set">finitely many</a> elements of the <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>ℳ</mi></mrow><annotation encoding="application/x-tex">\mathcal{M}</annotation></semantics></math>, of the sum, over these measurable sets, of the expected surprisals of these sets. This supremum can also be expressed as a <a class="existingWikiWord" href="/nlab/show/convergence">limit</a> as we take <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>ℱ</mi></mrow><annotation encoding="application/x-tex">\mathcal{F}</annotation></semantics></math> to be finer and finer, since <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msub><mi>h</mi> <mi>μ</mi></msub></mrow><annotation encoding="application/x-tex">h_\mu</annotation></semantics></math> is concave and the partitions are <a class="existingWikiWord" href="/nlab/show/directed+set">directed</a>.</p> <p>We have written this so that <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>ℱ</mi></mrow><annotation encoding="application/x-tex">\mathcal{F}</annotation></semantics></math> is a finite partition of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math>; without loss of generality, we may require only that <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>ℱ</mi></mrow><annotation encoding="application/x-tex">\mathcal{F}</annotation></semantics></math> be a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math>-almost partition. In <a class="existingWikiWord" href="/nlab/show/constructive+mathematics">constructive mathematics</a>, it seems that we <em>must</em> use this weakened condition, at least the part that allows <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo lspace="thinmathspace" rspace="thinmathspace">⋃</mo><mi>ℱ</mi></mrow><annotation encoding="application/x-tex">\bigcup \mathcal{F}</annotation></semantics></math> to merely be full.</p> <p>This definition is very general, and it is instructive to look at special cases.</p> <h3 id="entropy_of_a_probability_space">Entropy of a probability space</h3> <p>Given a probability space <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>μ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(X,\mu)</annotation></semantics></math>, the <strong>entropy</strong> of this probability space is the entropy, with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math>, of the <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra of <em>all</em> measurable subsets of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math>.</p> <h3 id="entropy_of_a_partition_of_a_probability_space">Entropy of a partition of a probability space</h3> <p>Every measurable almost-partition of a measure space (indeed, any family of measurable subsets) generates a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra. The <strong>entropy</strong> of a measurable almost-partition <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math> of a probability measure space <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>μ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(X,\mu)</annotation></semantics></math> is the entropy, with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math>, of the <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra generated by <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math>. The formula <a class="maruku-eqref" href="#eq:general">(4)</a> may then be written</p> <div class="maruku-equation" id="eq:partition"><span class="maruku-eq-number">(5)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>H</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>𝒫</mi><mo stretchy="false">)</mo><mo>=</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>A</mi><mo>∈</mo><mi>𝒫</mi></mrow></munder><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>A</mi><mo>∈</mo><mi>𝒫</mi></mrow></munder><mi>log</mi><mo stretchy="false">(</mo><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><msup><mo stretchy="false">)</mo> <mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo></mrow></msup><mo stretchy="false">)</mo><mo>,</mo></mrow><annotation encoding="application/x-tex"> H_\mu(\mathcal{P}) = \sum_{A \in \mathcal{P}} h_\mu(A) = -\sum_{A \in \mathcal{P}} \log(\mu(A)^{\mu(A)}) ,</annotation></semantics></math></div> <p>since an infinite sum (of nonnegative terms) may also be defined as a supremum. (Actually, the supremum in the infinite sum does not quite match the supremum in <a class="maruku-eqref" href="#eq:general">(4)</a>, so there is a bit of a theorem to prove here.)</p> <p>In most of the following special cases, we will consider only partitions, although it would be possible also to consider more general <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebras.</p> <h3 id="entropy_of_a_partition_of_a_discrete_probability_space">Entropy of (a partition of) a discrete probability space</h3> <p>Recall that a <strong>discrete probability space</strong> is a <a class="existingWikiWord" href="/nlab/show/set">set</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> equipped with a function <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo lspace="verythinmathspace">:</mo><mi>X</mi><mo>→</mo><mo stretchy="false">]</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\mu\colon X \to ]0,1]</annotation></semantics></math> such that <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msub><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>∈</mo><mi>X</mi></mrow></msub><mi>μ</mi><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\sum_{i \in X} \mu(i) = 1</annotation></semantics></math>; since <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo><mo>></mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\mu(i) \gt 0</annotation></semantics></math> is possible for only countably many <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math>, <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> must be <a class="existingWikiWord" href="/nlab/show/countable+set">countable</a>. We make <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> into a measure space (with every subset measurable) by defining <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>≔</mo><msub><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>∈</mo><mi>A</mi></mrow></msub><mi>μ</mi><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\mu(A) \coloneqq \sum_{i \in A} \mu(i)</annotation></semantics></math>. Since every inhabited set has positive measure, every almost-partition of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> is a partition; since every set is measurable, any partition is measurable.</p> <p>Given a discrete probability space <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>μ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(X,\mu)</annotation></semantics></math> and a partition <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math> of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math>, the <strong>entropy</strong> of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math> with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math> is defined to be the entropy of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math> with respect to the probability measure induced by <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math>. Simplifying <a class="maruku-eqref" href="#eq:partition">(5)</a>, we find</p> <div class="maruku-equation" id="eq:EqD"><span class="maruku-eq-number">(6)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>H</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>𝒫</mi><mo stretchy="false">)</mo><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>A</mi><mo>∈</mo><mi>𝒫</mi></mrow></munder><mi>log</mi><mo stretchy="false">(</mo><mo stretchy="false">(</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>∈</mo><mi>A</mi></mrow></munder><mi>μ</mi><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo><msup><mo stretchy="false">)</mo> <mrow><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>∈</mo><mi>A</mi></mrow></munder><mi>μ</mi><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msup><mo stretchy="false">)</mo><mspace width="thinmathspace"></mspace><mo>.</mo></mrow><annotation encoding="application/x-tex"> H_\mu(\mathcal{P}) = -\sum_{A \in \mathcal{P}} \log((\sum_{i \in A} \mu(i))^{\sum_{i \in A} \mu(i)}) \,. </annotation></semantics></math></div> <p>More specially, the <strong>entropy</strong> of the discrete probability space <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>μ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(X,\mu)</annotation></semantics></math> is the entropy of the partition of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> into <a class="existingWikiWord" href="/nlab/show/singletons">singletons</a>; we find</p> <div class="maruku-equation" id="eq:EqE"><span class="maruku-eq-number">(7)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>H</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>X</mi><mo stretchy="false">)</mo><mo>=</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>∈</mo><mi>X</mi></mrow></munder><msub><mi>h</mi> <mi>μ</mi></msub><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>∈</mo><mi>X</mi></mrow></munder><mi>log</mi><mo stretchy="false">(</mo><mi>μ</mi><mo stretchy="false">(</mo><mi>i</mi><msup><mo stretchy="false">)</mo> <mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msup><mo stretchy="false">)</mo><mspace width="thinmathspace"></mspace><mo>.</mo></mrow><annotation encoding="application/x-tex"> H_\mu(X) = \sum_{i \in X} h_\mu(i) = -\sum_{i \in X} \log(\mu(i)^{\mu(i)}) \,. </annotation></semantics></math></div> <p>This is actually a special case of the entropy of a probability space, since the <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra generated by the singletons is the power set of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math>.</p> <p>Yet more specially, the <strong>entropy</strong> of a <a class="existingWikiWord" href="/nlab/show/finite+set">finite set</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> is the entropy of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> equipped with the uniform discrete probability measure; we find</p> <div class="maruku-equation" id="eq:Boltzmann"><span class="maruku-eq-number">(8)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>H</mi> <mi>unif</mi></msub><mo stretchy="false">(</mo><mi>X</mi><mo stretchy="false">)</mo><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>∈</mo><mi>X</mi></mrow></munder><mi>log</mi><mo stretchy="false">(</mo><mo stretchy="false">(</mo><mfrac><mn>1</mn><mrow><mo stretchy="false">|</mo><mi>X</mi><mo stretchy="false">|</mo></mrow></mfrac><msup><mo stretchy="false">)</mo> <mfrac><mn>1</mn><mrow><mo stretchy="false">|</mo><mi>X</mi><mo stretchy="false">|</mo></mrow></mfrac></msup><mo stretchy="false">)</mo><mo>=</mo><mi>log</mi><mrow><mo stretchy="false">|</mo><mi>X</mi><mo stretchy="false">|</mo></mrow><mo>,</mo></mrow><annotation encoding="application/x-tex"> H_{unif}(X) = -\sum_{i \in X} \log((\frac{1}{|X|})^{\frac{1}{|X|}}) = \log {|X|} ,</annotation></semantics></math></div> <p>which is probably the best known mathematical formula for entropy, due to <a class="existingWikiWord" href="/nlab/show/Max+Planck">Max Planck</a>, who attributed it to <a class="existingWikiWord" href="/nlab/show/Ludwig+Boltzmann">Ludwig Boltzmann</a>. (Its <a href="#physical">physical interpretation</a> appears below.)</p> <p>Of all probability measures on <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math>, the uniform measure has the <a class="existingWikiWord" href="/nlab/show/maximum+entropy">maximum entropy</a>.</p> <h3 id="entropy_with_respect_to_an_absolutely_continuous_probability_measure_on_the_real_line">Entropy with respect to an absolutely continuous probability measure on the real line</h3> <p>Recall that a <a class="existingWikiWord" href="/nlab/show/Borel+measure">Borel measure</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math> on an <a class="existingWikiWord" href="/nlab/show/interval">interval</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> in the <a class="existingWikiWord" href="/nlab/show/real+line">real line</a> is <strong><a class="existingWikiWord" href="/nlab/show/absolutely+continuous+measure">absolutely continuous</a></strong> if <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\mu(A) = 0</annotation></semantics></math> whenever <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math> is a <a class="existingWikiWord" href="/nlab/show/null+set">null set</a> (with respect to <a class="existingWikiWord" href="/nlab/show/Lebesgue+measure">Lebesgue measure</a>), or better such that <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>></mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\mu(A) \gt 0</annotation></semantics></math> whenever the Lebesgue measure of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>A</mi></mrow><annotation encoding="application/x-tex">A</annotation></semantics></math> is positive. In this case, we can take the <a class="existingWikiWord" href="/nlab/show/Radon%E2%80%93Nikodym+derivative">Radon–Nikodym derivative</a> of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math> with respect to Lebesgue measure, to get an <a class="existingWikiWord" href="/nlab/show/integrable+function">integrable function</a> <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math>, called the <strong>probability distribution function</strong>; we recover <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math> by</p> <div class="maruku-equation" id="eq:pdf"><span class="maruku-eq-number">(9)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><mi>μ</mi><mo stretchy="false">(</mo><mi>A</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mo>∫</mo> <mi>A</mi></msub><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi mathvariant="normal">d</mi><mi>x</mi><mo>,</mo></mrow><annotation encoding="application/x-tex"> \mu(A) = \int_A f(x) \mathrm{d}x ,</annotation></semantics></math></div> <p>where the integral is taken with respect to Lebesgue measure.</p> <p>If <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math> is a partition (or a Lebesgue-almost-partition) of an interval <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math> into <a class="existingWikiWord" href="/nlab/show/Borel+sets">Borel sets</a>, then the <strong>entropy</strong> of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math> with respect to an integrable function <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math> is the entropy of <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>𝒫</mi></mrow><annotation encoding="application/x-tex">\mathcal{P}</annotation></semantics></math> with respect to the measure induced by <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math> using the integral formula <a class="maruku-eqref" href="#eq:pdf">(9)</a>; we find</p> <div class="maruku-equation" id="eq:EqF"><span class="maruku-eq-number">(10)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>H</mi> <mi>f</mi></msub><mo stretchy="false">(</mo><mi>𝒫</mi><mo stretchy="false">)</mo><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><munder><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>A</mi><mo>∈</mo><mi>𝒫</mi></mrow></munder><mi>log</mi><mo maxsize="1.2em" minsize="1.2em">(</mo><mo stretchy="false">(</mo><msub><mo>∫</mo> <mi>A</mi></msub><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi mathvariant="normal">d</mi><mi>x</mi><msup><mo stretchy="false">)</mo> <mrow><msub><mo>∫</mo> <mi>A</mi></msub><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi mathvariant="normal">d</mi><mi>x</mi></mrow></msup><mo maxsize="1.2em" minsize="1.2em">)</mo><mspace width="thinmathspace"></mspace><mo>.</mo></mrow><annotation encoding="application/x-tex"> H_f(\mathcal{P}) = -\sum_{A \in \mathcal{P}} \log\big( (\int_A f(x) \mathrm{d}x)^{\int_A f(x) \mathrm{d}x} \big) \,. </annotation></semantics></math></div> <p>On the other hand, the <strong>entropy</strong> of the probability distribution space <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo stretchy="false">(</mo><mi>X</mi><mo>,</mo><mi>f</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">(X,f)</annotation></semantics></math> is the entropy of the entire <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra of all Borel sets (which is <em>not</em> generated by a partition) with respect to <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math>; we find</p> <div class="maruku-equation" id="eq:EqG"><span class="maruku-eq-number">(11)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>H</mi> <mi>f</mi></msub><mo stretchy="false">(</mo><mi>X</mi><mo stretchy="false">)</mo><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><msub><mo>∫</mo> <mrow><mi>x</mi><mo>∈</mo><mi>X</mi></mrow></msub><mi>log</mi><mo stretchy="false">(</mo><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><msup><mo stretchy="false">)</mo> <mrow><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow></msup><mo stretchy="false">)</mo><mi mathvariant="normal">d</mi><mi>x</mi></mrow><annotation encoding="application/x-tex"> H_f(X) = -\int_{x \in X} \log(f(x)^{f(x)}) \mathrm{d}x </annotation></semantics></math></div> <p>by a fairly complicated argument.</p> <div class="query"> <p>I haven't actually managed to check this argument yet, although my memory tags it as a true fact. —Toby</p> </div> <h3 id="entropy_of_a_density_matrix">Entropy of a density matrix</h3> <p>In the analogy between <a class="existingWikiWord" href="/nlab/show/classical+physics">classical physics</a> and <a class="existingWikiWord" href="/nlab/show/quantum+physics">quantum physics</a>, we move from probability distributions on a <a class="existingWikiWord" href="/nlab/show/phase+space">phase space</a> to <a class="existingWikiWord" href="/nlab/show/density+operators">density operators</a> on a <a class="existingWikiWord" href="/nlab/show/Hilbert+space">Hilbert space</a>.</p> <p>So just as the entropy of a probability distribution <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math> is given by <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mo>∫</mo><mi>f</mi><mi>log</mi><mi>f</mi></mrow><annotation encoding="application/x-tex">- \int f \log f</annotation></semantics></math>, so the <strong>entropy</strong> of a density operator <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>ρ</mi></mrow><annotation encoding="application/x-tex">\rho</annotation></semantics></math> is</p> <div class="maruku-equation" id="eq:EqH"><span class="maruku-eq-number">(12)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><msub><mi>H</mi> <mi>ρ</mi></msub><mo>≔</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mi>Tr</mi><mo stretchy="false">(</mo><mi>ρ</mi><mi>log</mi><mi>ρ</mi><mo stretchy="false">)</mo><mspace width="thinmathspace"></mspace><mo>.</mo></mrow><annotation encoding="application/x-tex"> H_\rho \coloneqq -Tr (\rho \log \rho) \,. </annotation></semantics></math></div> <p>using the <a class="existingWikiWord" href="/nlab/show/functional+calculus">functional calculus</a>.</p> <p>These are both special cases of the entropy of a <a class="existingWikiWord" href="/nlab/show/state">state</a> on a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>C</mi> <mo>*</mo></msup></mrow><annotation encoding="application/x-tex">C^*</annotation></semantics></math>-<a class="existingWikiWord" href="/nlab/show/C-star-algebra">algebra</a>.</p> <p>There is a way to fit this into the framework given by <a class="maruku-eqref" href="#eq:general">(4)</a>, but I don't remember it (and never really understood it).</p> <h3 id="RelativeEntropy">Relative entropy</h3> <p>For two finite probability distributions <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math> and <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>q</mi></mrow><annotation encoding="application/x-tex">q</annotation></semantics></math>, their <strong>relative entropy</strong> is</p> <div class="maruku-equation" id="eq:EqJ"><span class="maruku-eq-number">(13)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><mi>S</mi><mo stretchy="false">(</mo><mi>p</mi><mo stretchy="false">/</mo><mi>q</mi><mo stretchy="false">)</mo><mo>≔</mo><munderover><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow> <mi>n</mi></munderover><msub><mi>p</mi> <mi>k</mi></msub><mo stretchy="false">(</mo><mi>log</mi><msub><mi>p</mi> <mi>k</mi></msub><mo>−</mo><mi>log</mi><msub><mi>q</mi> <mi>k</mi></msub><mo stretchy="false">)</mo><mspace width="thinmathspace"></mspace><mo>.</mo></mrow><annotation encoding="application/x-tex"> S(p/q) \coloneqq \sum_{k = 1}^n p_k(log p_k - log q_k) \,. </annotation></semantics></math></div> <p>Or alternatively, for <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>ρ</mi><mo>,</mo><mi>ϕ</mi></mrow><annotation encoding="application/x-tex">\rho, \phi</annotation></semantics></math> two <a class="existingWikiWord" href="/nlab/show/density+matrix">density matrices</a>, their relative entropy is</p> <div class="maruku-equation" id="eq:EqK"><span class="maruku-eq-number">(14)</span><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" class="maruku-mathml"><semantics><mrow><mi>S</mi><mo stretchy="false">(</mo><mi>ρ</mi><mo stretchy="false">/</mo><mi>ϕ</mi><mo stretchy="false">)</mo><mo>≔</mo><mi>tr</mi><mi>ρ</mi><mo stretchy="false">(</mo><mi>log</mi><mi>ρ</mi><mo>−</mo><mi>log</mi><mi>ϕ</mi><mo stretchy="false">)</mo><mspace width="thinmathspace"></mspace><mo>.</mo></mrow><annotation encoding="application/x-tex"> S(\rho/\phi) \coloneqq tr \rho(log \rho - log \phi) \,. </annotation></semantics></math></div> <p>There is a generalization of these definitions to <a class="existingWikiWord" href="/nlab/show/state">state</a>s on general <a class="existingWikiWord" href="/nlab/show/von+Neumann+algebra">von Neumann algebra</a>s, due to (<a href="#Araki">Araki</a>).</p> <p>For more on this see <em><a class="existingWikiWord" href="/nlab/show/relative+entropy">relative entropy</a></em>.</p> <h2 id="physical">Physical entropy</h2> <p>As hinted above, any probability distribution on a <a class="existingWikiWord" href="/nlab/show/phase+space">phase space</a> in <a class="existingWikiWord" href="/nlab/show/classical+physics">classical physics</a> has an entropy, and any <a class="existingWikiWord" href="/nlab/show/density+matrix">density matrix</a> on a <a class="existingWikiWord" href="/nlab/show/Hilbert+space">Hilbert space</a> in <a class="existingWikiWord" href="/nlab/show/quantum+physics">quantum physics</a> has an entropy. However, these are <strong>microscopic entropy</strong>, which is not the usual entropy in <a class="existingWikiWord" href="/nlab/show/thermodynamics">thermodynamics</a> and most other branches of <a class="existingWikiWord" href="/nlab/show/physics">physics</a>. (In particular, microscopic entropy is conserved, rather than increasing with time.)</p> <p>Instead, physicists use <em>coarse-grained</em> entropy, which corresponds mathematically to taking the entropy of a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra much smaller than the <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra of all measurable sets. Given a classical system with <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math> microscopic degrees of freedom, we identify <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math> macroscopic degrees of freedom that we can reasonably expect to measure, giving a map from <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>ℝ</mi> <mi>N</mi></msup></mrow><annotation encoding="application/x-tex">\mathbb{R}^N</annotation></semantics></math> to <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>ℝ</mi> <mi>n</mi></msup></mrow><annotation encoding="application/x-tex">\mathbb{R}^n</annotation></semantics></math> (or more generally, a map from an <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math>-dimensional microscopic phase space to an <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math>-dimensional macroscopic phase space). Then the <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra of all measurable sets in <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>ℝ</mi> <mi>n</mi></msup></mrow><annotation encoding="application/x-tex">\mathbb{R}^n</annotation></semantics></math> <a class="existingWikiWord" href="/nlab/show/pullback">pulls back</a> to a <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra on <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>ℝ</mi> <mi>N</mi></msup></mrow><annotation encoding="application/x-tex">\mathbb{R}^N</annotation></semantics></math>, and the <strong>macroscopic entropy</strong> of a statistical state is the <span class="newWikiWord">conditional entropy<a href="/nlab/new/conditional+entropy">?</a></span> of this <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math>-algebra. (Typically, <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math> is on the order of <a class="existingWikiWord" href="/nlab/show/Avogadro%27s+number">Avogadro's number</a>, while <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mi>n</mi></mrow><annotation encoding="application/x-tex">n</annotation></semantics></math> is rarely more than half a dozen, and often as small as <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mn>2</mn></mrow><annotation encoding="application/x-tex">2</annotation></semantics></math>.)</p> <p>If we specify a state by a point in <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>ℝ</mi> <mi>n</mi></msup></mrow><annotation encoding="application/x-tex">\mathbb{R}^n</annotation></semantics></math>, a macroscopic pure state, and assume a uniform probability distribution on its <a class="existingWikiWord" href="/nlab/show/fibre">fibre</a> in <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>ℝ</mi> <mi>N</mi></msup></mrow><annotation encoding="application/x-tex">\mathbb{R}^N</annotation></semantics></math>, then this results in the <a class="existingWikiWord" href="/nlab/show/maximum+entropy">maximum entropy</a>. If this fibre were a finite set, then we would recover Boltzmann's formula <a class="maruku-eqref" href="#eq:Boltzmann">(8)</a>. This is never exactly true in classical statistical physics, but it is often nevertheless a very good approximation. (Boltzmann's formula actually makes better physical sense in quantum statistical physics, even though Boltzmann himself did not live to see this.)</p> <p>A more sophisticated approach (pioneered by <span class="newWikiWord">Josiah Gibbs<a href="/nlab/new/Josiah+Gibbs">?</a></span>) is to consider all possible mixed microstates (that is all possible probability distributions on the space <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>ℝ</mi> <mi>N</mi></msup></mrow><annotation encoding="application/x-tex">\mathbb{R}^N</annotation></semantics></math> of pure microstates) whose <a class="existingWikiWord" href="/nlab/show/expectation+values">expectation values</a> of total energy and other <a class="existingWikiWord" href="/nlab/show/extensive+quantities">extensive quantities</a> (among those that are functions of the macrostate) match the given pure macrostate (point in <math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><msup><mi>ℝ</mi> <mi>n</mi></msup></mrow><annotation encoding="application/x-tex">\mathbb{R}^n</annotation></semantics></math>). We pick the mixed microstate with the <a class="existingWikiWord" href="/nlab/show/maximum+entropy">maximum entropy</a>. If this is a <span class="newWikiWord">thermal state<a href="/nlab/new/thermal+state">?</a></span>, then we say that the macrostate has a <a class="existingWikiWord" href="/nlab/show/temperature">temperature</a>, but it has an entropy in any case.</p> <h3 id="gravitational_entropy">Gravitational entropy</h3> <ul> <li> <p>gravitational entropy</p> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/Bekenstein-Hawking+entropy">Bekenstein-Hawking entropy</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/generalized+second+law+of+thermodynamics">generalized second law of thermodynamics</a></p> </li> </ul> </li> </ul> <h2 id="related_concepts">Related concepts</h2> <div> <table><thead><tr><th>order</th><th></th><th><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mphantom><mo>→</mo></mphantom><mn>0</mn></mrow><annotation encoding="application/x-tex">\phantom{\to} 0</annotation></semantics></math></th><th></th><th><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo>→</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\to 1</annotation></semantics></math></th><th></th><th><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mphantom><mo>→</mo></mphantom><mn>2</mn></mrow><annotation encoding="application/x-tex">\phantom{\to}2</annotation></semantics></math></th><th></th><th><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo>→</mo><mn>∞</mn></mrow><annotation encoding="application/x-tex">\to \infty</annotation></semantics></math></th></tr></thead><tbody><tr><td style="text-align: left;"><strong><a class="existingWikiWord" href="/nlab/show/R%C3%A9nyi+entropy">Rényi entropy</a></strong></td><td style="text-align: left;"></td><td style="text-align: left;"><a class="existingWikiWord" href="/nlab/show/Hartley+entropy">Hartley entropy</a></td><td style="text-align: left;"><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo>≥</mo></mrow><annotation encoding="application/x-tex">\geq</annotation></semantics></math></td><td style="text-align: left;"><a class="existingWikiWord" href="/nlab/show/Shannon+entropy">Shannon entropy</a></td><td style="text-align: left;"><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo>≥</mo></mrow><annotation encoding="application/x-tex">\geq</annotation></semantics></math></td><td style="text-align: left;"><a class="existingWikiWord" href="/nlab/show/collision+entropy">collision entropy</a></td><td style="text-align: left;"><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline" class="maruku-mathml"><semantics><mrow><mo>≥</mo></mrow><annotation encoding="application/x-tex">\geq</annotation></semantics></math></td><td style="text-align: left;"><a class="existingWikiWord" href="/nlab/show/min-entropy">min-entropy</a></td></tr> </tbody></table> </div> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/relative+entropy">relative entropy</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/entanglement+entropy">entanglement entropy</a>, <a class="existingWikiWord" href="/nlab/show/holographic+entanglement+entropy">holographic entanglement entropy</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/entropic+force">entropic force</a></p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/dissipative+system">dissipative system</a></p> </li> </ul> <h2 id="References">References</h2> <h3 id="ReferencesGeneral">General</h3> <p>The concept of entropy was introduced, by <a class="existingWikiWord" href="/nlab/show/Rudolf+Clausius">Rudolf Clausius</a> in 1865, in the context of <a class="existingWikiWord" href="/nlab/show/physics">physics</a>, and then adapted to <a class="existingWikiWord" href="/nlab/show/information+theory">information theory</a> by <a class="existingWikiWord" href="/nlab/show/Claude+Shannon">Claude Shannon</a> in 1948, to <a class="existingWikiWord" href="/nlab/show/quantum+mechanics">quantum mechanics</a> by <a class="existingWikiWord" href="/nlab/show/John+von+Neumann">John von Neumann</a> in 1955, to <a class="existingWikiWord" href="/nlab/show/ergodic+theory">ergodic theory</a> by <a class="existingWikiWord" href="/nlab/show/Andrey+Kolmogorov">Andrey Kolmogorov</a> and Sinai in 1958, and to <a class="existingWikiWord" href="/nlab/show/topological+dynamics">topological dynamics</a> by Adler, Konheim and McAndrew in 1965.</p> <p>A survey at the introductory level:</p> <ul> <li><a class="existingWikiWord" href="/nlab/show/John+Baez">John Baez</a>. <em>What is Entropy?</em> (2024). (<a href="https://arxiv.org/abs/2409.09232">arXiv:2409.09232</a>).</li> </ul> <p>Survey with an eye towards <a class="existingWikiWord" href="/nlab/show/black+hole+entropy">black hole entropy</a>:</p> <ul> <li><a class="existingWikiWord" href="/nlab/show/Ted+Jacobson">Ted Jacobson</a>, <em>Entropy from Carnot to Bekenstein</em> (<a href="https://arxiv.org/abs/1810.07839">arXiv:1810.07839</a>)</li> </ul> <h3 id="ReferencesInQuantumProbabilityTheory">In quantum probability theory</h3> <p>Discussion of entropy in <a class="existingWikiWord" href="/nlab/show/quantum+probability+theory">quantum probability theory</a>, hence for <a class="existingWikiWord" href="/nlab/show/quantum+states">quantum states</a> understood as positive linear functionals on the <a class="existingWikiWord" href="/nlab/show/star-algebra">star-algebra</a> <a class="existingWikiWord" href="/nlab/show/algebra+of+observables">of observables</a> (<a class="existingWikiWord" href="/nlab/show/operator+algebra">operator algebraic</a> <a class="existingWikiWord" href="/nlab/show/state+on+a+star-algebra">states on star-algebras</a>) and with their <a class="existingWikiWord" href="/nlab/show/density+matrices">density matrices</a> defined via the <a class="existingWikiWord" href="/nlab/show/GNS+construction">GNS construction</a>:</p> <p>Introduction and survey:</p> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/Ingemar+Bengtsson">Ingemar Bengtsson</a>, <a class="existingWikiWord" href="/nlab/show/Karol+%C5%BByczkowski">Karol Życzkowski</a>, Section 12.2 of: <em>Geometry of Quantum States — An Introduction to Quantum Entanglement</em>, Cambridge University Press (2006) [<a href="https://doi.org/10.1017/CBO9780511535048">doi:10.1017/CBO9780511535048</a>]</p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/A.+P.+Balachandran">A. P. Balachandran</a>, T. R. Govindarajan, Amilcar R. de Queiroz, A. F. Reyes-Lega, <em>Algebraic approach to entanglement and entropy</em>, Phys. Rev. A 88, 022301 (2013) (<a href="http://arxiv.org/abs/1301.1300">arXiv:1301.1300</a>)</p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/A.+P.+Balachandran">A. P. Balachandran</a>, A. R. de Queiroz, S. Vaidya, <em>Entropy of Quantum States: Ambiguities</em>, Eur. Phys. J. Plus 128, 112 (2013) (<a href="https://arxiv.org/abs/1212.1239">arXiv:1212.1239</a>, <a href="https://doi.org/10.1140/epjp/i2013-13112-3">doi:10.1140/epjp/i2013-13112-3</a>)</p> </li> <li> <p>Paolo Facchi, Giovanni Gramegna, Arturo Konderak, <em>Entropy of quantum states</em> (<a href="https://arxiv.org/abs/2104.12611">arXiv:2104.12611</a>)</p> </li> <li> <p>Erling Størmer, <em>Entropy in operator algebras</em>, Astérisque, no. 232 (1995), p. 211-230. (<a href="http://www.numdam.org/item/AST_1995__232__211_0/">web</a>, <a href="http://www.numdam.org/article/AST_1995__232__211_0.pdf">pdf</a>)</p> </li> </ul> <p>In quantum mechanics, the basic notion is the von Neumann entropy defined in terms of <a class="existingWikiWord" href="/nlab/show/density+matrix">density matrix</a>. For type III <a class="existingWikiWord" href="/nlab/show/von+Neumann+algebra">von Neumann algebra</a>s the density matrix is not well defined (physically, the problem is usually in ultraviolet divergences). Von Neumann entropy is generalized to arbitrary semifinite von Neumann algebra in</p> <ul> <li><a class="existingWikiWord" href="/nlab/show/Irving+Segal">I. E. Segal</a>, <em>A note on the concept of entropy</em>, J. Math. Mech. 9 (1960) 623–629</li> </ul> <p><a href="#RelativeEntropy">Relative entropy</a> of <a class="existingWikiWord" href="/nlab/show/states">states</a> on <a class="existingWikiWord" href="/nlab/show/von+Neumann+algebras">von Neumann algebras</a> was introduced in</p> <ul> <li id="Araki"><a class="existingWikiWord" href="/nlab/show/Huzihiro+Araki">Huzihiro Araki</a>, <em>Relative entropy of states of von Neumann algebras</em>, Publ. RIMS Kyoto Univ. <p>11 (1976) 809–833 (<a href="https://ems.press/content/serial-article-files/2833">pdf</a>)</p> </li> </ul> <p>A note relating I. Segal’s notion to relative entropy is</p> <ul> <li><a class="existingWikiWord" href="/nlab/show/Roberto+Longo">Roberto Longo</a>, <a class="existingWikiWord" href="/nlab/show/Edward+Witten">Edward Witten</a>, <em>A note on continuous entropy</em>, <a href="https://arxiv.org/abs/2202.03357">arXiv:2202.03357</a></li> </ul> <p>A large collection of references on quantum entropy is in</p> <ul> <li><a class="existingWikiWord" href="/nlab/show/Christopher+Fuchs">Christopher Fuchs</a>, <em>References for research in quantum distinguishability and state disturbance</em> (<a href="http://www.perimeterinstitute.ca/personal/cfuchs/BigRef.pdf">pdf</a>)</li> </ul> <h3 id="category_theoretic_and_cohomological_interpretations">Category theoretic and cohomological interpretations</h3> <p>A discussion of entropy with an eye towards the <a class="existingWikiWord" href="/nlab/show/presheaf+topos">presheaf topos</a> over the <a class="existingWikiWord" href="/nlab/show/site">site</a> of finite <a class="existingWikiWord" href="/nlab/show/measure+spaces">measure spaces</a> is in</p> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/Mikhail+Gromov">Mikhail Gromov</a>, <em>In a search for a structure, Part I: On entropy</em> (2012) (<a href="https://www.ihes.fr/~gromov/wp-content/uploads/2018/08/structre-serch-entropy-july5-2012.pdf">pdf</a>)</p> </li> <li> <p><a class="existingWikiWord" href="/nlab/show/William+Lawvere">William Lawvere</a>, <em>State categories, closed categories, and the existence</em> [of] <em>semi-continuous entropy functions</em>, IMA Preprint Series #86, 1984. (<a href="https://www.ima.umn.edu/preprints/State-Categories-Closed-Categories-and-Existence-Semi-Continuous-Entropy-Functions">web</a>)</p> </li> </ul> <p>Category-theoretic characterizations of entropy are discussed in</p> <ul> <li> <p><a class="existingWikiWord" href="/nlab/show/John+Baez">John Baez</a>, <em><a class="existingWikiWord" href="/johnbaez/show/Entropy+as+a+functor">Entropy as a functor</a></em></p> </li> <li> <p>Cheuk Ting Li, <em>A Characterization of Entropy as a Universal Monoidal Natural Transformation</em> [<a href="https://arxiv.org/abs/2308.05742">arXiv:2308.05742</a>]</p> </li> </ul> <p>A (co)homological viewpoint is discussed in</p> <ul> <li>Pierre Baudot, <a class="existingWikiWord" href="/nlab/show/Daniel+Bennequin">Daniel Bennequin</a>, <em>The homological nature of entropy</em>, Entropy, 17 (2015), 3253-3318. (<a href="https://doi.org/10.3390/e17053253">doi</a>)</li> </ul> <p>(for an update see also the abstract of a talk of Baudot <a href="https://calendar.math.cas.cz/content/information-cohomology-and-topological-information-data-analysis">here</a>)</p> <h3 id="ReferencesAxiomaticCharacterization">Axiomatic characterizations</h3> <p>After the concept of entropy proved enormously useful in practice, many people have tried to find a more abstract foundation for the concept (and its variants) by characterizing it as the unique measure satisfying some list of plausible-sounding axioms.</p> <p>A characterization of <a href="#RelativeEntropy">relative entropy</a> on finite-<a class="existingWikiWord" href="/nlab/show/dimension">dimensional</a> <a class="existingWikiWord" href="/nlab/show/C-star+algebras">C-star algebras</a> is given in</p> <ul> <li id="Petz92">D. Petz, <em>Characterization of the relative entropy of states of matrix algebras</em>, Acta Math. Hung. 59 (3-4) (1992) (<a href="http://www.renyi.hu/~petz/pdf/52.pdf">pdf</a>)</li> </ul> <p>A simple characterization of von Neumann entropy of <a class="existingWikiWord" href="/nlab/show/density+matrices">density matrices</a> (mixed <a class="existingWikiWord" href="/nlab/show/quantum+states">quantum states</a>) is discussed in</p> <ul> <li>Bernhard Baumgartner, <em>Characterizing Entropy in Statistical Physics and in Quantum Information Theory</em> (<a href="http://arxiv.org/abs/1206.5727">arXiv:1206.5727</a>)</li> </ul> <p>Entropy-like quantities appear in the study of many PDEs, with entropy estimates. For an intro see</p> <ul> <li>L. C. Evans, <em>A survey of entropy methods for partial differential equations</em>, Bull. Amer. Math. Soc. 41 (2004), 409-438 (<a href="https://www.ams.org/journals/bull/2004-41-04/S0273-0979-04-01032-8/">web</a>, <a href="http://math.berkeley.edu/~evans/ams.entropy.pdf">pdf</a>); and longer Berkeley graduate course text: <em>Entropy and partial differential equations</em>. (<a href="http://math.berkeley.edu/~evans/entropy.and.PDE.pdf">pdf</a>)</li> </ul> </body></html> </div> <div class="revisedby"> <p> Last revised on September 17, 2024 at 08:16:03. See the <a href="/nlab/history/entropy" style="color: #005c19">history</a> of this page for a list of all contributions to it. </p> </div> <div class="navigation navfoot"> <a href="/nlab/edit/entropy" accesskey="E" class="navlink" id="edit" rel="nofollow">Edit</a><a href="https://nforum.ncatlab.org/discussion/12828/#Item_9">Discuss</a><span class="backintime"><a href="/nlab/revision/entropy/62" accesskey="B" class="navlinkbackintime" id="to_previous_revision" rel="nofollow">Previous revision</a></span><a href="/nlab/show/diff/entropy" accesskey="C" class="navlink" id="see_changes" rel="nofollow">Changes from previous revision</a><a href="/nlab/history/entropy" accesskey="S" class="navlink" id="history" rel="nofollow">History (62 revisions)</a> <a href="/nlab/show/entropy/cite" style="color: black">Cite</a> <a href="/nlab/print/entropy" accesskey="p" id="view_print" rel="nofollow">Print</a> <a href="/nlab/source/entropy" id="view_source" rel="nofollow">Source</a> </div> </div>  </div>  </body> </html>