CINXE.COM
Language models can explain neurons in language models
<!DOCTYPE html> <head> <title>Language models can explain neurons in language models</title> <link rel="icon" type="image/svg+xml" href="data:image/svg+xml, <svg width="96" height="96" viewBox="0 0 96 96" fill="none" style="color:rgba(255,255,255,1);fill:rgba(0,0,0,1)" xmlns="http://www.w3.org/2000/svg"> <rect width="96" height="96"/> <path d="M78.8303 41.4616C79.637 39.0379 79.9168 36.4697 79.651 33.9291C79.3852 31.3885 78.5798 28.9339 77.2888 26.7296C75.3745 23.3959 72.4506 20.7564 68.9389 19.1921C65.4273 17.6277 61.5095 17.2193 57.7507 18.0258C56.0552 16.1152 53.9712 14.5888 51.6382 13.5487C49.3051 12.5086 46.7768 11.9788 44.2224 11.9948C40.3794 11.9857 36.6328 13.1972 33.5226 15.4546C30.4125 17.7119 28.0995 20.8985 26.917 24.5551C24.4136 25.0679 22.0486 26.1094 19.9801 27.61C17.9117 29.1105 16.1876 31.0355 14.923 33.2561C12.9938 36.5809 12.1704 40.4326 12.5717 44.2556C12.973 48.0787 14.5782 51.6755 17.1558 54.5272C16.3487 56.9509 16.0686 59.519 16.3342 62.0596C16.5998 64.6003 17.405 67.0549 18.6958 69.2592C20.6102 72.593 23.5341 75.2325 27.0457 76.7968C30.5573 78.3612 34.4752 78.7695 38.2339 77.963C39.9291 79.874 42.0131 81.4007 44.3462 82.4408C46.6793 83.4809 49.2078 84.0105 51.7623 83.9941C55.6081 84.0053 59.358 82.794 62.4706 80.5351C65.5832 78.2762 67.8974 75.0865 69.079 71.4267C71.5826 70.9144 73.9478 69.8731 76.0163 68.3725C78.0849 66.8719 79.8089 64.9467 81.073 62.7257C82.9995 59.4007 83.8204 55.5498 83.4173 51.7282C83.0143 47.9065 81.4081 44.3116 78.8303 41.4616ZM51.9099 79.2873C48.3273 79.2873 45.5538 78.1873 43.1309 76.1646C43.2402 76.105 43.4318 76 43.5567 75.9233L57.8927 67.6425C58.2527 67.438 58.5517 67.1412 58.7587 66.7827C58.9658 66.4242 59.0735 66.0169 59.0708 65.6029V45.392L65.1316 48.8908C65.1633 48.9068 65.1905 48.9303 65.211 48.9593C65.2316 48.9882 65.2447 49.0217 65.2494 49.0569V65.7902C65.248 73.3826 58.9274 79.2873 51.9099 79.2873ZM22.778 66.9059C21.197 64.1743 20.6272 60.9736 21.1684 57.8643C21.2749 57.9281 21.4608 58.0417 21.5942 58.1183L35.9302 66.3991C36.2874 66.6082 36.6938 66.7184 37.1076 66.7184C37.5215 66.7184 37.9279 66.6082 38.285 66.3991L55.7877 56.293V63.2906C55.7898 63.3264 55.783 63.3621 55.7679 63.3946C55.7528 63.427 55.7298 63.4552 55.7011 63.4766L41.209 71.844C38.1091 73.6293 34.4277 74.112 30.9724 73.1862C27.517 72.2604 24.5701 70.0018 22.778 66.9059ZM19.001 35.6094C20.5754 32.8745 23.0615 30.7803 26.0242 29.6933C26.0242 29.8168 26.0242 30.0354 26.0242 30.1873V46.7489C26.0212 47.1626 26.1287 47.5696 26.3356 47.9279C26.5424 48.2862 26.8411 48.5828 27.2009 48.7872L44.7021 58.8919L38.6427 62.3907C38.6127 62.4103 38.5784 62.4222 38.5428 62.4254C38.5072 62.4287 38.4713 62.4231 38.4383 62.4092L23.9448 54.0347C20.8494 52.2436 18.5907 49.2981 17.6639 45.8441C16.7371 42.39 17.218 38.7094 19.001 35.6094ZM68.7909 47.196L51.2882 37.0899L57.3476 33.5925C57.3776 33.5729 57.4119 33.561 57.4475 33.5577C57.4832 33.5545 57.519 33.5601 57.552 33.574L72.0455 41.9442C74.2647 43.2275 76.0724 45.1162 77.2572 47.3894C78.4421 49.6626 78.955 52.2263 78.7359 54.7804C78.5169 57.3344 77.575 59.7733 76.0204 61.8116C74.4658 63.8498 72.3628 65.4032 69.9576 66.2898V49.2329C69.961 48.8204 69.8548 48.4145 69.6498 48.0566C69.4448 47.6987 69.1483 47.4017 68.7909 47.196ZM74.8219 38.1118C74.7154 38.0465 74.5295 37.9344 74.3961 37.8578L60.0601 29.577C59.7027 29.3685 59.2964 29.2586 58.8827 29.2586C58.469 29.2586 58.0627 29.3685 57.7053 29.577L40.2026 39.6831V32.6855C40.2006 32.6497 40.2075 32.6141 40.2226 32.5816C40.2377 32.5492 40.2606 32.521 40.2892 32.4995L54.7813 24.1392C57.0015 22.859 59.5404 22.2375 62.1008 22.3474C64.6613 22.4574 67.1375 23.2941 69.2398 24.7599C71.3421 26.2257 72.9836 28.2598 73.9721 30.6243C74.9606 32.9888 75.2554 35.5859 74.8219 38.1118ZM36.9082 50.5898L30.8473 47.091C30.8152 47.0756 30.7876 47.0522 30.767 47.0232C30.7464 46.9941 30.7335 46.9603 30.7295 46.9249V30.1873C30.7312 27.6237 31.463 25.1137 32.8394 22.951C34.2158 20.7882 36.1797 19.0623 38.5014 17.9752C40.823 16.8881 43.4063 16.4848 45.9488 16.8125C48.4914 17.1402 50.8879 18.1854 52.858 19.8256C52.7487 19.8853 52.5585 19.9903 52.4322 20.0669L38.0962 28.3478C37.7367 28.5525 37.4381 28.8491 37.2311 29.2073C37.0241 29.5655 36.9161 29.9723 36.9181 30.386L36.9082 50.5898ZM40.1998 43.4928L47.9952 38.9904L55.7905 43.49V52.4918L47.9952 56.9899L40.1998 52.4904V43.4928Z" fill="currentColor"/> </svg> "> <meta charset="utf-8" /> <meta name="viewport" content="width=1300" /> <style id="distill-article-specific-styles"> @media (min-height: 900px) { d-article hr { margin-top: 120px !important; margin-bottom: 100px !important; } } pre.citation { font-size: 11px; line-height: 15px; border-left: 1px solid rgba(0, 0, 0, 0.1); padding-left: 18px; border: 1px solid rgba(0,0,0,0.1); background: rgba(255,255,255,0.4); padding: 10px 18px; border-radius: 3px; color: rgba(150, 150, 150, 1); overflow: hidden; margin-top: -12px; white-space: pre-wrap; word-wrap: break-word; } .subgrid { grid-column: screen; display: grid; grid-template-columns: inherit; grid-template-rows: inherit; grid-column-gap: inherit; grid-row-gap: inherit; } d-figure.base-grid { grid-column: screen; background: hsl(0, 0%, 97%); padding: 20px 0; border-top: 1px solid rgba(0, 0, 0, 0.1); border-bottom: 1px solid rgba(0, 0, 0, 0.1); } d-figure { margin-bottom: 1em; position: relative; } d-figure > figure { margin-top: 0; margin-bottom: 0; } .shaded-figure { background-color: hsl(0, 0%, 97%); border-top: 1px solid hsla(0, 0%, 0%, 0.1); border-bottom: 1px solid hsla(0, 0%, 0%, 0.1); padding: 30px 0; } .pointer { position: absolute; width: 26px; height: 26px; top: 26px; left: -48px; } .fullscreen-diagram { grid-column: screen; background: #f8f8fb; padding: 10px; padding-top: 40px; padding-bottom: 40px; border-top: 1px solid #f0f0f0; border-bottom: 1px solid #f0f0f0; margin-top: 40px; margin-bottom: 60px; } .centered-img { margin-left: auto; margin-right: auto; display: block; } .margin-diagram { grid-column: 12 / 15; grid-row: auto / span 3; margin-top: 6px; } @media (min-width: 1800px) { .margin-diagram { margin-left: 60px; } } .placeholder { padding: 40px; font-size: 80%; background: #f8f8fb; border: 1px solid #f0f0f0; border-radius: 4px; grid-column: text; margin-top: 20px; margin-bottom: 20px; } .todo { margin-top: 8px; padding: 10px; font-size: 80%; line-height: 140%; background: rgb(250, 249, 242); border: 1px solid rgb(233, 230, 214); border-radius: 4px; max-width: 250px; margin-right: 4px; grid-column-start: gutter-start; grid-column-end: screen-end; height: fit-content; } /* .sensitivity-warn { margin-bottom: 16px; padding: 10px; font-size: 80%; line-height: 140%; background: #EEE; border: 1px solid #AAA; border-radius: 4px; height: fit-content; } */ .sensitivity-warn { border-left: 8px solid hsl(40, 70%, 80%); padding: 12px; padding-left: 16px; margin-bottom: 16px; border-radius: 4px; background: hsl(40, 80%, 95%); } .sensitivity-warn:before { content: "Content Warning"; font-weight: bold; display: block; padding-bottom: 4px; } d-footnote { margin-left: -1px; margin-right: 4px; } .footnote-sep { width: 0px; } .footnote-sep::before { line-height: 1em; top: -0.5em; font-size: 0.75em; color: #999; margin-left: -6px; margin-right: 1px; content: ','; } d-footnote[comma]::before { vertical-align: super; line-height: 1em; top: -0.5em; font-size: 0.75em; color: #999; margin-left: -6px; margin-right: 1px; content: ','; } d-article h2 { border-bottom: none; } /* to make toc sticky */ d-article { overflow: visible !important; } .toc { position: sticky; top: 20px; } .toc h4 { margin: 5px 0px 0px 0px } .toc h5 { margin: 5px 0px 0px 10px } .toc h6 { margin: 0px 0px 0px 20px } .toc h6.toch7 { margin: 0px 0px 0px 30px } /* **************************************** * TOC ******************************************/ /* @media (max-width: 1300px) { */ /* d-contents { */ /* display: none; */ /* } */ /* } */ /* d-article d-contents { align-self: start; grid-column: 1 / 4; grid-row: auto / span 3; width: calc(100% + 32px); margin-right: -16px; margin-top: 0em; display: grid; grid-template-columns: minmax(8px, 1fr) [toc] auto minmax(8px, 1fr) [toc-line] 1px minmax(32px, 2fr ); } d-article d-contents nav { grid-column: toc; } d-article d-contents .toc-line { border-right: 1px solid rgba(0, 0, 0, 0.1); grid-column: toc-line; } */ b i { display: inline; } d-article d-contents { align-self: start; grid-column: 1 / 4; grid-row: auto / span 4; justify-self: end; margin-top: 0em; padding-right: 3em; padding-left: 2em; border-right: 1px solid rgba(0, 0, 0, 0.1); display: none; } @media (min-width: 1024px) { d-article d-contents { display: block; } } d-contents a:hover { border-bottom: none; } d-contents nav h3 { margin-top: 0; margin-bottom: 1em; } d-contents nav div { color: rgba(0, 0, 0, 0.8); font-weight: bold; } d-contents nav a { color: rgba(0, 0, 0, 0.8); border-bottom: none; text-decoration: none; } d-contents li { list-style-type: none; } d-contents ul { padding-left: 1em; } d-contents nav ul li { margin-bottom: 0.25em; } d-contents nav a:hover { text-decoration: underline solid rgba(0, 0, 0, 0.6); } d-contents nav ul { margin-top: 0; margin-bottom: 6px; } d-contents nav > div { display: block; outline: none; margin-bottom: 0.5em; } d-contents nav > div > a { font-size: 13px; font-weight: 600; } d-contents nav > div > a:hover, d-contents nav > ul > li > a:hover { text-decoration: none; } d-article h2 { border-bottom: none !important; } /* **************************************** * toggle switch * https://www.w3schools.com/howto/howto_css_switch.asp ******************************************/ .switch { position: relative; display: inline-block; width: 4em; height: 2em; } /* Hide default HTML checkbox */ .switch input { opacity: 0; width: 0; height: 0; } /* The slider */ .toggleslider { position: absolute; cursor: pointer; top: 0; left: 0; right: 0; bottom: 0; background-color: #ccc; -webkit-transition: .4s; transition: .4s; } .toggleslider:before { position: absolute; content: ""; height: 1.6em; width: 1.8em; left: 0.2em; bottom: 0.2em; background-color: #2196F3; -webkit-transition: .4s; transition: .4s; } input + .toggleslider { /* background-color: #2196F3; */ } input:focus + .toggleslider { box-shadow: 0 0 1px #2196F3; } input:checked + .toggleslider:before { -webkit-transform: translateX(1.8em); -ms-transform: translateX(1.8em); transform: translateX(1.8em); } /* **************************************** * toggle switch * https://www.w3schools.com/howto/howto_css_switch.asp ******************************************/ .slidecontainer { width: 100%; /* Width of the outside container */ } /* The slider itself */ .slider { -webkit-appearance: none; /* Override default CSS styles */ appearance: none; width: 100%; /* Full-width */ height: 25px; /* Specified height */ background: #d3d3d3; /* Grey background */ outline: none; /* Remove outline */ opacity: 0.7; /* Set transparency (for mouse-over effects on hover) */ -webkit-transition: .2s; /* 0.2 seconds transition on hover */ transition: opacity .2s; } /* Mouse-over effects */ .slider:hover { opacity: 1; /* Fully shown on mouse-over */ } /* The slider handle (use -webkit- (Chrome, Opera, Safari, Edge) and -moz- (Firefox) to override default look) */ .slider::-webkit-slider-thumb { -webkit-appearance: none; /* Override default look */ appearance: none; width: 25px; /* Set a specific slider handle width */ height: 25px; /* Slider handle height */ background: #04AA6D; /* Green background */ cursor: pointer; /* Cursor on hover */ } .slider::-moz-range-thumb { width: 25px; /* Set a specific slider handle width */ height: 25px; /* Slider handle height */ background: #04AA6D; /* Green background */ cursor: pointer; /* Cursor on hover */ } /* **************************************** * Byline ******************************************/ d-byline { display: none !important; } .newd-byline-container { border-top: 1px solid rgba(0, 0, 0, 0.1); border-bottom: 1px solid rgba(0, 0, 0, 0.1); padding-top: 20px; padding-bottom: 20px; } .newd-byline { grid-column: text; display: grid; grid-template-columns: [authors] 5fr [affiliations published] 1fr; grid-template-rows: [authors-start affiliations-start] auto [affiliations-end published-start] auto; font-size: 0.8rem; line-height: 1.8em; grid-gap: 20px; } .newd-byline h3 { font-size: 0.6rem; font-weight: 400; color: rgba(0, 0, 0, 0.5); margin: 0; text-transform: uppercase; margin-bottom: 4px; } .newd-byline .info { color: rgba(0, 0, 0, 0.5); font-size: 80%; line-height: 1.5; } .newd-byline .authors div { line-height: 1.5; } .newd-byline .authors sup { font-size: 80%; margin-right: -4px; } .newd-byline .author { margin-right: 6px; white-space: nowrap; } .newd-byline a { color: inherit; text-decoration: none; } </style> <script src="distill-template.v2.js"></script> </head> <body> <d-front-matter> <script type="text/json"> { "title": "Language models can explain neurons in language models", "description": "We use GPT-4 to automatically explain neurons in GPT-2, and to score those explanations.", "authors": [ { "author": "", "authorURL": "", "affiliation": "OpenAI", "affiliationURL": "http://openai.com" } ], "katex": { "delimiters": [ { "left": "$", "right": "$", "display": false }, { "left": "$$", "right": "$$", "display": true } ] } } </script> </d-front-matter> <d-title> <h1>Language models can explain neurons in language models</h1> </d-title> <div class='newd-byline-container base-grid'> <div class='newd-byline'> <div class='authors' style='grid-area: authors'> <h3>Authors</h3> <div> <span class='author'>Steven Bills<sup>∗</sup>,</span> <span class='author'>Nick Cammarata<sup>∗</sup>,</span> <span class='author'>Dan Mossing<sup>∗</sup>,</span> <span class='author'>Henk Tillman<sup>∗</sup>,</span> <span class='author'>Leo Gao<sup>∗</sup>,</span> <span class='author'>Gabriel Goh,</span> <span class='author'>Ilya Sutskever,</span> <span class='author'>Jan Leike,</span> <span class='author'>Jeff Wu<sup>∗</sup>,</span> <span class='author'>William Saunders<sup>∗</sup></span> </div> <div class='info' style='margin-top: 7px'> <div> <span style='margin-right: 10px; white-space:nowrap'>* Core Research Contributor;</span> <span style='margin-right: 10px; white-space:nowrap'><a href='#sec-author-contributions'>Author contributions statement below</a>.</span> <span style='margin-right: 10px; white-space:nowrap'> Correspondence to <a href='mailto:interpretability@openai.com'>interpretability@openai.com</a>. </span> </div> </div> </div> <div class='affiliations' style='grid-area: affiliations'> <h3>Affiliation</h3> <div><a href='https://www.openai.com/'>OpenAI</a></div> </div> <div class='published' style='grid-area: published'> <h3>Published</h3> <div>May 9, 2023</div> </div> </div> </div> <d-article> <d-article-contents> </d-article-contents> <d-contents style='height: 100%; position: absolute; margin-top: 50px'> <nav class="toc figcaption"> <h3>Contents</h3> <div> <h4><a href="#sec-intro">Introduction</a></h4> <h4><a href="#sec-methods">Methods</a></h4> <h5><a href="#sec-setting">Setting</a></h5> <h5><a href="#sec-algorithm">Overall algorithm</a></h5> <h6><a href="#sec-algorithm-explain">Step 1: Explanation</a></h6> <h6><a href="#sec-algorithm-simulate">Step 2: Simulation</a></h6> <h6><a href="#sec-algorithm-score">Step 3: Scoring</a></h6> <h6 class="toch7"><a href="#sec-ablation-scoring">Ablation scoring</a></h6> <h6 class="toch7"><a href="#sec-human-scoring">Human scoring</a></h6> <h6><a href="#sec-algorithm-details">Parameters and details</a></h6> <h4><a href="#sec-results">Results</a></h4> <h5><a href="#sec-baselines">Unigram baselines</a></h5> <h6><a href="#sec-token-weight">Token weights</a></h6> <h6><a href="#sec-token-table">Token lookup table</a></h6> <h5><a href="#sec-next-token">Next-token explanations</a></h5> <h5><a href="#sec-revisions">Revisions</a></h5> <h5><a href="#sec-direction-finding">Direction finding</a></h5> <h5><a href="#sec-assistant-trends">Assistant model trends</a></h5> <h6><a href="#sec-explainer-scaling">Explainer model size</a></h6> <h6><a href="#sec-simulator-scaling">Simulator model size</a></h6> <h5><a href="#sec-subject-trends">Subject model trends</a></h5> <h6><a href="#sec-subject-size">Subject model size</a></h6> <h6><a href="#sec-subject-activation">Activation function</a></h6> <h6><a href="#sec-subject-training">Training time</a></h6> <h5><a href="#sec-qualitative">Qualitative results</a></h5> <h6><a href="#sec-interesting-neurons">Interesting neurons</a></h6> <h6><a href="#sec-puzzles">Explaining puzzles</a></h6> <h4><a href="#sec-discussion">Discussion</a></h4> <h5><a href="#sec-limitations">Limitations</a></h5> <h5><a href="#sec-future-work">Future Work</a></h5> <br /> </div> </nav> <div class="toc-line"></div> </d-contents> </d-article> <d-appendix style="background-color: hsla(40,20%,30%,0.05);"> <h3 id="sec-author-contributions">Contributions</h3> <p> <b>Methodology:</b> Nick effectively started the project by having the initial idea to have GPT-4 explain neurons, and showing a simple explanation methodology worked. William came up with the initial simulation and scoring methodology and implementation. Dan and Steven ran many experiments resulting in ultimate choices of prompts and explanation/scoring parameters. </p> <p> <b>ML infrastructure:</b> William and Nick set up the initial version of the codebase. Leo and Jeff implemented the initial core internal infrastructure for doing interpretability. Steven implemented the top activations pipeline. Steven and William developed the pipeline for explanations and scoring. Many other miscellaneous contributions came from William, Jeff, Dan, and Steven. Steven created the open source version. </p> <p> <b>Web infrastructure:</b> Nick and William implemented the neuron viewer, with smaller contributions from Steven, Dan, and Jeff. Nick implemented many other UIs exploring various kinds of neuron explanation. Steven implemented human data gathering UIs. </p> <p> <b>Human data:</b> Steven implemented and analyzed all experiments involving contractor human data: the human explanation baseline, and human scoring experiments. Nick and William implemented early researcher explanation baselines. </p> <p> <b>Alternative token and weight-based explanations:</b> Dan implemented all experiments and analysis on token weight and token lookup baselines, next token explanations, as well as infrastructure and UIs for neuron-neuron connection weights. </p> <p> <b>Revisions:</b> Henk implemented and analyzed the main revision experiments. Nick championed and implemented an initial proof of concept for revisions. Leo implemented a small scale derisking experiment. Steven helped design the final revision pipeline. Leo and Dan did many crucial investigations into negative finding. </p> <p> <b>Direction finding:</b> Leo had the idea and implementated all experiments related to direction finding. </p> <p> <b>Neuron puzzles:</b> Henk implemented all the neuron puzzles and related experiments. William came up with the initial idea. Steven and William gave feedback on data and strategies. </p> <p> <b>Subject, explainer, and simulator scaling:</b> Steven implemented and analyzed assistant size, simulator size, and subject size experiments. Jeff implemented and analyzed subject training time experiments. </p> <p> <b>Ablation scoring:</b> Jeff implemented ablation infrastructure and initial scoring experiments, and Dan contributed lots of useful thinking and carried out final experiments. Leo did related investigations into understanding and prediction of ablation effects. </p> <p> <b>Activation function experiments:</b> Jeff implemented the experiments and analysis. Gabe suggested the sparse activation function, and William suggested correlation-based community detection. </p> <p> <b>Qualitative results:</b> Everyone contributed throughout the project to qualitative findings. Nick and William discovered many of the earliest nontrivial neurons. Dan found many non-trivial neurons by comparing to token baselines, such as simile neurons. Steven found the pattern break neuron, and other context-sensitive neurons. Leo discovered the "don't stop" neuron and first noticed explanations were overly broad. Henk had many qualitative findings about explanation and scoring quality. Nick found interesting neuron-neuron connections and interesting pairs of neurons firing on the same token. William and Jeff investigated ablations of specific neurons. </p> <p> <b>Guidance and mentorship:</b> William and Jeff led and managed the project. Jan and Jeff managed team members who worked on the project. Many ideas from Jan, Nick, William, and Ilya influenced the direction of the project. Steven mentored Henk. </p> <h3>Acknowledgments</h3> <p> We thank Neel Nanda, Ryan Greenblatt, Paul Christiano, Chris Olah, and Evan Hubinger for useful discussions on direction during the project. </p> <p> We thank Ryan Greenblatt, Buck Shlegeris, Trenton Bricken, the OpenAI Alignment team, and the Anthropic Interpretability team for useful discussions and feedback. </p> <p> We thank Cathy Yeh for doing useful checks of our code and noticing tokenization concerns. </p> <p> We thank Carroll Wainwright and Chris Hesse for help with infrastructure. </p> <p> We thank the contractors who wrote and evaluated explanations for our human data experiments, and Long Ouyang for some feedback on our instructions. </p> <p> We thank Rajan Troll for providing a human baseline for the neuron puzzles. </p> <p> We thank Thomas Degry for the blog graphic. </p> <p> We thank other OpenAI teams for their support, including the supercomputing, research acceleration, and language teams. </p> <h3> Citation Information </h3> <p> Please cite as: <pre class='citation'> Bills, et al., "Language models can explain neurons in language models", 2023. </pre> <p> BibTeX Citation: </p> <pre class='citation'> @misc{bills2023language, title={Language models can explain neurons in language models}, author={ Bills, Steven and Cammarata, Nick and Mossing, Dan and Tillman, Henk and Gao, Leo and Goh, Gabriel and Sutskever, Ilya and Leike, Jan and Wu, Jeff and Saunders, William }, year={2023}, howpublished = {\url{https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html}} }</pre> </p> <d-footnote-list></d-footnote-list> <d-citation-list></d-citation-list> </d-appendix> <d-bibliography src="bibliography.bib"></d-bibliography> <script type="text/javascript" src="index.bundle.js"></script></body>