CINXE.COM

<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="apple-touch-icon" sizes="180x180" href="https://pair.withgoogle.com/images/favicon/apple-touch-icon.png"> <link rel="icon" type="image/png" sizes="32x32" href="https://pair.withgoogle.com/images/favicon/favicon-32x32.png"> <link rel="icon" type="image/png" sizes="16x16" href="https://pair.withgoogle.com/images/favicon/favicon-16x16.png"> <link rel="mask-icon" href="https://pair.withgoogle.com/images/favicon/safari-pinned-tab.svg" color="#00695c"> <script> !(function(){ var url = window.location.href if (url.split('#')[0].split('?')[0].slice(-1) != '/' && !url.includes('.html')) window.location = url + '/' })() </script> <title>Do Machine Learning Models Memorize or Generalize?</title> <meta property="og:title" content="Do Machine Learning Models Memorize or Generalize?"> <meta property="og:url" content="https://pair.withgoogle.com/explorables/grokking/"> <meta name="og:description" content="An interactive introduction to grokking and mechanistic interpretability."> <meta property="og:image" content="https://pair.withgoogle.com/explorables/images/grokking.png"> <meta name="twitter:card" content="summary_large_image"> <link rel="stylesheet" type="text/css" href="../style_v2.css"> <link href='https://fonts.googleapis.com/css?family=Roboto+Slab:400,500,700|Roboto:700,500,300' rel='stylesheet' type='text/css'> <link href="https://fonts.googleapis.com/css?family=Google+Sans:400,500,700" rel="stylesheet"> <meta name="viewport" content="width=device-width"> </head> <body> <div class='header'> <div class='header-left'> <a href='https://pair.withgoogle.com/'> <img src='../images/pair-logo.svg' style='width: 100px'></img> </a> <a href='../'>Explorables</a> </div> </div> <h1 class='headline'>Do Machine Learning Models Memorize or Generalize?</h1> <div class='byline'> <div class='authors'> By Adam Pearce, Asma Ghandeharioun, Nada Hussein, Nithum Thain, Martin Wattenberg and Lucas Dixon </div> <div class='date'>August 2023</div> </div> In 2021, researchers made a striking discovery while training a series of tiny models on toy tasks <a class='citestart' key='Grokking'></a>. They found a set of models that suddenly flipped from memorizing their training data to correctly generalizing on unseen inputs after training for much longer. This phenomenon – where generalization seems to happen abruptly and long after fitting the training data – is called grokking and has sparked a flurry of interest <a class='citestart' key='Omnigrok Universality Zhong23 ProgressParity gromov'></a>. <div class='sticky-container'> <div class='mod-top-accuracy row sticky'></div> Do more complex models also suddenly generalize after they’re trained longer? Large language models can certainly seem like they have a rich understanding of the world, but they might just be regurgitating memorized bits of the enormous amount of text they’ve been trained on <a class='citestart' key='Parrots Othello'></a>. How can we tell if they’re generalizing or memorizing? In this article we’ll examine the training dynamics of a tiny model and reverse engineer the solution it finds – and in the process provide an illustration of the exciting emerging field of mechanistic interpretability <a class='citestart' key='MechInterp ProgressMeasures'></a>. While it isn’t yet clear how to apply these techniques to today’s largest models, starting small makes it easier to develop intuitions as we progress towards answering these critical questions about large language models. <h3 id="grokking-modular-addition">Grokking Modular Addition</h3> Modular addition is essentially the fruit fly of grokking.<a class='footstart' key='modular'></a> The above line chart comes from a model trained to predict <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo>+</mo><mi>b</mi><mtext> </mtext><mo lspace="0.22em" rspace="0.22em"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow></mo><mtext> </mtext><mn>67</mn></mrow><annotation encoding="application/x-tex">a + b \bmod 67</annotation></semantics></math>a+bmod67.<a class='footstart' key='67' ></a> We start by randomly dividing all the <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo separator="true">,</mo><mi>b</mi></mrow><annotation encoding="application/x-tex">a, b</annotation></semantics></math>a,b pairs into test and training datasets. Over thousands of training steps, the training data is used to adjust the model into outputting correct answers, while the test data is only used to check if the model has learned a general solution. The model’s architecture is similarly simple: <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>ReLU</mtext><mrow><mo fence="true">(</mo><msub><mi mathvariant="bold">a</mi><mtext>one-hot</mtext></msub><msub><mi mathvariant="bold">W</mi><mtext>input</mtext></msub><mo>+</mo><msub><mi mathvariant="bold">b</mi><mtext>one-hot</mtext></msub><msub><mi mathvariant="bold">W</mi><mtext>input</mtext></msub><mo fence="true">)</mo></mrow><msub><mi mathvariant="bold">W</mi><mtext>output</mtext></msub></mrow><annotation encoding="application/x-tex">\text{ReLU}\left(\mathbf{a}_{\text{one-hot}} \mathbf{W}_{\text{input}} + \mathbf{b}_{\text{one-hot}} \mathbf{W}_{\text{input}}\right) \mathbf{W}_{\text{output}} </annotation></semantics></math>ReLU(aone-hotWinput+bone-hotWinput)Woutput — a one-layer MLP with 24 neurons.<a class='footstart' key='playground'></a> All the weights of the model are shown in the heatmap below; you can see how they change during training by mousing over the line chart above. <div class='sticky-container'> <div class='mod-top-weights row x-sticky x-sticky-lower'></div> The model makes a prediction by selecting the two columns of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>input</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{input}}</annotation></semantics></math>Winput corresponding to inputs <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math>a and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math>b then adding them together to create a vector of 24 separate numbers. Next it sets all the negative numbers in the vector to 0 and finally outputs the column of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>output</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{output}}</annotation></semantics></math>Woutput that’s closest to the updated vector. The weights of the model are initially quite noisy but start to exhibit periodic patterns as accuracy on the test data increases and the model <animate data-animate='top-switches'>switches</animate> to generalizing. By the end of training, each neuron — each row of the heatmap — cycles through high and low values several times as the input number increases from 0 to 66. This is easier to see if we group the neurons by how often they cycle at the end of training and chart each of them as a separate line: </div> </div> <div class='mod-top-waves row'></div> The periodic patterns suggest the model is learning some sort of mathematical structure; the fact that it happens when the model starts to solve the test examples hints that it’s related to the model generalizing. But why does the model move away from the memorizing solution? And what is the generalizing solution? <h3 id="generalizing-with-1s-and-0s">Generalizing With 1s and 0s</h3> Figuring out both of these questions simultaneously is hard. Let’s make an even simpler task, one where we know what the generalizing solution should look like and try to understand why the model eventually learns it. We’ll take random sequences of thirty 1s and 0s and train our model to predict if there is an odd number of 1s in the first three digits. e.g. <digits>000110010110001010111001001011</digits> is <digits>0</digits> while <digits>010110010110001010111001001011</digits> is <digits>1</digits> — basically a slightly trickier XOR with some distraction noise. A generalizing model should only use the first three digits of the sequence; if the model is memorizing the training data, it will also use the subsequent distracting digits <a class='citestart' key='ProgressParity TwoCircuits'></a>. Our model is again a one-layer MLP, trained on a fixed batch of 1,200 sequences.<a class='footstart' key='sp-model'></a> At first only training accuracy increases — the model is memorizing the training data. As with modular arithmetic, test accuracy is essentially random and then sharply rises as the model learns a general solution. <div class='parity-accuracy row'></div> While <animate data-animate='sp-mem'>memorizing</animate> , the model looks dense and noisy with lots of high magnitude weights (shown as dark red and blue squares) spread across the chart below – the model is using all the inputs to make a prediction. As the model <animate data-animate='sp-gen'>generalizes</animate> and gets perfect test accuracy, we see all the weights connected to the distracting digits gray out with very low values and the model focusing on the first three digits — mirroring the generalized structure we expected!<a class='footstart' key='sp-solution'></a> <div class='parity-weights row'></div> With this simplified example it’s easier to see why this happens: we’re pushing our model to do two things during training — output a high probability for the correct label (called minimizing loss <a class='footstart' key='loss'></a>) and have weights with low magnitudes (known as weight decay <a class='footstart' key='sp-l2'></a>). Train loss actually slightly increases before the model generalizes as it exchanges loss related to outputting the correct label for having lower weights. <div class='parity-loss row'></div> The sharp drop in test loss makes it appear like the model makes a sudden shift to generalization. But if we look at the weights of the model over training, most of them smoothly interpolate between the two solutions. The rapid generalization occurs when the last weights connected to the distracting digits are pruned by weight decay. <div class='parity-weights-trajectory row'></div> <h3 id="when-does-grokking-happen-">When Does Grokking Happen?</h3> It’s important to note that grokking is a contingent phenomenon — it goes away if model size, weight decay, data size and other hyper parameters aren’t just right. With too little weight decay, the model can’t escape overfitting the training data.<a class='footstart' key='overfit'></a> Adding more weight decay pushes the model to generalize after memorizing. Increasing weight decay even more causes test and train loss to fall together; the model goes straight to generalizing. And with too much weight decay the model will fail to learn anything. Below, we’ve trained over a thousand models on the 1s and 0s task with different hyperparameters. Training is noisy so nine models have been trained for each set of hyperparameters. <div class='sparse-parity-sweep'></div> We can induce memorization and generalization on this somewhat contrived 1s and 0s task — but why does it happen with modular addition? Let’s first understand a little more about how a one-layer MLP can solve modular addition by constructing a generalizing solution that’s interpretable. <h3 id="modular-addition-with-five-neurons">Modular Addition With Five Neurons</h3> Recall that our modular arithmetic problem <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo>+</mo><mi>b</mi><mtext> </mtext><mo lspace="0.22em" rspace="0.22em"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow></mo><mtext> </mtext><mn>67</mn></mrow><annotation encoding="application/x-tex">a + b \bmod 67</annotation></semantics></math>a+bmod67 is naturally periodic, with answers wrapping around if the sum ever passes 67. Mathematically, this can be mirrored by thinking of the sum as wrapping <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math>a and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math>b around a circle. The weights of the generalizing model also had periodic patterns, indicating that the solution might use this property. We can train a simpler model with a head start on the problem, constructing an embedding matrix that places <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math>a and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math>b on a circle by computing <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\cos</annotation></semantics></math>cos and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>sin</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\sin</annotation></semantics></math>sin for each possible input number <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>∈</mo><mo stretchy="false">{</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><mo separator="true">,</mo><mo>…</mo><mo separator="true">,</mo><mn>66</mn><mo stretchy="false">}</mo></mrow><annotation encoding="application/x-tex">i \in \{0, 1, \ldots, 66\}</annotation></semantics></math>i∈{0,1,…,66}.<a class='footstart' key='unit-circle'></a> <script type="math/tex"> $$ \mathbf{W}_{\text{embed}} = \begin{pmatrix} \dots & \ldots \\ \cos(i\frac{2\pi}{67}) & \sin(i \frac{2\pi}{67}) \\ \dots & \dots \\ \end{pmatrix} \quad $$ </script> <div class='row'><div class='embed'></div></div> Then we train <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>in-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{in-proj}}</annotation></semantics></math>Win-proj and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>out-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{out-proj}}</annotation></semantics></math>Wout-proj in this one-layer MLP: <script type="math/tex"> $$ \begin{aligned} \text{activations} & = \text{ReLU}\left(\mathbf{a}_{\text{one-hot}} \mathbf{W}_{\text{embed}} \mathbf{W}_{\text{in-proj}} + \mathbf{b}_{\text{one-hot}} \mathbf{W}_{\text{embed}} \mathbf{W}_{\text{in-proj}}\right) \\ \text{logits} & = \text{activations} \mathbf{W}_{\text{out-proj}} \mathbf{W}_{\text{embed}}^{\top} \end{aligned} $$ </script> With just five neurons the model finds a solution with perfect accuracy. <div class='sticky-container'> <div class='five-neuron-accuracy row sticky'></div> <div class='five-neuron-embed row'></div> Eyeballing the trained parameters, all the neurons <animate data-animate='five-neuron-converge'>converge</animate> to roughly equal norms. If we directly plot their <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\cos</annotation></semantics></math>cos and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>sin</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\sin</annotation></semantics></math>sin components, they’re essentially evenly distributed around a circle: <div class='five-neuron-circle row'></div> Connect the adjacent neurons on the <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>in-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{in-proj}}</annotation></semantics></math>Win-proj circle and an intriguing pattern emerges: <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>out-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{out-proj}}</annotation></semantics></math>Wout-proj is rotating around the circle twice as fast as <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>in-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{in-proj}}</annotation></semantics></math>Win-proj. </div> <div class='five-neuron-circle-2 row'></div> The details of how this solution works aren’t essential — check out <a href="#appendix-a-how-the-circular-construction-works">Appendix A</a> to see how the doubled rotation allows the model to map inputs like <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo><mn>0</mn><mtext> </mtext><mo lspace="0.22em" rspace="0.22em"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow></mo><mtext> </mtext><mn>67</mn></mrow><annotation encoding="application/x-tex">1 + 0 \bmod 67</annotation></semantics></math>1+0mod67 and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mo>+</mo><mn>66</mn><mtext> </mtext><mo lspace="0.22em" rspace="0.22em"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow></mo><mtext> </mtext><mn>67</mn></mrow><annotation encoding="application/x-tex">2 + 66 \bmod 67</annotation></semantics></math>2+66mod67 to the same place — but we have found a 20 parameter construction that solves modular addition. Can we find the same algorithm hidden in the 3,216 parameter model we started with? And why does the larger model switch to the generalizing solution after memorizing? <h3 id="it-s-full-of-stars">It’s Full of Stars</h3> Here’s the <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo>+</mo><mi>b</mi><mtext> </mtext><mo lspace="0.22em" rspace="0.22em"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow></mo><mtext> </mtext><mn>67</mn></mrow><annotation encoding="application/x-tex">a + b \bmod 67</annotation></semantics></math>a+bmod67 model that we started with — it’s trained from scratch with no built-in periodicity. <div class='sticky-container'> <div class='mod-bot-accuracy sticky'></div> <div class='mod-bot-waves row'></div> Unlike the constructed solution, where <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>embed</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{embed}}</annotation></semantics></math>Wembed rotates around the circle once, this model has many different frequencies. Below, we’ve isolated the frequencies using the discrete Fourier transform (DFT).<a class='footstart' key='dft'></a> This factors out the learned periodic patterns across inputs, leaving us with the equivalent of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>in-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{in-proj}}</annotation></semantics></math>Win-proj and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>out-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{out-proj}}</annotation></semantics></math>Wout-proj from the constructed solution. For each neuron, this gives a <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\cos</annotation></semantics></math>cos and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>sin</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\sin</annotation></semantics></math>sin value for every possible periodic frequency from 1 to 33. The wave charts we show above use this to group neurons into frequencies by finding their largest <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\cos</annotation></semantics></math>cos and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>sin</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\sin</annotation></semantics></math>sin value across all frequencies.<a class='footstart' key='dft-sort'></a> <div class='mod-bot-dft row'></div> Just like in the 1s and 0s task, weight decay encourages this representation to become much sparser as the model <animate data-animate='bot-gen'>generalizes</animate> . Grouping neurons by their final trained frequencies, and plotting the <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\cos</annotation></semantics></math>cos and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>sin</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\sin</annotation></semantics></math>sin components of the DFT for each neuron, we see the same star shapes from the constructed solution appear. <div class='mod-bot-freqs-hidden row'></div> <div class='mod-bot-freqs-out row'></div> This trained model is using the same algorithm as our constructed solution! Below, the contribution to the output generated by the neurons in each frequency are shown and we can see them calculating <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo><mfrac><mrow><mn>2</mn><mi>π</mi><mo stretchy="false">(</mo><mi>a</mi><mo>+</mo><mi>b</mi><mo stretchy="false">)</mo><mi>f</mi><mi>r</mi><mi>e</mi><mi>q</mi></mrow><mn>67</mn></mfrac></mrow><annotation encoding="application/x-tex"> \cos\frac{2\pi (a + b) freq}{67}</annotation></semantics></math>cos672π(a+b)freq .<a class='footstart' key='logit-wave'></a> Notice what happens to the group of neurons with a frequency of 7 when test loss <animate data-animate='bot-improve'>improves</animate> after the short plateau at 45,000 steps — they start to snap into a star shape and their outputs more closely approximate a wave. <div class='row mod-bot-sliders'></div> <div class='row mod-bot-logits'></div> To lower loss without using higher weights (which would be punished by weight decay), the model uses several frequencies, taking advantage of constructive interference.<a class='citestart' key='ProgressMeasures'></a> There’s nothing magical about the frequencies 4, 5, 7 and 26 — click through other training runs below to see variations of this algorithm get learned. </div> <div class='row mod-bot-seeds'></div> <h3 id="open-questions">Open Questions</h3> While we now have a solid understanding of the mechanisms a one-layer MLP uses to solve modular addition and why they emerge during training, there are still many interesting open questions about memorization and generalization. <h4 id="which-model-constraints-work-best-">Which Model Constraints Work Best?</h4> Directly training the model visualized above — <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>ReLU</mtext><mrow><mo fence="true">(</mo><msub><mi>a</mi><mtext>one-hot</mtext></msub><msub><mtext mathvariant="bold">W</mtext><mtext>input</mtext></msub><mo>+</mo><msub><mi>b</mi><mtext>one-hot</mtext></msub><msub><mtext mathvariant="bold">W</mtext><mtext>input</mtext></msub><mo fence="true">)</mo></mrow><msub><mtext mathvariant="bold">W</mtext><mtext>output</mtext></msub></mrow><annotation encoding="application/x-tex">\text{ReLU} \left(a_{\text{one-hot}}\textbf{W}_{\text{input}} + b_{\text{one-hot}}\textbf{W}_{\text{input}} \right) \textbf{W}_{\text{output}}</annotation></semantics></math>ReLU(aone-hotWinput+bone-hotWinput)Woutput — does not actually result in generalization on modular arithmetic, even with the addition of weight decay. At least one of the matrices has to be factored: <math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>input</mtext></msub><mo>=</mo><msub><mi mathvariant="bold">W</mi><mtext>embed</mtext></msub><msub><mi mathvariant="bold">W</mi><mtext>in-proj</mtext></msub></mrow><annotation encoding="application/x-tex"> \textbf{W}_{\text{input}} = \mathbf{W}_{\text{embed}} \mathbf{W}_{\text{in-proj}} </annotation></semantics></math>Winput=WembedWin-proj <math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>output</mtext></msub><mo>=</mo><msub><mtext mathvariant="bold">W</mtext><mtext>out-proj</mtext></msub><msubsup><mtext mathvariant="bold">W</mtext><mtext>embed</mtext><mi mathvariant="normal">⊤</mi></msubsup></mrow><annotation encoding="application/x-tex"> \textbf{W}_{\text{output}} = \textbf{W}_{\text{out-proj}} \textbf{W}_{\text{embed}}^{\top} </annotation></semantics></math>Woutput=Wout-projWembed⊤ We observed that the generalizing solution is sparse after taking the discrete Fourier transformation, but the collapsed matrices have high norms. This suggests that direct weight decay on <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>output</mtext></msub></mrow><annotation encoding="application/x-tex">\textbf{W}_\text{output}</annotation></semantics></math>Woutput and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>input</mtext></msub></mrow><annotation encoding="application/x-tex">\textbf{W}_{\text{input}}</annotation></semantics></math>Winput doesn’t provide the right inductive bias for the task. Broadly speaking, weight decay does steer a wide variety of models away from memorizing their training data <a class='citestart' key='DoubleDescent double-demystified'></a>. Other techniques that help avoid overfitting include dropout, smaller models and even numerically unstable optimization algorithms <a class='citestart' key='Slingshot'></a>. These approaches interact in complex, non-linear ways, making it difficult to predict a priori which will ultimately induce generalization. Collapsing <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>embed</mtext></msub><msub><mi mathvariant="bold">W</mi><mtext>in-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{embed}} \mathbf{W}_{\text{in-proj}}</annotation></semantics></math>WembedWin-proj instead of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>out-proj</mtext></msub><msubsup><mtext mathvariant="bold">W</mtext><mtext>embed</mtext><mi mathvariant="normal">⊤</mi></msubsup></mrow><annotation encoding="application/x-tex">\textbf{W}_{\text{out-proj}} \textbf{W}_{\text{embed}}^{\top}</annotation></semantics></math>Wout-projWembed⊤, for example, helps in some setups and hurts in others: <div class='sweep-mod'></div> <h4 id="why-is-memorization-easier-than-generalization-">Why Is Memorization Easier Than Generalization?</h4> One theory: there can be many more ways to memorize a training set than there are generalizing solutions. So statistically, memorization should be more likely to happen first, especially if we have no or little regularization. Regularization techniques, like weight decay, prioritize certain solutions over others, for example, preferring “sparse” solutions over “dense” ones. Recent work suggests that generalization is associated with well-structured representations <a class='citestart' key='EffectiveTheory'></a>. However, it’s not a necessary condition; some MLP variations without symmetric inputs learn less “circular” representations when solving modular addition <a class='citestart' key='Zhong23'></a>. We also observed that well-structured representations are not a sufficient condition for generalization. This small model (trained with no weight decay) starts generalizing, then switches to memorizing with periodic embeddings. <div class='open-q-mem-0-accuracy row'></div> <div class='open-q-mem-0-weights row'></div> It’s even possible to find hyperparameters where models start generalizing, then switch to memorizing, then switch back to generalizing! <a class='footstart' key='open-q-mem'></a> <div class='open-q-mem-1-accuracy row'></div> <div class='open-q-mem-1-weights row'></div> <h4 id="what-about-larger-models-">What About Larger Models?</h4> Does grokking happen in larger models trained on real world tasks? Earlier observations reported the grokking phenomenon in algorithmic tasks in small transformers and MLPs <a class='citestart' key='Grokking ProgressMeasures Zhong23'></a>. Grokking has subsequently been found in more complex tasks involving images, text, and tabular data within certain ranges of hyperparameters <a class='citestart' key='Omnigrok Goldilocks'></a>. It’s also possible that the largest models, which are able to do many types of tasks, may be grokking many things at different speeds during training <a class='citestart' key='quantization'></a>. There have also been promising results in predicting grokking before it happens. Though some require knowledge of the generalizing solution <a class='citestart' key='ProgressMeasures'></a> or the overall data domain <a class='citestart' key='StructuralGrokking'></a>, some rely solely on the analysis of the training loss <a class='citestart' key='PredictingGrokking'></a> and might also apply to larger models — hopefully we’ll be able to build tools and techniques that can tell us when a model is parroting memorized information and when it’s using richer models. Understanding the solution to modular addition wasn’t trivial. Do we have any hope of understanding larger models? One route forward — like our digression into the 20 parameter model and the even simpler boolean parity problem — is to: 1) train simpler models with more inductive biases and fewer moving parts, 2) use them to explain inscrutable parts of how a larger model works, 3) repeat as needed. We believe this could be a fruitful approach to better understanding larger models, and complementary to efforts that aim to use larger models to explain smaller ones and other work to disentangle internal representations <a class='citestart' key='explain multiple-choice TMOS'></a>. Moreover, this kind of mechanistic approach to interpretability, in time, may help identify patterns that themselves ease or automate the uncovering of algorithms learned by neural networks. <h3 id="credits">Credits</h3> Thanks to Ardavan Saeedi, Crystal Qian, Emily Reif, Fernanda Viégas, Kathy Meier-Hellstern, Mahima Pushkarna, Minsuk Chang, Neel Nanda and Ryan Mullins for their help with this piece. <a href="https://github.com/PAIR-code/ai-explorables/tree/master/server-side/grokking">Model training code</a> // <a href="https://github.com/PAIR-code/ai-explorables/tree/master/source/grokking">Visualization code</a> <h3 id="appendix-a-how-the-circular-construction-works">Appendix A: How the Circular Construction Works</h3> We can almost calculate <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo>+</mo><mi>b</mi><mtext> </mtext><mo lspace="0.22em" rspace="0.22em"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow></mo><mtext> </mtext><mn>67</mn></mrow><annotation encoding="application/x-tex">a + b \bmod 67</annotation></semantics></math>a+bmod67 using two circular embeddings and a completely linear model. <div class='sticky-container'> <div class='slider-container row sticky appendix'></div> <div class='circle-vis row'></div> It works! But we’re cheating a bit, do you see how unembed loops around the circle twice? We need to output a single prediction for “<v></v>“ — not separate predictions for “<v></v>“ and “<v2></v2>“. Directly adding the two predictions for a number together won’t work since they’re on opposite sides of the circles and will cancel each other out. Instead, let’s incorporate a <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>ReLU</mtext><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\text{ReLU}(x)</annotation></semantics></math>ReLU(x) to fix the repeated outputs. <div class='proj-vis row'></div> We’ve essentially wrapped the circle around in on itself and the model outputs a single prediction for “<v></v>“. Formally, this is the constructed model: <script type="math/tex"> $$ \begin{aligned} \text{activations} & = \text{ReLU}\left(\mathbf{a}_{\text{one-hot}} \mathbf{W}_{\text{embed}} \mathbf{W}_{\text{in-proj}} + \mathbf{b}_{\text{one-hot}} \mathbf{W}_{\text{embed}} \mathbf{W}_{\text{in-proj}}\right) \\ \text{logits} & = \text{activations} \mathbf{W}_{\text{out-proj}} \mathbf{W}_{\text{embed}}^{\top} \end{aligned} $$ </script> With modulus <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math>M and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi></mrow><annotation encoding="application/x-tex">N</annotation></semantics></math>N evenly spaced neurons/directions: <script type="math/tex"> $$ \mathbf{W}_{\text{embed}} = \begin{pmatrix} \dots & \ldots \\ \cos(i\frac{2\pi}{M}) & \sin(i \frac{2\pi}{M}) \\ \dots & \dots \\ \end{pmatrix}, \quad \mathbf{W}_{\text{in-proj}}^T = \begin{pmatrix} \dots & \ldots \\ \cos(i\frac{2\pi}{N}) & \sin(i \frac{2\pi}{N}) \\ \dots & \dots \\ \end{pmatrix}, \quad \mathbf{W}_{\text{out-proj}} = \begin{pmatrix} \dots & \dots \\ \cos(2i\frac{2\pi}{N}) & \sin(2i\frac{2\pi}{N}) \\ \dots & \dots \\ \end{pmatrix}. $$ </script> Interestingly this circle has a few wrinkles: this construction doesn’t give an exact answer! </div> <div class='debug-vis row'></div> <div class='appendix num-inputs row'> Neurons <input type="number" class='n_neurons' min="3" max="10" value="5"> Modulus <input type="number" class='modulus' min="12" max="500" value="67"> </div> Using <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>x</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">x^2</annotation></semantics></math>x2 instead of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>ReLU</mtext><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\text{ReLU}(x)</annotation></semantics></math>ReLU(x) as the activation function, as suggested by <a class='citestart' key='gromov'></a> gives a provably exact solution! For simplicity, let <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ω</mi><mo>:</mo><mo>=</mo><mfrac><mrow><mn>2</mn><mi>π</mi></mrow><mi>M</mi></mfrac></mrow><annotation encoding="application/x-tex">\omega:=\frac{2\pi}{M}</annotation></semantics></math>ω:=M2π (the angle between numbers in <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>embed</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{embed}}</annotation></semantics></math>Wembed) and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi><mo>:</mo><mo>=</mo><mfrac><mrow><mn>2</mn><mi>π</mi></mrow><mi>N</mi></mfrac></mrow><annotation encoding="application/x-tex">\theta := \frac{2\pi}{N}</annotation></semantics></math>θ:=N2π (the angle between neurons in <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi mathvariant="bold">W</mi><mtext>in-proj</mtext><mi>T</mi></msubsup></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{in-proj}}^T</annotation></semantics></math>Win-projT). Let’s rewrite <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mtext>logits</mtext><mrow><mi>a</mi><mo separator="true">,</mo><mi>b</mi></mrow></msup></mrow><annotation encoding="application/x-tex">\text{logits}^{a, b}</annotation></semantics></math>logitsa,b as an <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>M</mi></mrow><annotation encoding="application/x-tex">M</annotation></semantics></math>M-dimensional vector <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">∥</mo><mrow><mi>l</mi><msup><mo stretchy="false">∥</mo><mi>M</mi></msup></mrow></mrow><annotation encoding="application/x-tex"> \lVert \it{l} \rVert ^M </annotation></semantics></math>∥l∥M where: <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>l</mi><mi>j</mi></msub><mo>=</mo><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mrow><mi>N</mi><mo>−</mo><mn>1</mn></mrow></msubsup><mo fence="true" stretchy="true" minsize="2.4em" maxsize="2.4em">(</mo><mo fence="true" stretchy="true" minsize="1.2em" maxsize="1.2em">[</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>a</mi><mi>ω</mi><mo>−</mo><mi>i</mi><mi>θ</mi><mo stretchy="false">)</mo><mo>+</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>b</mi><mi>ω</mi><mo>−</mo><mi>i</mi><mi>θ</mi><mo stretchy="false">)</mo><msup><mo fence="true" stretchy="true" minsize="1.2em" maxsize="1.2em">]</mo><mn>2</mn></msup><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>j</mi><mi>ω</mi><mo>−</mo><mn>2</mn><mi>i</mi><mi>θ</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo fence="true" stretchy="true" minsize="2.4em" maxsize="2.4em">)</mo></mrow><annotation encoding="application/x-tex">l_{j} = \sum_{i=0}^{N-1} \biggl(\bigl[ \cos(a\omega-i\theta) + \cos(b\omega-i\theta) \bigl]^2\cos(j\omega-2i\theta)) \biggr) </annotation></semantics></math>lj=∑i=0N−1([cos(aω−iθ)+cos(bω−iθ)]2cos(jω−2iθ))) This follows from the <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mtext>logits</mtext><mrow><mi>a</mi><mo separator="true">,</mo><mi>b</mi></mrow></msup></mrow><annotation encoding="application/x-tex">\text{logits}^{a,b}</annotation></semantics></math>logitsa,b equation above by plugging in the definitions of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>in-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\textbf{W}_\text{in-proj}</annotation></semantics></math>Win-proj and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>out-proj</mtext></msub></mrow><annotation encoding="application/x-tex">\textbf{W}_\text{out-proj}</annotation></semantics></math>Wout-proj and applying the trigonometric identity that <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo>+</mo><mi>sin</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi>sin</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo>=</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>x</mi><mo>−</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\cos(x)\cos(y) + \sin(x)\sin(y) = \cos(x-y)</annotation></semantics></math>cos(x)cos(y)+sin(x)sin(y)=cos(x−y). We can then prove the following: <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi mathvariant="normal">arg max</mi><mo>⁡</mo></mrow><mi>c</mi></msub><mtext> ⁣</mtext><msup><mtext>logits</mtext><mrow><mi>a</mi><mo separator="true">,</mo><mi>b</mi></mrow></msup><mo>=</mo><mi>a</mi><mo>+</mo><mi>b</mi><mtext> </mtext><mo lspace="0.22em" rspace="0.22em"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow></mo><mtext> </mtext><mi>M</mi></mrow><annotation encoding="application/x-tex"> \argmax_c \! \text{logits}^{a,b} = a + b \bmod M </annotation></semantics></math>argmaxclogitsa,b=a+bmodM Applying the two trigonometric identities of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>+</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo>=</mo><mn>2</mn><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mfrac><mrow><mi>x</mi><mo>−</mo><mi>y</mi></mrow><mn>2</mn></mfrac><mo stretchy="false">)</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mfrac><mrow><mi>x</mi><mo>+</mo><mi>y</mi></mrow><mn>2</mn></mfrac><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\cos(x) + \cos(y) = 2 \cos(\frac{x-y}{2}) \cos(\frac{x+y}{2})</annotation></semantics></math>cos(x)+cos(y)=2cos(2x−y)cos(2x+y) and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mrow><mi>cos</mi><mo>⁡</mo></mrow><mn>2</mn></msup><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi>c</mi><mi>o</mi><mi>s</mi><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mi mathvariant="normal">/</mi><mn>4</mn><mo fence="true" stretchy="true" minsize="1.2em" maxsize="1.2em">[</mo><mn>2</mn><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo>+</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mn>2</mn><mi>x</mi><mo>−</mo><mi>y</mi><mo stretchy="false">)</mo><mo>+</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mn>2</mn><mi>x</mi><mo>+</mo><mi>y</mi><mo stretchy="false">)</mo><mo fence="true" stretchy="true" minsize="1.2em" maxsize="1.2em">]</mo></mrow><annotation encoding="application/x-tex">\cos^2(x)cos(y) = 1/4 \bigl[ 2\cos(y) + \cos(2x-y) + \cos (2x+y) \bigl] </annotation></semantics></math>cos2(x)cos(y)=1/4[2cos(y)+cos(2x−y)+cos(2x+y)], we have: <script type="math/tex"> $$ \begin{aligned} \text{logits}^{a, b} & = \sum_{i=0}^{N-1} \biggl(\bigl[ \cos(a\omega-i\theta) + \cos(b\omega-i\theta) \bigl]^2 \cos(c\omega-2i\theta) \biggl) \\ & = \sum_{i=0}^{N-1} \biggl(2 \bigl[ \cos(\frac{a-b}{2}\omega)\cos(\frac{a+b}{2}\omega-i\theta) \bigl] ^2 \cos(c\omega-2i\theta) \biggl) \\ & = \cos^2(\frac{a-b}{2}\omega) \sum_{i=0}^{N-1} \biggl(2\cos(c\omega - 2i\theta) + \cos((a+b-c)\omega) +\cos((a+b+c)\omega-4i\theta) \biggl) \end{aligned} $$ </script> Note that <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>∑</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><msub><mi>γ</mi><mi>i</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\sum \cos(\gamma_{i})=0</annotation></semantics></math>∑cos(γi)=0 where <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>γ</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\gamma_{i}</annotation></semantics></math>γi is equally spread around the circle. The first and the third sum terms wrap around the circle with <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>2</mn><mi>θ</mi></mrow><annotation encoding="application/x-tex">2\theta</annotation></semantics></math>2θ and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>4</mn><mi>θ</mi></mrow><annotation encoding="application/x-tex">4\theta</annotation></semantics></math>4θ increments respectively. The sum of the first terms equals zero for <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>></mo><mn>2</mn></mrow><annotation encoding="application/x-tex">N \gt 2</annotation></semantics></math>N>2 and the sum of the third terms equals zero for <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>></mo><mn>4</mn></mrow><annotation encoding="application/x-tex">N \gt 4</annotation></semantics></math>N>4. Therefore, we have: <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mtext>logits</mtext><mrow><mi>a</mi><mo separator="true">,</mo><mi>b</mi></mrow></msup><mo>=</mo><msup><mrow><mi>cos</mi><mo>⁡</mo></mrow><mn>2</mn></msup><mo stretchy="false">(</mo><mfrac><mrow><mi>a</mi><mo>−</mo><mi>b</mi></mrow><mn>2</mn></mfrac><mi>ω</mi><mo stretchy="false">)</mo><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mo stretchy="false">(</mo><mi>a</mi><mo>+</mo><mi>b</mi><mo>−</mo><mi>c</mi><mo stretchy="false">)</mo><mi>ω</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex"> \text{logits}^{a,b} = \cos^2(\frac{a-b}{2}\omega) \cos((a+b-c)\omega) </annotation></semantics></math>logitsa,b=cos2(2a−bω)cos((a+b−c)ω) Since the first term is a positive constant w.r.t inputs, the equation is maximized when <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mo stretchy="false">(</mo><mi>a</mi><mo>+</mo><mi>b</mi><mo>−</mo><mi>c</mi><mo stretchy="false">)</mo><mi>ω</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\cos((a+b-c)\omega)</annotation></semantics></math>cos((a+b−c)ω) is maximized, which is when <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi><mo>=</mo><mi>a</mi><mo>+</mo><mi>b</mi><mtext> </mtext><mo lspace="0.22em" rspace="0.22em"><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">o</mi><mi mathvariant="normal">d</mi></mrow></mo><mtext> </mtext><mi>M</mi></mrow><annotation encoding="application/x-tex">c = a + b \bmod M</annotation></semantics></math>c=a+bmodM. Essentially <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>ReLU</mtext><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\text{ReLU}(x)</annotation></semantics></math>ReLU(x) activations with weight decay (a very typical model setup) gives the model an inductive bias that’s close enough to the exact generalizing solution of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>x</mi><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">x^2</annotation></semantics></math>x2 activations with a sparse discrete Fourier transform to push in the direction of generalization but not so close that it won’t also learn to fit the training data with memorization. <h3 id="footnotes">Footnotes</h3> <a class='footend' key='modular'></a> In modular addition, we have two input numbers, <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi></mrow><annotation encoding="application/x-tex">a</annotation></semantics></math>a and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi></mrow><annotation encoding="application/x-tex">b</annotation></semantics></math>b, and a modulus <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math>m. We want to find the remainder of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo>+</mo><mi>b</mi></mrow><annotation encoding="application/x-tex">a + b</annotation></semantics></math>a+b when divided by <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>m</mi></mrow><annotation encoding="application/x-tex">m</annotation></semantics></math>m. This type of addition is often called clock-face addition, because when adding two times, we often report the result modulo 12 (i.e. 5 hours after 8 o’clock is 1 o’clock). Modular addition sounds simple and it is. We can easily train 1,000s of models and treat them like fruit flies in neuroscience: small enough such that it is feasible to extract their <a href="https://www.science.org/doi/abs/10.1126/science.add9330">connectome</a> synapse-by-synapse, yet providing new interesting insights about the system more broadly. We can get a good understanding of the small models we’ve trained by visualizing all their internals. <a class='footend' key='67'></a>67 isn’t a magic number – we could pick many numbers to illustrate grokking, but 67 is not so small that the task is trivial and also not so large that the visualizations are overwhelming. <a class='footend' key='playground'></a> The model is trained with cross-entropy loss, AdamW and full batches. The <a href="#which-model-constraints-work-best-">section on regularization</a> and <a href="https://colab.research.google.com/github/PAIR-code/ai-explorables/blob/master/server-side/grokking/MLP_Modular_Addition.ipynb">training colab</a> have additional details. If you’re not familiar with <a href="https://en.wikipedia.org/wiki/Multilayer_perceptron">MLPs</a>, <a href="http://playground.tensorflow.org">playground.tensorflow.org</a> is a great place to start. A quick notation explanation: The columns of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>input</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{input}}</annotation></semantics></math>Winput and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>ouput</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{ouput}}</annotation></semantics></math>Wouput represent the numbers from 0 to 66. <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">a</mi><mtext>one-hot</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{a}_{\text{one-hot}}</annotation></semantics></math>aone-hot and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">b</mi><mtext>one-hot</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{b}_{\text{one-hot}}</annotation></semantics></math>bone-hot are how we <a href="https://en.wikipedia.org/wiki/One-hot">encode</a> the model’s inputs; each pick a single column from <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>input</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{input}}</annotation></semantics></math>Winput. <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>ReLU</mtext></mrow><annotation encoding="application/x-tex">\text{ReLU}</annotation></semantics></math>ReLU replaces negative numbers with 0s; it is a <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks">fancy</a>) way of writing <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>max</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\max(x, 0)</annotation></semantics></math>max(x,0). <a class='footend' key='sp-model'></a> With a small twist — we’re only outputting 1 or 0, so <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>output</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{output}}</annotation></semantics></math>Woutput can be a single column. In the modular addition task we needed a column for every output number. The last column of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">W</mi><mtext>input</mtext></msub></mrow><annotation encoding="application/x-tex">\mathbf{W}_{\text{input}}</annotation></semantics></math>Winput is also fixed to 1 to provide a <a href="https://stackoverflow.com/questions/2480650/what-is-the-role-of-the-bias-in-neural-networks">bias term</a>. <a class='footend' key='sp-solution'></a><a href="https://arxiv.org/pdf/2303.11873.pdf#page=8">Appendix D</a> of “A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks” has an explanation of the 4 neuron solution generalizing solution here <a class='footend' key='loss'></a> So far we’ve been charting <a href="https://developers.google.com/machine-learning/crash-course/classification/accuracy">accuracy</a>, the percentage of sequences where the correct label is the most likely. Training typically instead optimizes a differentiable objective function. All the models in this post use <a href="https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html">cross entropy loss</a> which heavily penalizes incorrect predictions with high probabilities. Note that while some formulations of loss include a weight decay or regularization term, the loss plots here depict the cross entropy component alone. <a class='footend' key='sp-l2'></a> On the 1s and 0s task here, we use L1 weight decay <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mn>1</mn><mo stretchy="false">(</mo><mi mathvariant="bold">w</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mo>∑</mo><mi>i</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>w</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">L1(\mathbf{w}) = \sum_{i} |w_i|</annotation></semantics></math>L1(w)=∑i∣wi∣. L2 weight decay <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mn>2</mn><mo stretchy="false">(</mo><mi mathvariant="bold">w</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mo>∑</mo><mi>i</mi></msub><msubsup><mi>w</mi><mi>i</mi><mn>2</mn></msubsup></mrow><annotation encoding="application/x-tex">L2(\mathbf{w}) = \sum_{i} w_i^2</annotation></semantics></math>L2(w)=∑iwi2 is a more typical choice. It pushes for <a href="https://explained.ai/regularization/L1vsL2.html">lots of small weights</a> leading to redundant neurons on this task: <img src='img/sp-l2.gif'></img> <a class='footend' key='overfit'></a> A model overfits the training data when it performs well on the training data but poorly on the test data — this is what we see with our memorizing models. In general, simpler models are less prone to overfitting as, due to their simplicity, decision rules are coarser and are required to make more generalizations. Of course, if a model is too simple for a task, it may not be able to learn good decision rules that capture the nuances of the task. Researchers force models to be simpler through a variety of techniques, including having models with fewer parameters or encouraging the parameters that the model does have to be small in size with weight decay. <a class='footend' key='unit-circle'></a> Computing <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>cos</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>i</mi><mfrac><mrow><mn>2</mn><mi>π</mi></mrow><mn>67</mn></mfrac><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\cos(i\frac{2\pi}{67})</annotation></semantics></math>cos(i672π) and <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>sin</mi><mo>⁡</mo><mo stretchy="false">(</mo><mi>i</mi><mfrac><mrow><mn>2</mn><mi>π</mi></mrow><mn>67</mn></mfrac><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\sin (i\frac{2\pi}{67})</annotation></semantics></math>sin(i672π) gives us points evenly spaced around the unit circle. Here’s what <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>embed</mtext></msub></mrow><annotation encoding="application/x-tex">\textbf{W}_{\text{embed}}</annotation></semantics></math>Wembed looks like on the unit circle: <img src='img/w_embed.png' width=319></img> <a class='footend' key='dft'></a> The <a href="https://www.youtube.com/watch?v=spUNpyF58BY">Discrete Fourier Transform</a> helps analyze the periodic nature of a sequence of values (in this case the <a href="https://colab.research.google.com/drive/1F6_1_cWXE5M7WocUcpQWp3v8z4b1jL20#scrollTo=iSPxi3ElsujY">weights for a particular neuron</a>) by breaking it down into sine and cosine functions. The more periodic a function is, the easier it is to represent with sine and cosines, and the sparser the output of the DFT. <a class='footend' key='dft-sort'></a> We’ve reindexed the neurons by their final frequency and phase to make this grouping easier to see . <a class='footend' key='logit-wave'></a> The model generates probabilities by taking the dot product of the neuron activations for a given input with <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>output</mtext></msub></mrow><annotation encoding="application/x-tex">\textbf{W}_{\text{output}}</annotation></semantics></math>Woutput and softmaxing. If we calculate the dot product using only the activations from neurons of a single frequency, we can see which outputs the frequency group is making more or less likely. <a href="#appendix-a-how-the-circular-construction-works">Appendix A</a> explains why these logits form a wave — each group of frequencies is essentially outputting how close the correct answer is to every number on a version of <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mtext mathvariant="bold">W</mtext><mtext>embed</mtext></msub></mrow><annotation encoding="application/x-tex">\textbf{W}_{\text{embed}}</annotation></semantics></math>Wembed with the group’s frequency. <a class='footend' key='open-q-mem'></a>Both of these models are <a href="https://colab.sandbox.google.com/github/PAIR-code/ai-explorables/blob/master/server-side/grokking/MLP_Modular_Addition.ipynb#scrollTo=5hJqK4jx0vC7">quite small</a>. The bottom model has tweaked hyperparameters to encourage eventual generalization: it’s slightly larger to allow it to exit local minimums, it has more training data (making low loss memorizing solutions harder to find) and it has weight decay. <h3 id="references">References</h3> <a class='citeend' key='Grokking'></a> <a href="https://arxiv.org/pdf/2201.02177.pdf">Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets</a> Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). arXiv preprint arXiv:2201.02177. <a class='citeend' key='Omnigrok'></a> <a href="https://arxiv.org/pdf/2210.01117.pdf">Omnigrok: Grokking Beyond Algorithmic Data</a> Liu, Z., Michaud, E. J., & Tegmark, M. (2022, September). In The Eleventh International Conference on Learning Representations. <a class='citeend' key='Universality'></a> <a href="https://arxiv.org/abs/2302.03025">A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations</a> Chughtai, B., Chan, L., Nanda, N. (2023). International Conference on Machine Learning. <a class='citeend' key='Zhong23'></a><a href="https://arxiv.org/pdf/2306.17844.pdf">The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks</a> Zhong, Z., Liu, Z., Tegmark, M., & Andreas, J. (2023). arXiv preprint arXiv:2306.17844. <a class='citeend' key='ProgressParity'></a> <a href="https://arxiv.org/abs/2207.08799">Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit</a> Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, Cyril Zhang. (2022) Advances in Neural Information Processing Systems, 35, 21750-21764. <a class='citeend' key='gromov'></a><a href="https://arxiv.org/abs/2301.02679">Grokking modular arithmetic</a> Andrey Gromov (2023). arXiv preprint arXiv:2301.02679. <a class='citeend' key='Parrots'></a><a href="https://dl.acm.org/doi/pdf/10.1145/3442188.3445922?uuid=f2qngt2LcFCbgtaZ2024">On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?🦜</a> Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623). <a class='citeend' key='Othello'></a> <a href="https://openreview.net/pdf?id=DeG07_TcZvT">Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task</a> Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2022, September). In The Eleventh International Conference on Learning Representations. <a class='citeend' key='MechInterp'></a> <a href="https://transformer-circuits.pub/2022/mech-interp-essay/index.html">Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases</a> Olah, C., 2022. Transformer Circuits Thread. <a class='citeend' key='ProgressMeasures'></a> <a href="https://openreview.net/pdf?id=9XFSbDPmdW">Progress Measures for Grokking via Mechanistic Interpretability</a> Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2022, September). In The Eleventh International Conference on Learning Representations. <a class='citeend' key='TwoCircuits'></a> <a href="https://arxiv.org/abs/2303.11873">A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks</a> William Merrill, Nikolaos Tsilivis, Aman Shukla. (2023). arXiv preprint arXiv:2303.11873. <a class='citeend' key='DoubleDescent'></a><a href="https://arxiv.org/pdf/2303.06173.pdf">Unifying Grokking and Double Descent</a> Davies, X., Langosco, L., & Krueger, D. (2022, November). In NeurIPS ML Safety Workshop. <a class='citeend' key='double-demystified'></a><a href="https://arxiv.org/abs/2303.14151">Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle</a> Rylan Schaeffer, R., Khona, M., Robertson, Z., Boopathy, A., Pistunova, K., Rocks, J., Rani Fiete, I., & Koyejo, O. (2023). arXiv preprint arXiv:2303.14151. <a class='citeend' key='Slingshot'></a> <a href="https://arxiv.org/pdf/2206.04817.pdf">The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon</a> Thilak, V., Littwin, E., Zhai, S., Saremi, O., Paiss, R., & Susskind, J. (2022). arXiv preprint arXiv:2206.04817. <a class='citeend' key='EffectiveTheory'></a><a href="https://arxiv.org/pdf/2205.10343.pdf">Towards Understanding Grokking: An Effective Theory of Representation Learning</a> Liu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark, M., & Williams, M. (2022). Advances in Neural Information Processing Systems, 35, 34651-34663. <a class='citeend' key='Goldilocks'></a><a href="https://arxiv.org/pdf/1807.02581.pdf">The Goldilocks Zone: Towards Better Understanding of Neural Network Loss Landscapes</a> Fort, S., & Scherlis, A. (2019, July). In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 3574-3581). <a class='citeend' key='quantization'></a><a href="https://arxiv.org/abs/2303.13506">The Quantization Model of Neural Scaling</a> Eric J. Michaud, Ziming Liu, Uzay Girit, Max Tegmark, O. (2023). arXiv preprint arXiv:2303.13506. <a class='citeend' key='StructuralGrokking'></a> <a href="https://arxiv.org/pdf/2305.18741.pdf">Grokking of Hierarchical Structure in Vanilla Transformers</a> Murty, S., Sharma, P., Andreas, J., & Manning, C. D. (2023). arXiv preprint arXiv:2305.18741. <a class='citeend' key='PredictingGrokking'></a> <a href="https://arxiv.org/pdf/2306.13253.pdf">Predicting Grokking Long Before it Happens: A Look Into the Loss Landscape of Models Which Grok</a> Notsawo Jr, P., Zhou, H., Pezeshki, M., Rish, I., & Dumas, G. (2023). arXiv preprint arXiv:2306.13253. <a class='citeend' key='explain'></a><a href="https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html">Language models can explain neurons in language models</a> Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., & Saunders, W. 2023. OpenAI Blog <a class='citeend' key='multiple-choice'></a><a href="https://arxiv.org/abs/2307.09458">Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla</a> Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, Vladimir Mikulik (2023). arXiv preprint arXiv:2307.09458. <a class='citeend' key='TMOS'></a><a href="https://transformer-circuits.pub/2022/toy_model/index.html">Toy Models of Superposition</a> Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M. and Olah, C., 2022. Transformer Circuits Thread. <a class='citeend' key='Connectome'></a> <a href="https://www.science.org/doi/abs/10.1126/science.add9330">The Connectome of an Insect Brain</a> Winding, M., Pedigo, B. D., Barnes, C. L., Patsolic, H. G., Park, Y., Kazimiers, T., … & Zlatic, M. (2023). Science, 379(6636), eadd9330. <a class='citeend' key='Multiscale'></a> <a href="https://proceedings.mlr.press/v162/pezeshki22a/pezeshki22a.pdf">Multi-Scale Feature Learning Dynamics: Insights for Double Descent</a> Pezeshki, M., Mitra, A., Bengio, Y., & Lajoie, G. (2022, June). In the International Conference on Machine Learning (pp. 17669-17690). PMLR. <a class='citeend' key='superposition'></a><a href="https://transformer-circuits.pub/2023/toy-double-descent/index.html">Superposition, Memorization, and Double Descent</a> Henighan, T., Carter, S., Hume, T., Elhage, N., Lasenby, R., Fort, S., Schiefer, N., and Olah, C., 2023. Transformer Circuits Thread. <h3 id="more-explorables">More Explorables</h3> <div class='recirc-feedback-form'></div> <link rel='stylesheet' href='../third_party/footnote_v2.css'> <link rel='stylesheet' href='../third_party/citation_v2.css'> <link rel='stylesheet' href='style.css'> <script id='MathJax-script' async src='https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js'></script> <script defer src='https://cdn.jsdelivr.net/npm/katex@0.16.8/dist/contrib/mathtex-script-type.min.js' integrity='sha384-jiBVvJ8NGGj5n7kJaiWwWp9AjC+Yh8rhZY3GtAX8yU28azcLgoRo4oukO87g7zDT' crossorigin='anonymous'></script> <script src='../third_party/d3_.js'></script> <script src='../third_party/d3-scale-chromatic.v1.min.js'></script> <script src='../third_party/tfjsv3.18.0.js'></script> <script src='../third_party/npyjs-global.js'></script> <script src='../third_party/swoopy-drag.js'></script> <script src='../third_party/footnote_v2.js'></script> <script src='../third_party/citation_v2.js'></script> <script src='util.js'></script> <script src='init-accuracy-chart.js'></script> <script src='init-animate-steps.js'></script> <script src='init-embed-vis.js'></script> <script src='init-input-sliders.js'></script> <script src='init-swoopy.js'></script> <link rel='stylesheet' href='mod-top/style.css'> <script src='mod-top/init-waves.js'></script> <script src='mod-top/init.js'></script> <link rel='stylesheet' href='sparse-parity/style.css'> <script src='sparse-parity/init.js'></script> <script src='sparse-parity/init-weight-trajectory.js'></script> <link rel='stylesheet' href='sweep-sparse-parity/style.css'> <script defer src='sweep-sparse-parity/init.js'></script> <link rel='stylesheet' href='sweep-mod/style.css'> <script defer src='sweep-mod/sweep-mod-charts.js'></script> <script defer src='sweep-mod/init.js'></script> <link rel='stylesheet' href='hand-weights/style.css'> <link rel='stylesheet' href='hand-weights/sliders.css'> <script src='hand-weights/init-embed-vis.js'></script> <script src='hand-weights/init-activation-vis.js'></script> <script src='hand-weights/init-circle-weights-vis.js'></script> <script src='hand-weights/init-circle-input-vis.js'></script> <script src='hand-weights/init-circle-weights-freq.js'></script> <script defer src='hand-weights/init.js'></script> <link rel='stylesheet' href='five-neurons/style.css'> <script src='five-neurons/five-circle.js'></script> <script defer src='five-neurons/init.js'></script> <link rel='stylesheet' href='mod-bot/style.css'> <script src='mod-bot/init-bot-freqs.js'></script> <script src='mod-bot/init-bot-logits.js'></script> <script src='mod-bot/seeds/init-seeds.js'></script> <script defer src='mod-bot/init.js'></script> <script src='open-q-mem/init-0.js'></script> <script src='open-q-mem/init-1.js'></script> <script defer src='circle-freq/circle-freq-init.js'></script> <link rel='stylesheet' href='appendix/style.css'> <script src='appendix/debug-vis.js'></script> <script src='appendix/init-circle-weights-vis.js'></script> <script src='appendix/init-proj-vis.js'></script> <script src='appendix/init-sliders.js'></script> <script src='appendix/line-error-vis.js'></script> <script src='appendix/debug-reuleaux.js'></script> <script defer src='appendix/init.js'></script> <script defer src='../third_party/recirc.js'></script> </body> <script async src="https://www.googletagmanager.com/gtag/js?id=UA-138505774-1"></script> <script> if (window.location.origin === 'https://pair.withgoogle.com'){ window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-138505774-1'); } </script> <script> // Tweaks for displaying in an iframe if (window !== window.parent){ // Open links in a new tab Array.from(document.querySelectorAll('a')) .forEach(e => { // skip anchor links if (e.href && e.href[0] == '#') return e.setAttribute('target', '_blank') e.setAttribute('rel', 'noopener noreferrer') }) // Remove recirc h3 Array.from(document.querySelectorAll('h3')) .forEach(e => { if (e.textContent != 'More Explorables') return e.parentNode.removeChild(e) }) // Remove recirc container var recircEl = document.querySelector('#recirc') recircEl.parentNode.removeChild(recircEl) } </script> </html>