Research

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Research</title> <meta name="description" content="Resources to learn about Magenta research">  <meta property="og:image" content="https://magenta.tensorflow.org/assets/magenta-logo-card.jpg"> <meta property="og:title" content="Research"> <meta property="og:description" content="Resources to learn about Magenta research"> <meta property="og:url" content="https://magenta.tensorflow.org/research/"> <meta property="og:site_name" content="Magenta">  <meta name="twitter:card" content="summary"> <meta name="twitter:title" content="Research"> <meta name="twitter:description" content="Resources to learn about Magenta research"> <meta name="twitter:image" content="https://magenta.tensorflow.org/assets/magenta-logo-card.jpg"> <link rel="stylesheet" href="/css/main.css"> <link rel="canonical" href="https://magenta.tensorflow.org/research/"> <link rel="alternate" type="application/rss+xml" title="Magenta" href="https://magenta.tensorflow.org/feed.xml"> <link href="https://fonts.googleapis.com/css?family=Google+Sans:+400,500,700" media="all" rel="stylesheet"> <script src="//www.google.com/js/gweb/analytics/autotrack.js"></script> <script> new gweb.analytics.AutoTrack({ profile: 'UA-80107903-1' }); </script> </head> <body> <div class="scrim" onclick="document.body.classList.toggle('drawer-opened', false)"></div> <header> <div class="top-bar background"> <div class="top-bar-content"> <div class="logo"> <a href="/"><img src="/assets/magenta-logo.png" height="70" alt="magenta logo"></a> </div> <nav> <button class="menu-button" onclick="document.body.classList.toggle('drawer-opened', true)" aria-label="open nav menu"> <svg viewBox="0 0 18 15"> <path fill="#424242" d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.031C17.335,0,18,0.665,18,1.484L18,1.484z"/> <path fill="#424242" d="M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0c0-0.82,0.665-1.484,1.484-1.484 h15.031C17.335,6.031,18,6.696,18,7.516L18,7.516z"/> <path fill="#424242" d="M18,13.516C18,14.335,17.335,15,16.516,15H1.484C0.665,15,0,14.335,0,13.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.031C17.335,12.031,18,12.696,18,13.516L18,13.516z"/> </svg> </button> <div class="links"> <a href="/get-started">Get Started</a> <a href="/studio">Studio</a> <a href="/ddsp-vst">DDSP-VST</a> <a href="/demos">Demos</a> <a href="/blog">Blog</a> <a href="/research">Research</a> <a href="/talks">Talks</a> <a href="/community">Community</a> </div> </nav> </div> </div> </header> <div class="drawer"> <div class="drawer-content"> <a href="/get-started">Get Started</a> <a href="/studio">Studio</a> <a href="/ddsp-vst">DDSP-VST</a> <a href="/demos">Demos</a> <a href="/blog">Blog</a> <a href="/research">Research</a> <a href="/talks">Talks</a> <a href="/community">Community</a> </div> </div> <div class="main"> <section class="white"> <div class="content single"> <h1>馃摎 Full list of publications</h1> <p>Here is a link to <a href="https://ai.google/research/pubs/?collection=magenta">all of our research publications</a>.</p> </div> </section> <section class="grey"> <div class="content single"> <h1>馃捑 Datasets</h1> <p>As part of the project, we open source some of the <a href="/datasets">datasets</a> that were used in our research. </p> </div> </section> <section class="alternate research"> <div class="content single"> <h1>馃攷 Research highlights</h1> <br> <h2 class="section-title">DDSP</h2> <p><p>A library that lets you combine the interpretable structure of classical DSP elements (such as filters, oscillators, reverberation, etc.) with the expressivity of deep learning.</p> </p> <img class="project-image" src="/assets/ddsp/ddsp_cat_jamming.png" alt="overview of DDSP">  <h3>Papers</h3> <ul> <li><a href="https://openreview.net/forum?id=B1x1ma4tDr">DDSP: Differentiable Digital Signal Processing</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/ddsp"> DDSP: Differentiable Digital Signal Processing</a></li> <li><a href="/tone-transfer"> Tone Transfer</a></li> <li><a href="/transcultural"> Stepping Towards Transcultural Machine Learning in Music</a></li> </ul>  <h3>Colab Notebooks</h3> <ul> <li><a href="http://goo.gl/magenta/ddsp-demo"> DDSP Timbre Transfer</a></li> </ul> </div> </section> <section class="alternate research"> <div class="content single no-padding-bottom"> <h2 class="section-title">GANSynth</h2> <p><p>A method to synthesize high-fidelity audio with GANs.</p> </p> </div> <div class="content double center"> <div> <img class="project-image" src="/assets/gansynth/coherence.png" alt="overview of GANSynth"> </div> <div>  <h3>Papers</h3> <ul> <li><a href="http://goo.gl/magenta/gansynth-paper">GANSynth: Adversarial Neural Audio Synthesis</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/gansynth"> GANSynth: Making music with GANs</a></li> </ul>  <h3>Colab Notebooks</h3> <ul> <li><a href="http://goo.gl/magenta/gansynth-demo"> GANSynth</a></li> </ul> </div> </div> </section> <section class="alternate research"> <div class="content single"> <h2 class="section-title">Music Transformer</h2> <p><p>A self-attention-based neural network that can generate music with long-term coherence.</p> </p> <img class="project-image" src="/assets/music_transformer/overview.png" alt="overview of Music Transformer">  <h3>Papers</h3> <ul> <li><a href="https://arxiv.org/abs/1809.04281">Music Transformer: Generating Music with Long-Term Structure</a></li> <li><a href="https://ai.google/research/pubs/pub48631">Visualizing Music Self-Attention</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/music-transformer"> Music Transformer: Generating Music with Long-Term Structure</a></li> <li><a href="/piano-transformer"> Generating Piano Music with Transformer</a></li> <li><a href="/transformer-autoencoder"> Encoding Musical Style with Transformer Autoencoders</a></li> <li><a href="/listen-to-transformer"> Listen to Transformer</a></li> <li><a href="/nobodys-songs"> Making an Album with Music Transformer</a></li> </ul>  <h3>Colab Notebooks</h3> <ul> <li><a href="https://colab.research.google.com/notebooks/magenta/piano_transformer/piano_transformer.ipynb"> Music Transformer</a></li> </ul> </div> </section> <section class="alternate research"> <div class="content single no-padding-bottom"> <h2 class="section-title">Wave2Midi2Wave</h2> <p><p>A new process able to transcribe, compose, and synthesize audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s).</p> </p> </div> <div class="content double center"> <div> <img class="project-image" src="/assets/maestro/MAESTRO_models_diagram.png" alt="overview of Wave2Midi2Wave"> </div> <div>  <h3>Papers</h3> <ul> <li><a href="https://arxiv.org/abs/1810.12247">Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/maestro-wave2midi2wave"> The MAESTRO Dataset and Wave2Midi2Wave</a></li> </ul>  </div> </div> </section> <section class="alternate research"> <div class="content single no-padding-bottom"> <h2 class="section-title">Music VAE</h2> <p><p>A hierarchical latent vector model for learning long-term structure in music</p> </p> </div> <div class="content double center"> <div> <img class="project-image" src="/assets/research_highlights/musicvae.png" alt="overview of Music VAE"> </div> <div>  <h3>Papers</h3> <ul> <li><a href="https://ai.google/research/pubs/pub47078">A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music</a></li> <li><a href="https://arxiv.org/abs/1806.00195">Learning a Latent Space of Multitrack Measures</a></li> <li><a href="https://ai.google/research/pubs/pub48628">MidiMe: Personalizing a MusicVAE model with user data</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/music-vae"> MusicVAE: Creating a palette for musical scores with machine learning.</a></li> <li><a href="/multitrack"> Multitrack MusicVAE: Interactively Exploring Musical Styles</a></li> <li><a href="/chain-tripping"> YACHT's new album is powered by ML + Artists</a></li> </ul>  <h3>Colab Notebooks</h3> <ul> <li><a href="https://g.co/magenta/multitrack-musicvae-colab"> Multitrack MusicVAE</a></li> <li><a href="https://g.co/magenta/musicvae-colab"> MusicVAE</a></li> </ul> </div> </div> </section> <section class="alternate research"> <div class="content single no-padding-bottom"> <h2 class="section-title">Onsets and Frames</h2> <p><p>We advance the state of the art in polyphonic piano music transcription by using a deep convolutional and recurrent neural network which is trained to jointly predict onsets and frames.</p> </p> </div> <div class="content double center"> <div> <img class="project-image" src="/assets/research_highlights/onsetsandframes.png" alt="overview of Onsets and Frames"> </div> <div>  <h3>Papers</h3> <ul> <li><a href="https://ai.google/research/pubs/pub46812">Onsets and Frames: Dual-Objective Piano Transcription</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/onsets-frames"> Onsets and Frames: Dual-Objective Piano Transcription</a></li> <li><a href="/oaf-js"> Piano Transcription in the Browser with Onsets and Frames</a></li> <li><a href="/oaf-drums"> Improving Perceptual Quality of Drum Transcription with the Expanded Groove MIDI Dataset</a></li> </ul>  <h3>Colab Notebooks</h3> <ul> <li><a href="https://colab.research.google.com/notebook#fileId=/v2/external/notebooks/magenta/onsets_frames_transcription/onsets_frames_transcription.ipynb"> Onsets and Frames</a></li> </ul> </div> </div> </section> <section class="alternate research"> <div class="content single no-padding-bottom"> <h2 class="section-title">Latent Constraints</h2> <p><p>A method to condition generation without retraining the model, by post-hoc learning latent constraints, value functions that identify regions in latent space that generate outputs with desired attributes. We can conditionally sample from these regions with gradient-based optimization or amortized actor functions.</p> </p> </div> <div class="content double center"> <div> <img class="project-image" src="/assets/research_highlights/latentconstraints.png" alt="overview of Latent Constraints"> </div> <div>  <h3>Papers</h3> <ul> <li><a href="https://ai.google/research/pubs/pub46648">Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models</a></li> <li><a href="https://ai.google/research/pubs/pub46827">Learning via social awareness: improving sketch representations with facial feedback</a></li> <li><a href="https://ai.google/research/pubs/pub48628">MidiMe: Personalizing a MusicVAE model with user data</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/midi-me"> MidiMe: Personalizing MusicVAE</a></li> </ul>  <h3>Colab Notebooks</h3> <ul> <li><a href="https://colab.research.google.com/notebooks/latent_constraints/latentconstraints.ipynb"> Latent Constraints</a></li> </ul> </div> </div> </section> <section class="alternate research"> <div class="content single no-padding-bottom"> <h2 class="section-title">COCONET</h2> <p><p>An instance of orderlessNADE, Coconet uses deep convolutional neural networks to perform music inpaintings through Gibbs sampling.</p> </p> </div> <div class="content double center"> <div> <img class="project-image" src="/assets/research_highlights/coconet.png" alt="overview of COCONET"> </div> <div>  <h3>Papers</h3> <ul> <li><a href="https://ai.google/research/pubs/pub46746">Counterpoint by Convolution</a></li> <li><a href="https://goo.gl/magenta/bach-doodle-paper">The Bach Doodle: Approachable music composition with machine learning at scale</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/coconet"> Coconet: the ML model behind today鈥檚 Bach Doodle</a></li> <li><a href="/bach-doodle-viz"> Visualizing the Bach Doodle Dataset</a></li> </ul>  </div> </div> </section> <section class="alternate research"> <div class="content single no-padding-bottom"> <h2 class="section-title">Performance RNN</h2> <p><p>An LSTM-based recurrent neural network designed to model polyphonic music with expressive timing and dynamics.</p> </p> </div> <div class="content double center"> <div> <img class="project-image" src="/assets/research_highlights/performancernn.png" alt="overview of Performance RNN"> </div> <div>  <h3>Papers</h3> <ul> <li><a href="https://ai.google/research/pubs/pub46748">Learning to Create Piano Performances</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/performance-rnn"> Performance RNN: Generating Music with Expressive Timing and Dynamics</a></li> <li><a href="/performance-rnn-browser"> Real-time Performance RNN in the Browser</a></li> </ul>  <h3>Colab Notebooks</h3> <ul> <li><a href="https://colab.sandbox.google.com/notebooks/magenta/performance_rnn/performance_rnn.ipynb"> Performance RNN</a></li> </ul> </div> </div> </section> <section class="alternate research"> <div class="content single"> <h2 class="section-title">Sketch RNN</h2> <p><p>A recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes.</p> </p> <img class="project-image" src="/assets/research_highlights/sketchrnn.png" alt="overview of Sketch RNN">  <h3>Papers</h3> <ul> <li><a href="https://ai.google/research/pubs/pub46008">A Neural Representation of Sketch Drawings</a></li> <li><a href="https://ai.google/research/pubs/pub48299">collabdraw: An environment for collaborative sketching with an artificial agent</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/sketch_rnn"> SketchRNN model released in Magenta</a></li> <li><a href="/sketch-rnn-demo"> Draw Together with a Neural Network</a></li> </ul>  </div> </section> <section class="alternate research"> <div class="content single"> <h2 class="section-title">NSynth</h2> <p><p>A powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform.</p> </p> <img class="project-image" src="/assets/research_highlights/nsynth.png" alt="overview of NSynth">  <h3>Papers</h3> <ul> <li><a href="https://ai.google/research/pubs/pub46119">Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders</a></li> </ul>  <h3>Blog Posts</h3> <ul> <li><a href="/nsynth"> NSynth: Neural Audio Synthesis</a></li> <li><a href="/nsynth-instrument"> Making a Neural Synthesizer Instrument</a></li> <li><a href="/nsynth-fastgen"> Generate your own sounds with NSynth</a></li> <li><a href="/blog/2017/09/12/outside-hacks/"> Using NSynth to win the Outside Hacks Music Hackathon 2017</a></li> <li><a href="/nsynth-super"> Hands on, with NSynth Super</a></li> </ul>  <h3>Colab Notebooks</h3> <ul> <li><a href="https://g.co/magenta/nsynth-colab"> E-Z NSynth</a></li> </ul> </div> </section> </div> <footer> <div class="footer-content"> <div class="logo"> <a href="https://ai.google/" target="_blank" rel="noopener" title="Google AI"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 157.2 40.88"><defs><style>.cls-1{fill:none;}.cls-2{fill:#80868b;}.cls-3{fill:#80868b;}.cls-4{fill:#80868b;}.cls-5{fill:#80868b;}.cls-6{fill:#80868b;}</style></defs><g id="Running_copy" data-name="Running copy"><path class="cls-1" d="M82.91,16.35A4.8,4.8,0,0,0,79.29,18a5.66,5.66,0,0,0-1.49,4,5.53,5.53,0,0,0,1.49,3.94,4.78,4.78,0,0,0,3.62,1.58,4.47,4.47,0,0,0,3.49-1.58A5.7,5.7,0,0,0,87.81,22a5.84,5.84,0,0,0-1.41-4A4.48,4.48,0,0,0,82.91,16.35Z"></path><path class="cls-1" d="M42.8,16.35a4.92,4.92,0,0,0-3.66,1.57,5.49,5.49,0,0,0-1.51,4,5.52,5.52,0,0,0,1.52,4,5,5,0,0,0,7.3,0,5.48,5.48,0,0,0,1.53-4,5.49,5.49,0,0,0-1.51-4A4.93,4.93,0,0,0,42.8,16.35Z"></path><path class="cls-1" d="M62.89,16.35a4.93,4.93,0,0,0-3.67,1.57,5.53,5.53,0,0,0-1.51,4,5.48,5.48,0,0,0,1.53,4,5,5,0,0,0,7.3,0,5.48,5.48,0,0,0,1.53-4,5.49,5.49,0,0,0-1.51-4A4.93,4.93,0,0,0,62.89,16.35Z"></path><path class="cls-1" d="M111,16.82a4.15,4.15,0,0,0-2.12-.54,4.79,4.79,0,0,0-3.32,1.46,4.9,4.9,0,0,0-1.47,3.9l8.2-3.41A2.82,2.82,0,0,0,111,16.82Z"></path><rect class="cls-2" x="94.13" y="3.56" width="4.03" height="26.97"></rect><path class="cls-3" d="M42.8,12.74a9,9,0,0,0-6.53,2.62,8.83,8.83,0,0,0-2.68,6.55,8.84,8.84,0,0,0,2.68,6.56,9.46,9.46,0,0,0,13.07,0A8.83,8.83,0,0,0,52,21.91a8.82,8.82,0,0,0-2.67-6.55A9,9,0,0,0,42.8,12.74Zm3.65,13.15a5,5,0,0,1-7.3,0,5.52,5.52,0,0,1-1.52-4,5.49,5.49,0,0,1,1.51-4,5.06,5.06,0,0,1,7.33,0,5.49,5.49,0,0,1,1.51,4A5.48,5.48,0,0,1,46.45,25.89Z"></path><path class="cls-4" d="M18.89,15.55v3.9h9.32a8.27,8.27,0,0,1-2.12,4.9,9.76,9.76,0,0,1-7.2,2.85,9.75,9.75,0,0,1-7.24-3,10.07,10.07,0,0,1-3-7.33,10.07,10.07,0,0,1,3-7.33,9.75,9.75,0,0,1,7.24-3,9.89,9.89,0,0,1,7,2.78l2.75-2.74a13.63,13.63,0,0,0-9.77-3.93A14.07,14.07,0,0,0,8.71,6.78,13.58,13.58,0,0,0,4.44,16.84,13.56,13.56,0,0,0,8.71,26.9a14.07,14.07,0,0,0,10.18,4.19,13.12,13.12,0,0,0,9.94-4q3.38-3.36,3.37-9.1a12.59,12.59,0,0,0-.2-2.44Z"></path><path class="cls-4" d="M87.53,14.79h-.14a5.64,5.64,0,0,0-2-1.46,6.66,6.66,0,0,0-2.83-.59,8.37,8.37,0,0,0-6.15,2.69A9,9,0,0,0,73.77,22a8.86,8.86,0,0,0,2.64,6.46,8.36,8.36,0,0,0,6.15,2.68A5.87,5.87,0,0,0,87.39,29h.14v1.32a5.63,5.63,0,0,1-1.3,4,4.69,4.69,0,0,1-3.6,1.39,4.34,4.34,0,0,1-2.88-1A5.94,5.94,0,0,1,78,32.44L74.5,33.9a9.43,9.43,0,0,0,3,3.79,8.07,8.07,0,0,0,5.14,1.64,8.61,8.61,0,0,0,6.27-2.39c1.64-1.58,2.45-4,2.45-7.17V13.3H87.53ZM86.4,25.89a4.47,4.47,0,0,1-3.49,1.58,4.78,4.78,0,0,1-3.62-1.58A5.53,5.53,0,0,1,77.8,22a5.66,5.66,0,0,1,1.49-4,4.8,4.8,0,0,1,3.62-1.6A4.48,4.48,0,0,1,86.4,18a5.84,5.84,0,0,1,1.41,4A5.7,5.7,0,0,1,86.4,25.89Z"></path><path class="cls-5" d="M62.89,12.74a9,9,0,0,0-6.53,2.62,8.79,8.79,0,0,0-2.68,6.55,8.8,8.8,0,0,0,2.68,6.56,9.45,9.45,0,0,0,13.06,0,8.8,8.8,0,0,0,2.68-6.56,8.79,8.79,0,0,0-2.68-6.55A9,9,0,0,0,62.89,12.74Zm3.65,13.15a5,5,0,0,1-7.3,0,5.48,5.48,0,0,1-1.53-4,5.53,5.53,0,0,1,1.51-4,5.07,5.07,0,0,1,7.34,0,5.49,5.49,0,0,1,1.51,4A5.48,5.48,0,0,1,66.54,25.89Z"></path><path class="cls-3" d="M109.22,27.47a4.68,4.68,0,0,1-4.45-2.78L117,19.62l-.42-1a11,11,0,0,0-.91-1.81,10.64,10.64,0,0,0-1.49-1.86,7.14,7.14,0,0,0-2.36-1.56,7.73,7.73,0,0,0-3.1-.61,8.27,8.27,0,0,0-6.13,2.57,9.05,9.05,0,0,0-2.52,6.6,8.93,8.93,0,0,0,2.61,6.54,8.74,8.74,0,0,0,6.5,2.64,8.43,8.43,0,0,0,4.69-1.25,10.13,10.13,0,0,0,3-2.82l-3.13-2.08A5.26,5.26,0,0,1,109.22,27.47Zm-3.64-9.73a4.79,4.79,0,0,1,3.32-1.46,4.15,4.15,0,0,1,2.12.54,2.82,2.82,0,0,1,1.29,1.41l-8.2,3.41A4.9,4.9,0,0,1,105.58,17.74Z"></path><path class="cls-6" d="M127.47,30.54h-3.55l9.39-24.9h3.62l9.39,24.9h-3.55l-2.4-6.75H129.9Zm7.58-21L131,20.8h8.28L135.19,9.57Z"></path><path class="cls-6" d="M152.44,30.54h-3.2V5.64h3.2Z"></path></g></svg> </a> </div> <ul> <li> <a href="https://twitter.com/search?q=%23madewithmagenta" target="_blank" rel="noopener"> Twitter </a> </li> <li> <a href="/blog" target="_blank" rel="noopener"> Blog </a> </li> <li> <a href="https://github.com/tensorflow/magenta" target="_blank" rel="noopener"> GitHub </a> </li> <li> <a href="https://www.google.com/policies/privacy/" target="_blank" rel="noopener"> Privacy </a> </li> <li> <a href="https://www.google.com/policies/terms/" target="_blank" rel="noopener"> Terms </a> </li> </ul> </div> </footer> </body> <script src="/js/main.js"></script> </html>

CINXE.COM

Research