Faces2Voices

<!DOCTYPE html> <html style="overflow: scroll;"> <head> <meta charset="utf-8"> <meta name="description" content="From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech">  <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Faces2Voices</title>  <script async src="https://www.googletagmanager.com/gtag/js?id=G-PYVRSFMDRL"></script> <script> window.dataLayer = window.dataLayer || []; function gtag() { dataLayer.push(arguments); } gtag('js', new Date()); gtag('config', 'G-PYVRSFMDRL'); </script> <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> <link rel="stylesheet" href="./static/css/bulma.min.css"> <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> <link rel="stylesheet" href="./static/css/index.css">  <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> <script defer src="./static/js/fontawesome.all.min.js"></script> <script src="./static/js/bulma-carousel.min.js"></script> <script src="./static/js/bulma-slider.min.js"></script> <script src="./static/js/index.js"></script> <script type="text/x-mathjax-config"> MathJax.Hub.Config({ tex2jax: {inlineMath: [['$','$'], ['\$','\$']]} }); </script> <script src='https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-MML-AM_CHTML' async> </script> </head> <body> <nav class="navbar" role="navigation" aria-label="main navigation"> <div class="navbar-brand"> <a role="button" class="navbar-burger" aria-label="menu" aria-expanded="false"> <span aria-hidden="true"></span> <span aria-hidden="true"></span> <span aria-hidden="true"></span> </a> </div> <div class="navbar-menu"> <div class="navbar-start" style="flex-grow: 1; justify-content: center;">  <a class="navbar-item" href="https://mm.kaist.ac.kr/"> <span class="icon"> <i class="fas fa-home"></i> </span> </a> </div> </div> </nav> <section class="hero"> <div class="hero-body"> <div class="container is-max-desktop"> <div class="columns is-centered"> <div class="column has-text-centered"> <h1 class="title is-1 publication-title">From Faces to Voices: <br> Learning Hierarchical Representations for High-quality Video-to-Speech</h1> <div class="is-size-5 publication-authors"> <a href="https://sites.google.com/view/jhoonkim/">Ji-Hoon Kim</a>, </span> <span class="author-block"> <a href="https://choijeongsoo.github.io/">Jeongsoo Choi</a>, </span> <span class="author-block"> Jaehun Kim, </span> <span class="author-block"> Chaeyoung Jung, </span> <span class="author-block"> <a href="https://mmai.io/joon/">Joon Son Chung</a> </span> </div> <br> <div class="is-size-5 publication-authors"> <span class="author-block">Korea Advanced Institute of Science and Technology, Republic of Korea</span> </div> <div class="column has-text-centered"> <div class="publication-links"> <span class="link-block"> <a href="http://arxiv.org/abs/2503.16956" class="external-link button is-normal is-rounded is-dark"> <span class="icon"> <i class="ai ai-arxiv"></i> </span> <span>arXiv</span> </a> </span>   </div> </div> </div> </div> </div> </section> <section class="section"> <div class="container is-max-desktop">  <div class="columns is-centered has-text-centered"> <div class="column is-four-fifths"> <h2 class="title is-3">Abstract</h2> <div class="content has-text-justified"> <p> The objective of this study is to generate high-quality speech from silent talking face videos, a task also known as video-to-speech synthesis. A significant challenge in video-to-speech synthesis lies in the substantial modality gap between silent video and multi-faceted speech. In this paper, we propose a novel video-to-speech system that effectively bridges this modality gap, significantly enhancing the quality of synthesized speech. This is achieved by learning of hierarchical representations from video to speech. Specifically, we gradually transform silent video into acoustic feature spaces through three sequential stages - content, timbre, and prosody modeling. In each stage, we align visual factors - lip movements, face identity, and facial expressions - with corresponding acoustic counterparts to ensure the seamless transformation. Additionally, to generate realistic and coherent speech from the visual representations, we employ a flow matching model that estimates direct trajectories from a simple prior distribution to the target speech distribution. Extensive experiments demonstrate that our method achieves exceptional generation quality comparable to real utterances, outperforming existing methods by a significant margin. </p> </div> </div> </div>  </div> </section> <section class="section"> <div class="content"> <h2 class="title is-3">Results from LRS3-TED Dataset</h2> <p> </p> <table> <thead> <tr> <td align="center"><b>Silent Video</b><br> </td> <td align="center"><b>Ground Truth</b><br> </td> <td align="center"><b>SVTS</b><br> </td> <td align="center"><b>Intelligible</b><br> </td> <td align="center"><b>LTBS</b><br> </td> <td align="center"><b>DiffV2S</b><br> </td> <td align="center"><b><b>Ours</b> (10)<br> </td> <td align="center"><b><b>Ours</b> (1000)<br> </td> </tr> </thead> <tbody> <tr> <td colspan="8">     Sample 1:<i> We really don't talk anymore</i> </td> </tr> </tbody> <tbody> <tr> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/0Fi83BHQsMA-00002-silence.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/0Fi83BHQsMA-00002-gt_wav.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/0Fi83BHQsMA-00002-SVTS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/0Fi83BHQsMA-00002-Intelligible.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/0Fi83BHQsMA-00002-LTBS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/0Fi83BHQsMA-00002-DiffV2S.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/0Fi83BHQsMA-00002-ours-10step.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/0Fi83BHQsMA-00002-ours-1000step.mp4" type="video/mp4"></video> </td> </tr> </tbody> <tbody> <tr> <td colspan="8">     Sample 2:<i> How would I know what I was doing differently</i> </td> </tr> </tbody> <tbody> <tr> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/81Ub0SMxZQo-00002-silence.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/81Ub0SMxZQo-00002-gt_wav.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/81Ub0SMxZQo-00002-SVTS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/81Ub0SMxZQo-00002-Intelligible.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/81Ub0SMxZQo-00002-LTBS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/81Ub0SMxZQo-00002-DiffV2S.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/81Ub0SMxZQo-00002-ours-10step.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/81Ub0SMxZQo-00002-ours-1000step.mp4" type="video/mp4"></video> </td> </tr> </tbody> <tbody> <tr> <td colspan="8">     Sample 3:<i> This meant that the movement demanding change knew what they were against crushing poverty widening</i> </td> </tr> </tbody> <tbody> <tr> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/JSSc7hYKstI-00007-silence.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/JSSc7hYKstI-00007-gt_wav.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/JSSc7hYKstI-00007-SVTS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/JSSc7hYKstI-00007-Intelligible.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/JSSc7hYKstI-00007-LTBS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/JSSc7hYKstI-00007-DiffV2S.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/JSSc7hYKstI-00007-ours-10step.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/JSSc7hYKstI-00007-ours-1000step.mp4" type="video/mp4"></video> </td> </tr> </tbody> <tbody> <tr> <td colspan="8">     Sample 4:<i> And that's really what I find so intriguing about the reactions that we've had to the</i> </td> </tr> </tbody> <tbody> <tr> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/YzGjO5aHShQ-00002-silence.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/YzGjO5aHShQ-00002-gt_wav.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/YzGjO5aHShQ-00002-SVTS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/YzGjO5aHShQ-00002-Intelligible.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/YzGjO5aHShQ-00002-LTBS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/YzGjO5aHShQ-00002-DiffV2S.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/YzGjO5aHShQ-00002-ours-10step.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs3/YzGjO5aHShQ-00002-ours-1000step.mp4" type="video/mp4"></video> </td> </tr> </tbody> </table> </div> </section> <section class="section"> <div class="content"> <h2 class="title is-3">Results from LRS2-BBC Dataset</h2> <p> </p> <table> <thead> <tr> <td align="center"><b>Silent Video</b><br> </td> <td align="center"><b>Ground Truth</b><br> </td> <td align="center"><b>SVTS</b><br> </td> <td align="center"><b>Intelligible</b><br> </td> <td align="center"><b>LTBS</b><br> </td> <td align="center"><b>DiffV2S</b><br> </td> <td align="center"><b><b>Ours</b> (10)<br> </td> <td align="center"><b><b>Ours</b> (1000)<br> </td> </tr> </thead> <tbody> <tr> <td colspan="8">     Sample 1:<i> Came up with the perfect name</i> </td> </tr> </tbody> <tbody> <tr> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6387701707842623991-00010-silence.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6387701707842623991-00010-gt_wav.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6387701707842623991-00010-SVTS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6387701707842623991-00010-Intelligible.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6387701707842623991-00010-LTBS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6387701707842623991-00010-DiffV2S.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6387701707842623991-00010-ours-1000step.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6387701707842623991-00010-ours-1000step.mp4" type="video/mp4"></video> </td> </tr> </tbody> <tbody> <tr> <td colspan="8">     Sample 2:<i> As a result of smoking</i> </td> </tr> </tbody> <tbody> <tr> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6336117003133970558-00003-silence.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6336117003133970558-00003-gt_wav.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6336117003133970558-00003-SVTS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6336117003133970558-00003-Intelligible.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6336117003133970558-00003-LTBS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6336117003133970558-00003-DiffV2S.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6336117003133970558-00003-ours-10step.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6336117003133970558-00003-ours-1000step.mp4" type="video/mp4"></video> </td> </tr> </tbody> <tbody> <tr> <td colspan="8">     Sample 3:<i> If you lived in the building</i> </td> </tr> </tbody> <tbody> <tr> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6335305254315087182-00035-silence.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6335305254315087182-00035-gt_wav.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6335305254315087182-00035-SVTS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6335305254315087182-00035-Intelligible.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6335305254315087182-00035-LTBS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6335305254315087182-00035-DiffV2S.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6335305254315087182-00035-ours-10step.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6335305254315087182-00035-ours-1000step.mp4" type="video/mp4"></video> </td> </tr> </tbody> <tbody> <tr> <td colspan="8">     Sample 4:<i> And, and for me the surprise was</i> </td> </tr> </tbody> <tbody> <tr> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6330311066473698535-00011-silence.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6330311066473698535-00011-gt_wav.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6330311066473698535-00011-SVTS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6330311066473698535-00011-Intelligible.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6330311066473698535-00011-LTBS.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6330311066473698535-00011-DiffV2S.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6330311066473698535-00011-ours-10step.mp4" type="video/mp4"></video></td> <td style="width: 224px !important; min-width: 224px !important; max-width: 224px !important;"> <video width="224" controls preload="metadata"><source src="static/video/synthesized_results/lrs2/6330311066473698535-00011-ours-1000step.mp4" type="video/mp4"></video> </td> </tr> </tbody> </table> </div> </section> <footer class="footer"> <div class="container"> <div class="content has-text-centered"> <a class="icon-link" href="./static/videos/nerfies_paper.pdf"> <i class="fas fa-file-pdf"></i> </a> <a class="icon-link" href="https://github.com/keunhong" class="external-link" disabled> <i class="fab fa-github"></i> </a> </div> <div class="columns is-centered"> <div class="column is-8"> <div class="content"> <p> This website is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>. </p> <p> This means you are free to borrow the <a href="https://github.com/nerfies/nerfies.github.io">source code</a> of this website, we just ask that you link back to this page in the footer. Please remember to remove the analytics code included in the header of the website which you do not want on your website. </p> </div> </div> </div> </div> </footer> </body> </html>

CINXE.COM

Faces2Voices