Institute of Software Chinese Academy of Sciences

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Institute of Software Chinese Academy of Sciences</title> <link href="../../../images/style_en.css" rel="stylesheet" type="text/css"> </head> <body> <div id="main_container"> <div id="header"> <div id="logo"> <a href="#"><img src="http://english.is.cas.cn/images/en_logo.png" width="341" height="58" border="0" /></a> </div> <div class="banner_adds"> <ul> <li><a href="http://english.is.cas.cn">Home</a></li> <li><a href="http://english.is.cas.cn/sitemap/">Sitemap</a></li> <li><a href="http://english.is.cas.cn/au/ct/">Contact</a></li> <li><a href="http://www.is.cas.cn/" target="_blank">中文</a></li> <li style="border:none"><a href="http://english.cas.cn/" target="_blank">CAS</a></li> </ul> </div> <div class="menu"> <ul> <li><a href="http://english.is.cas.cn/au/">About us</a>   <ul> <li><a href="http://english.is.cas.cn/au/bi/">Brief Introduction</a></li> <li><a href="http://english.is.cas.cn/au/hy/">History</a></li> <li><a href="http://english.is.cas.cn/au/ds/">Director</a></li> <li><a href="http://english.is.cas.cn/au/aftd/">Address from the Director</a></li> <li><a href="http://english.is.cas.cn/au/an/">Organization</a></li> <li><a href="http://english.is.cas.cn/au/ct/">Contact</a></li> </ul>  </li> <li><a href="http://english.is.cas.cn/rh/">Research</a>    <ul> <li><a href="http://english.is.cas.cn/rh/rd/">Research Divisions</a></li> <li><a href="http://english.is.cas.cn/rh/rp/">Research Progress</a></li> <li><a href="http://english.is.cas.cn/rh/as/">Achievements</a></li> <li><a href="http://english.is.cas.cn/rh/rps/">Research Programs</a></li> <li><a href="http://english.is.cas.cn/rh/tt/">Technology Transfer</a></li> </ul>  </li> <li><a href="http://english.is.cas.cn/pe/">People</a>    <ul> <li><a href="http://english.is.cas.cn/pe/fas/">Faculty and Staff</a></li> <li><a href="http://english.is.cas.cn/pe/LT/">Managing Team</a></li> <li><a href="http://english.is.cas.cn/pe/cm/">CAS Members</a></li> </ul>  </li> <li><a href="http://english.is.cas.cn/ic/">International Cooperation</a>    <ul> <li><a href="http://english.is.cas.cn/ic/in/">Introduction</a></li> <li><a href="http://english.is.cas.cn/ic/ip/">International Projects </a></li> <li><a href="http://english.is.cas.cn/ic/ic/">International Conferences </a></li> <li><a href="http://english.is.cas.cn/ic/ios/">International Organizations </a></li> </ul>  </li> <li><a href="http://english.is.cas.cn/ns/">News</a>    <ul> <li><a href="http://english.is.cas.cn/ns/es/">Events </a></li> <li><a href="http://english.is.cas.cn/ns/icn/">Int’l Cooperation News </a></li> <li><a href="http://english.is.cas.cn/ns/ue/">Upcoming Events</a></li> </ul>  </li> <li><a href="http://english.is.cas.cn/rs/">Resources</a>    <ul> <li><a href="http://english.is.cas.cn/rs/ma/">Multimedia</a></li> <li><a href="http://english.is.cas.cn/rs/fs/">Facilities</a></li> </ul>  </li> <li><a href="http://english.is.cas.cn/et/">Education & Training</a>    <ul> <li><a href="http://english.is.cas.cn/et/education_intro/">Introduction</a></li> <li><a href="http://english.is.cas.cn/et/Enrollment/">Enrollment </a></li> <li><a href="http://english.is.cas.cn/et/Fellowship_Scholarship/">Fellowship&Scholarship </a></li> </ul>  </li> <li><a href="http://english.is.cas.cn/ju/">Join Us</a></li> <li><a href="http://english.is.cas.cn/sp/">Societies & Publications</a></li> <li><a href="http://english.is.cas.cn/ls/">Links</a></li> </ul> </div> </div>  <div id="main_content">  <div class="column1">  <div class="left_box"> <div class="top_left_box"> </div> <div class="center_left_box"> <div class="box_title"><span>Research</span> :</div> <div class="box_text"> <p><img src="../../../images/bit02.gif" width="11" height="7" alt="" class="dot" /><a href="../../rd/">Research Divisions</a></p> <p><img src="../../../images/bit02.gif" width="11" height="7" alt="" class="dot" /><a href="../">Research Progress</a></p> <p><img src="../../../images/bit02.gif" width="11" height="7" alt="" class="dot" /><a href="../../as/">Achievements</a></p> <p><img src="../../../images/bit02.gif" width="11" height="7" alt="" class="dot" /><a href="../../rps/">Research Programs</a></p> <p><img src="../../../images/bit02.gif" width="11" height="7" alt="" class="dot" /><a href="../../tt/">Technology Transfer</a></p> </div> </div> <div class="bottom_left_box"> </div> </div>   <div class="left_box"> <div class="top_left_box"> </div> <div class="center_left_box"> <div class="box_title"><span>Find</span> information:</div> <div class="form"> <div class="form_row"> <label class="left">Search: </label> <input type="text" class="form_input" /> </div> <div style="float:right; padding:10px 25px 10px 0;"> <input type="image" src="../../../images/go.gif" /> </div> </div> </div> <div class="bottom_left_box"> </div> </div>  </div>   <div class="column_right_2"> <div class="column5"> <div class="small_title" style="color:#000000"><a href="../../../" title="Home" class="CurrChnlCls">Home</a> > <a href="../../" title="Research" class="CurrChnlCls">Research</a> > <a href="../" title="Research Progress" class="CurrChnlCls">Research Progress</a></div> <div class="main_text_box"> <div class="main_text_box_5"> <p class="main_text_box_5_title">Scientists Develop A Novel Multitask Learning Method for Surface Reconstruction and Image Synthesis <br /><br /> <span>Date:2019-08-29</span> </p> <div class=TRS_Editor><font face="Times New Roman"> </font><p style="margin-top:3.75pt;margin-right:0cm;margin-bottom:13.5pt;margin-left:
0cm;text-align:justify;text-justify:inter-ideograph;background:white"><span lang="EN-US" style="font-size: 14pt; color: black;"><font face="Times New Roman">Surface reconstruction and photorealistic rendering of water have always been two independent and <a name="OLE_LINK2">tricky</a>subjects in computer graphics and computer vision. Traditional methods solve the two issues respectively based on physical models. While in fact, there is an <a name="OLE_LINK7">inverse</a> relationship between them, a two-way transfer between water images and surface geometry. 3D water surface reconstruction from a single image is a well-known ill-posed problem. It is eagerly to put forward an efficient and stable water surface reconstruction algorithm for the computer vision community.<o:p></o:p></font></span></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-size: 14pt; color: black;"><font face="Times New Roman">Recently, Assoc. Prof. HOU Fei from State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences and his collaborators from Beihang University and Stony Brook University, put forward a joint deep learning architecture based on the inverse relationship, to efficiently handle water surface reconstruction and water rendering together. <o:p></o:p></font></span></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-size: 14pt; color: black;"><o:p><font face="Times New Roman"> </font></o:p></span></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-size: 14pt; color: black;"><font face="Times New Roman">The researchers designed an end-to-end cycle multitask learning framework, in which the two ends are respectively RGB water images and surface normal maps. The model included a forward surface estimation net and a backward surface rendering net, which was very similar to the work on cycle image-to-image transformation with encoder-transformation-decoder structures. To reuse the lighting situation from the source images in the rendering process, they further employed an extra subnetwork to encode the imaging parameters in the forward part and exerted those parameters on the backward part. The inverse relation of the two parts was exploited to expedite the learning process of each other and improved overall robustness.<o:p></o:p></font></span></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-size: 14pt; color: black;"><o:p><font face="Times New Roman"> </font></o:p></span></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-size: 14pt; color: black;"><font face="Times New Roman">They further improved the performance <a name="OLE_LINK9">usin</a><a name="OLE_LINK10"><span style="mso-bookmark:OLE_LINK9">g</span>other techniques, for example, adding universality loss to make the estimated parameters independent of geometries and applying conditional GAN architecture to measure the high-level consistency.<o:p></o:p></a></font></span></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><span style="mso-bookmark:OLE_LINK10"><span lang="EN-US" style="font-size: 14pt; color: black;"><o:p><font face="Times New Roman"> </font></o:p></span></span></p><font face="Times New Roman"> </font><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-size: 14pt; color: black;"><font face="Times New Roman">Abundant experimental results validate that the method is both visually plausible and computationally efficient. which makes estimating fluid geometries and editing fluid images/videos possible in real time. This work facilitates real-time modeling and rendering of water surfaces and makes them in a unified framework. It prompts the practical application of the water rendering and modeling and is possible to be applied in the field of computer games, film production and simulation.<o:p></o:p></font></span></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><font face="Times New Roman"><a name="OLE_LINK13"><span lang="EN-US" style="font-size: 14pt; color: black;">This work was presented in the 32nd International Conference on Computer Animation and Social Agents and published in the Computer Animation and Virtual Worlds</span></a><span lang="EN-US" style="font-size: 14pt; color: black;">, entitled " <a name="OLE_LINK14">Multitask learning on monocular water images: Surface reconstruction and image synthesis</a> ".</span><span lang="EN-US" style="font-size: 14pt; color: red;"><o:p></o:p></span></font></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><font face="Times New Roman"><span lang="EN-US" style="font-size: 14pt; color: black;"><br /></span></font></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><font face="Times New Roman"><span style="font-size:14.0pt;mso-ascii-font-family:"Times New Roman";mso-hansi-font-family:
"Times New Roman";mso-bidi-font-family:"Times New Roman"">文章链接：</span><span lang="EN-US"><b><span style="color: rgb(0, 82, 116); background: white;"><a href="https://doi.org/10.1002/cav.1896">https://doi.org/10.1002/cav.1896</a></span></b></span></font><span lang="EN-US" style="font-size:14.0pt;font-family:"Times New Roman",serif;
color:red"><o:p></o:p></span></p><p class="MsoNormal" style="text-align: center; "><font face="Times New Roman"><span lang="EN-US"><b><img src="./W020190829353798923935.jpg" style="border-width: 0px;" alt="" OLDSRC="W020190829353798923935.jpg" /><br /></b></span></font></p><p class="MsoNormal" style="line-height:12.0pt;mso-line-height-rule:exactly;
mso-layout-grid-align:none;text-autospace:none"><span lang="EN-US" style="font-size:12.0pt;font-family:"Times New Roman",serif">Figure: Network architecture. Our network includes a forward estimating net E and a backward rendering net R, both based on the encoder–transformation–decoder structure. The estimating net E also includes a Sub-Net 2 for extracting imaging parameters to reuse the lighting conditions in the backward rendering pass of net R.<span style="color:red"><o:p></o:p></span></span></p><p class="MsoNormal" style="text-align: center; "> </p><p class="MsoNormal"><span lang="EN-US" style="font-size: 14pt; font-family: "Times New Roman", serif;">Image by HOU Fei <o:p></o:p></span></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><font face="Times New Roman"><span lang="EN-US"><b><br /></b></span></font></p><p class="MsoNormal" style="mso-layout-grid-align:none;text-autospace:none"><font face="Times New Roman"><span lang="EN-US"><b><br /></b></span></font></p></div> </div> </div> </div> </div>  </div>   <div id="footer"> <div id="copyright"> <div>©2011 Institute of Software, CAS. All rights reserved.info(at)iscas.ac.cn</div> </div> </div> </div>  </div> </body> </html>

CINXE.COM

Institute of Software Chinese Academy of Sciences