Revisiting the Calibration of Modern Neural Networks

<!doctype html> <html lang="en"> <head>  <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <title>Revisiting the Calibration of Modern Neural Networks</title> <link rel="stylesheet" href="/static/papers/css/papers.css"> <meta name="citation_title" content="Revisiting the Calibration of Modern Neural Networks"> <meta name="citation_author" content="Minderer, Matthias"> <meta name="citation_author" content="Djolonga, Josip"> <meta name="citation_author" content="Romijnders, Rob"> <meta name="citation_author" content="Hubis, Frances"> <meta name="citation_author" content="Zhai, Xiaohua"> <meta name="citation_author" content="Houlsby, Neil"> <meta name="citation_author" content="Tran, Dustin"> <meta name="citation_author" content="Lucic, Mario"> <meta name="citation_journal_title" content="Advances in Neural Information Processing Systems"> <meta name="citation_volume" content="34"> <meta name="citation_firstpage" content="15682"> <meta name="citation_lastpage" content="15694"> <meta name="citation_pdf_url" content="https://proceedings.neurips.cc/paper_files/paper/2021/file/8420d359404024567b5aefda1231af24-Paper.pdf"> <meta name="citation_publication_date" content="2021-12-06">   <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css" integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous"> <link href="/static/menus/css/menus.css" rel="stylesheet" id="bootstrap-css"> <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.8.1/css/all.css" integrity="sha384-50oBUHEmvpQ+1lW4y57PTFmhCaXp0ML5d60M1M7uH2+nqUivzIebhndOJK28anvf" crossorigin="anonymous"> <script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-MML-AM_CHTML"> </script> <script type="text/x-mathjax-config"> MathJax.Hub.Config({ "tex2jax": { "inlineMath": [["$","$"], ["\$","\$"]], "displayMath": [["\\[","\\]"]], "processEscapes": true } } ); </script> <style> @media (prefers-color-scheme: dark) { body { background-color: #333; color: #eee; } } .btn-spacer { margin: 2px; } .footer { position: fixed; left: 0; bottom: 0; width: 100%; background-color: #eee; color: black; } </style> </head> <body> <nav class="navbar navbar-expand-md navbar-light bg-light"> <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarToggler6" aria-controls="navbarToggler6" aria-expanded="false" aria-label="Toggle navigation"> <span class="navbar-toggler-icon"></span> </button> <div class="collapse navbar-collapse" id="navbarToggler6"> <a class="navbar-brand" href="/">NeurIPS Proceedings</a> <ul class="navbar-nav mr-auto mt-2 mt-md-0"> <li class="nav-item"> <a class="nav-link" href="/admin/login/?next=/admin/"><i class="fas fa-sign-in-alt" title="Login"></i></a> </li> <li class="nav-item"> <a class="nav-link" href="/admin/logout/?nextp=/admin"><i class="fas fa-sign-out-alt" title="Logout"></i></a> </li> </ul> <form class="form-inline my-2 my-lg-0" method="get" role="search" action="/papers/search"> <input class="form-control mr-sm-2" type="text" name="q" placeholder="Search" aria-label="Search" id="navsearch"> <button class="btn btn-outline-success my-2 my-sm-0" type="submit">Search</button> </form> </div> </nav> <br> <div class="container-fluid"> <div class="col"> <h4>Revisiting the Calibration of Modern Neural Networks</h4> <p> Part of <a href="/paper_files/paper/2021">Advances in Neural Information Processing Systems 34 (NeurIPS 2021) </a> </p> <div><a href="/paper_files/paper/12823-/bibtex" class="btn btn-light btn-spacer">Bibtex</a> <a class='btn btn-light btn-spacer' href='/paper_files/paper/2021/file/8420d359404024567b5aefda1231af24-Paper.pdf'>Paper</a> <a target="_blank" href='https://openreview.net/forum?id=QRBvLayFXI' class='btn btn-light btn-spacer'>Reviews And Public Comment »</a> <a class='btn btn-light btn-spacer' href='/paper_files/paper/2021/file/8420d359404024567b5aefda1231af24-Supplemental.pdf'>Supplemental</a></div><br> <h4>Authors</h4> <p><i>Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, Mario Lucic</i></p> <h4>Abstract</h4> <p><p>Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.</p> </p> </div> </div> <script src="https://code.jquery.com/jquery-3.6.0.min.js" integrity="sha256-/xUj+3OJU5yExlq6GSYGSHk7tPXikynS7ogEvDej/m4=" crossorigin="anonymous"></script> <script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.0/dist/umd/popper.min.js" integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo" crossorigin="anonymous"></script> <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/js/bootstrap.min.js" integrity="sha384-ChfqqxuZUCnJSK3+MXmPNIyE6ZbWh2IMqE241rYiqJxyMiZ6OW/JmZQ5stwEULTy" crossorigin="anonymous"></script> <script> $('.dropdown-menu a.dropdown-toggle').on('click', function (e) { if (!$(this).next().hasClass('show')) { $(this).parents('.dropdown-menu').first().find('.show').removeClass("show"); } var $subMenu = $(this).next(".dropdown-menu"); $subMenu.toggleClass('show'); $(this).parents('li.nav-item.dropdown.show').on('hidden.bs.dropdown', function (e) { $('.dropdown-submenu .show').removeClass("show"); }); return false; }); </script> <br> <div class="modal fade" id="myModal"> <div class="modal-dialog"> <div class="modal-content"> <div class="modal-header"> <h4 class="modal-title">Name Change Policy</h4> <button type="button pull-right" class="close" data-dismiss="modal" aria-hidden="true">×</button> </div> <div class="modal-body"> <p>Requests for name changes in the electronic proceedings will be accepted with no questions asked. However name changes may cause bibliographic tracking issues. Authors are asked to consider this carefully and discuss it with their co-authors prior to requesting a name change in the electronic proceedings.</p> <p>Use the "Report an Issue" link to request a name change.</p> </div> <div class="modal-footer"> </div> </div> </div> </div> <footer class="footer"> <div class="btn-spacer"> <a href="https://neurips.cc/Help/Contact?select=Conference">Report an Issue</a>    |    <a href="#" class="modal-link" data-toggle="modal" data-target="#myModal">Name Change Policy</a> </div> </div> </footer> <div class='hidden'>Do not remove: This comment is monitored to verify that the site is working properly</div>  </body> </html>

CINXE.COM

Revisiting the Calibration of Modern Neural Networks