CINXE.COM

<!DOCTYPE html> <html> <head> <meta name="viewport" content="width=device-width, initial-scale=1.0">   <script async src="https://www.googletagmanager.com/gtag/js?id=UA-46457317-11"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-46457317-11'); </script> <title>WIT - FAQs</title> <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons"> <link rel="stylesheet" href="/what-if-tool/assets/css/material.min.css"> <link rel="stylesheet" href="/what-if-tool/assets/css/material-components-web.min.css"> <link href="https://fonts.googleapis.com/css?family=Open+Sans:400,400italic,500,500italic,700,700italic" rel="stylesheet" type="text/css"> <link href="https://fonts.googleapis.com/css?family=Manrope:400,400italic,500,500italic,700,700italic" rel="stylesheet" type="text/css">  <link href='/what-if-tool/assets/css/new.css' rel='stylesheet' type='text/css'> <link rel="icon" href="/what-if-tool/assets/images/favicon.png" type="image/png"/> <style> .sub-title, .sub-copy { color: #8A2A04; } </style> </head> <body> <div class="mdl-layout mdl-layout--no-desktop-drawer-button mdl-js-layout mdl-layout--fixed-header"> <header class="mdl-layout__header"> <div class="mdl-layout__header-row">  <span class="mdl-layout__title"><a href="/what-if-tool/">What-If Tool</a></span>  <div class="mdl-layout-spacer"></div>  <nav class="mdl-navigation mdl-layout--large-screen-only"> <a class="mdl-navigation__link" href="/what-if-tool/get-started/">GET STARTED</a> <a class="mdl-navigation__link" href="/what-if-tool/learn/">TUTORIALS</a> <a class="mdl-navigation__link" href="/what-if-tool/explore/">DEMOS</a> <a class="mdl-navigation__link" href="/what-if-tool/faqs/">FAQs</a> <a class="mdl-navigation__link" href="https://groups.google.com/forum/#!forum/what-if-tool" target="-_blank">GET INVOLVED<img class="header-arrow" src="/what-if-tool/assets/images/arrow-link-out.png"/></a> <a class="mdl-navigation__link" href="https://github.com/pair-code/what-if-tool" target="-_blank">GITHUB<img class="header-arrow" src="/what-if-tool/assets/images/arrow-link-out.png"/></a> </nav> </div> </header> <div class="mdl-layout__drawer"> <span class="mdl-layout__title"><a href="/what-if-tool/">What-If Tool</a></span> <nav class="mdl-navigation"> <a class="mdl-navigation__link" href="/what-if-tool/get-started/">GET STARTED</a> <a class="mdl-navigation__link" href="/what-if-tool/learn/">TUTORIALS</a> <a class="mdl-navigation__link" href="/what-if-tool/explore/">DEMOS</a> <a class="mdl-navigation__link" href="/what-if-tool/faqs/">FAQs</a> <a class="mdl-navigation__link" href="https://groups.google.com/forum/#!forum/what-if-tool" target="_blank">GET INVOLVED<img class="header-arrow" src="/what-if-tool/assets/images/arrow-link-out.png"/></a> <a class="mdl-navigation__link" href="https://github.com/pair-code/what-if-tool" target="-_blank">GITHUB<img class="header-arrow" src="/what-if-tool/assets/images/arrow-link-out.png"/></a>  </nav> </div> <main class="mdl-layout__content hero-banner"> <div class="sub-hero-container" style="background-image:url( /what-if-tool/assets/images/banner-faqs.png ); height: 125px; "> <div class="hero-container mdl-grid"> <div class="mdl-cell--8-col mdl-cell--6-col-tablet mdl-cell--4-col-phone"> <div class="sub-title">Frequently Asked Questions</div> <div class="sub-copy"></div> </div> </div> </div> <div class="sub-page-container mdl-grid"> <div class="mdl-cell--12-col mdl-cell--8-col-tablet mdl-cell--4-col-phone"> <div class="mdl-cell--8-col mdl-cell--8-col-tablet mdl-cell--4-col-phone"> <div class="faq-element"> <h3 class="faq-title">Where do I go to report a bug in the What-If Tool?</h3> <div class="faq-copy"> <p>Submit bugs, ask questions, suggest content, and request features on our <a href="https://github.com/pair-code/what-if-tool/issues/new">Github issues list</a>.</div></p> </div> <div class="faq-element"> <h3 class="faq-title">How many datapoints can the What-If Tool handle?</h3> <div class="faq-copy"> <p>The number of datapoints depends on the size of the individual datapoints themselves. Some factors that impact the number of datapoints that can be handled are the per-page memory limit of web browsers and the front-end for simultaneous visualization.</p> <p>For datapoints that contain tabular data or short text segments, the What-If Tool can handle <strong>hundreds of thousands of datapoints</strong>. For datapoints that contain encoded images, the tool can only handle a few hundred datapoints, with the exact number depending on the size of the images.</p> </div> </div> <div class="faq-element"> <h3 class="faq-title">What kinds of models can the What-If Tool handle?</h3> <div class="faq-copy"> <p>The What-If Tool can work with any python-accessible model in Notebook environments, and will work with most models hosted by TF-serving in Tensorboard.</p> <p>The What-If Tool supports:</p> <ul> <li>binary classification*</li> <li>multi-class classification</li> <li>regression tasks</li> </ul> <p>* Fairness optimization strategies are available only with binary classification models due to the nature of the strategies themselves.</div></p> </div> <div class="faq-element"> <h3 class="faq-title">Why is the What-If Tool loading so slowly?</h3> <div class="faq-copy"> <p>The What-If Tool runs entirely in the browser, and may therefore load slowly for a variety of reasons. Some known causes are:</p> <ul> <li>if a loaded dataset has a high number of features</li> <li>if each data point in the loaded dataset is large in size (eg. large images)</li> <li>the model takes a long time to return predictions </div></li> </ul> </div> <div class="faq-element"> <h3 class="faq-title">How can I share my scatter plot or ROC curve or confusion matrix with my team?</h3> <div class="faq-copy"> <p>For now, there is no export feature in WIT. We recommend taking a screenshot and annotating it using a simple graphic editor tool. Alternatively, you can create a Colab, Jupyter or Cloud AI notebook with your analysis and share it with your peers. </div></p> </div> <div class="faq-element"> <h3 class="faq-title">How are counterfactuals calculated?</h3> <div class="faq-copy"> <p>In the What-If Tool, Counterfactuals are datapoints that are most similar to a selected datapoint, but are classified differently by a model.</p> <p>For binary classification models, counterfactuals are the most similar datapoint to a selected datapoint that is predicted in the opposite class or label by a model.</p> <p>For regression models, counterfactuals are calculated when the difference in prediction score between the selected datapoint and a candidate counterfactual is equal or greater to the “counterfactual threshold”. The counterfactual threshold default is set to the standard deviation of the prediction values and can be adjusted by the user.</p> <p>For multi-class models, the counterfactual is the most similar datapoint to a selected datapoint, but is classified as any class other than the selected datapoint’s class.</p> <p>By default, features that are fully unique across the dataset (such as encoded images and long text strings) are ignored for this distance calculation. However, it is possible to define your own custom distance between datapoints by specifying it as a custom distance function when loading the What-If Tool in a notebook, as seen in <a href="https://colab.research.google.com/github/pair-code/what-if-tool/blob/master/WIT_Toxicity_Text_Model_Comparison.ipynb#scrollTo=lVaMyc45HWwD">this example</a>.</p> <p>Further reading: Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. <br/> <a href="https://arxiv.org/abs/1711.00399">https://arxiv.org/abs/1711.00399</a></div></p> </div> <div class="faq-element"> <h3 class="faq-title">I have proprietary data. Is WIT secure for my team to use?</h3> <div class="faq-copy"> <p>WIT uses pre-trained models and runs entirely in the browser. We don't store, collect or share datasets loaded into the What-If Tool. If using the tool inside TensorBoard, then access to that TensorBoard instance can be controlled through the authorized_groups command-line flag to TensorBoard. Anyone with access to the TensorBoard instance will be able to see data from the datasets that the instance has permissions to load from disk. If using WIT inside of colab, access to the data is controlled by the colab kernel, outside the scope of WIT. </div></p> </div> <div class="faq-element"> <h3 class="faq-title">What are some examples of features people have added in the datapoint editor?</h3> <div class="faq-copy"> <p>The What-If Tool uses trained models to investigate prediction results on uploaded dataset, and models do not learn anything from edits or injunctions in the What-If Tool. Therefore, any new features added to the datapoint editor has no bearing on the model itself. Users typically add new features to:</p> <ul> <li>describe or annotate datapoints using features that are irrelevant to the model</li> <li>see if there are any trends or correlations that might make a feature useful to include when training another iteration of the model outside of the What-If Tool.</li> </ul> </div> </div> <div class="faq-element"> <h3 class="faq-title">Can I use WIT with LIME or SHAP?</h3> <div class="faq-copy"> <p>Yes! The What-If Tool supports feature attributions in notebook mode. You can use any feature attribution method, including LIME, SHAP, integrated gradients, and SmoothGrad, in your own custom predict function. Return these attributions along with the predictions for use in your analysis with the What-If Tool. See this <a href="https://colab.research.google.com/github/PAIR-code/what-if-tool/blob/master/WIT_COMPAS_with_SHAP.ipynb">demo</a>. </div></p> </div> </div> </div> <div class="fixed-sub-navigation hide-me"> <a href="https://github.com/pair-code/what-if-tool/issues/new" target="_blank">Ask a question</a> </div> </div> <div class="footer-container mdl-grid"> <div class="mdl-cell mdl-cell--2-col mdl-cell--2-col-tablet mdl-cell--4-col-phone"><a href="https://pair.withgoogle.com/" target="_blank"><img src="/what-if-tool/assets/images/pair-logo.svg"/></a></div> <div class="mdl-cell mdl-cell--2-col mdl-cell--2-col-tablet mdl-cell--4-col-phone"><a href="https://research.google/teams/brain/pair/" target="_blank">Google Research</a></div> <div class="mdl-cell mdl-cell--2-col mdl-cell--2-col-tablet mdl-cell--4-col-phone"><a href="https://groups.google.com/forum/#!forum/what-if-tool" target="_blank">Get Involved</a></div> <div class="mdl-cell mdl-cell--2-col mdl-cell--2-col-tablet mdl-cell--4-col-phone"><a href="https://github.com/pair-code" target="_blank">Github</a></div> <div class="footer-icons mdl-cell mdl-cell--4-col mdl-cell--8-col-tablet mdl-cell--4-col-phone"> <a href="mailto:peopleai@google.com" target="_blank"><img src="/what-if-tool/assets/images/mail.png"/></a> <a href="https://medium.com/people-ai-research" target="_blank"><img src="/what-if-tool/assets/images/medium.png"/></a> <a href="https://www.youtube.com/channel/UCnnns-uu4yy9BXfYSPIX5AA" target="_blank"><img src="/what-if-tool/assets/images/youtube.png"/></a> </div> </div> </main> </div> </body> <script defer src="/what-if-tool/assets/js/material.min.js"></script> <script defer src="/what-if-tool/assets/js/material-components-web.min.js"></script> </html>