Understanding Treeherder Results — Firefox Source Docs documentation

<!doctype html> <html class="writer-html5" lang="en" data-content_root="../../"> <head> <base href="https://firefox-source-docs.mozilla.org/testing/treeherder-try/index.html"> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Understanding Treeherder Results — Firefox Source Docs documentation</title> <link rel="stylesheet" type="text/css" href="../../_static/pygments.css?v=fa44fd50"> <link rel="stylesheet" type="text/css" href="../../_static/css/theme.css?v=19f00094"> <link rel="stylesheet" type="text/css" href="../../_static/graphviz.css?v=fd3f3429"> <link rel="stylesheet" type="text/css" href="../../_static/copybutton.css?v=76b2166b"> <link rel="stylesheet" type="text/css" href="../../_static/custom_theme.css?v=dc5ed815"> <link rel="stylesheet" type="text/css" href="../../_static/design-style.1e8bd061cd6da7fc9cf755528e8ffc24.min.css?v=0a3b3ea7"> <link rel="shortcut icon" href="../../_static/firefox.ico"> <script src="../../_static/jquery.js?v=5d32c60e"></script> <script src="../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script> <script src="../../_static/documentation_options.js?v=5929fcd5"></script> <script src="../../_static/doctools.js?v=9a2dae69"></script> <script src="../../_static/sphinx_highlight.js?v=dc90522c"></script> <script src="../../_static/clipboard.min.js?v=a7894cd8"></script> <script src="../../_static/copybutton.js?v=30646c52"></script> <script src="../../_static/design-tabs.js?v=36754332"></script> <script src="../../_static/js/theme.js"></script> <link rel="index" title="Index" href="../../genindex.html"> <link rel="search" title="Search" href="../../search.html"> <link rel="next" title="Sheriffed intermittent failures" href="../sheriffed-intermittents/index.html"> <link rel="prev" title="Adding Context to manifestparser based Manifests" href="../automated-testing/manifest-sandbox.html"> <meta http-equiv="X-Translated-By" content="Google"> <meta http-equiv="X-Translated-To" content="iw"> <script type="text/javascript" src="https://www.gstatic.com/_/translate_http/_/js/k=translate_http.tr.en_GB.tKc6KWkFf-8.O/am=gAE/d=1/rs=AN8SPfrf36LIV3DkhtRBGWFnLWWzaykPyw/m=corsproxy" data-sourceurl="https://firefox-source-docs.mozilla.org/testing/treeherder-try/index.html"></script> <link href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@20..48,100..700,0..1,-50..200" rel="stylesheet"> <script type="text/javascript" src="https://www.gstatic.com/_/translate_http/_/js/k=translate_http.tr.en_GB.tKc6KWkFf-8.O/am=gAE/d=1/exm=corsproxy/ed=1/rs=AN8SPfrf36LIV3DkhtRBGWFnLWWzaykPyw/m=phishing_protection" data-phishing-protection-enabled="false" data-forms-warning-enabled="true" data-source-url="https://firefox-source-docs.mozilla.org/testing/treeherder-try/index.html"></script> <meta name="robots" content="none"> </head> <body class="wy-body-for-nav"> <script type="text/javascript" src="https://www.gstatic.com/_/translate_http/_/js/k=translate_http.tr.en_GB.tKc6KWkFf-8.O/am=gAE/d=1/exm=corsproxy,phishing_protection/ed=1/rs=AN8SPfrf36LIV3DkhtRBGWFnLWWzaykPyw/m=navigationui" data-environment="prod" data-proxy-url="https://firefox--source--docs-mozilla-org.translate.goog" data-proxy-full-url="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB" data-source-url="https://firefox-source-docs.mozilla.org/testing/treeherder-try/index.html" data-source-language="pl" data-target-language="iw" data-display-language="en-GB" data-detected-source-language="" data-is-source-untranslated="false" data-source-untranslated-url="https://translate.google.com/website?sl=pl&tl=iw&hl=en-GB&u=https://firefox-source-docs.mozilla.org/testing/treeherder-try/index.html&anno=2" data-client="tr"></script> <div class="wy-grid-for-nav"> <nav data-toggle="wy-nav-shift" class="wy-nav-side"> <div class="wy-side-scroll"> <div class="wy-side-nav-search"><a href="https://firefox--source--docs-mozilla-org.translate.goog/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB" class="icon icon-home"> Firefox Source Docs <img src="../../_static/firefox-wordmark.svg" class="logo" alt="Logo"> </a> <div> <h3>Quick search</h3> <script> (function () { var cx = "dd12886298f75dbef"; var gcse = document.createElement("script"); gcse.async = true; gcse.src = "https://cse.google.com/cse.js?cx=" + cx; var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(gcse, s); })(); </script><gcse:search></gcse:search> </div> </div> <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> <p class="caption" role="heading"><span class="caption-text">Overview</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/glossary/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">A Glossary of Common Terms</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/overview/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">A Quick Guide to Mozilla Applications</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Getting Started</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/setup/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Getting Set Up To Work On The Firefox Codebase</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Working On Firefox</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/contributing/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Working on Firefox</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/bug-mgmt/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Bug Handling</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Firefox User Guide</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/devtools-user/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Firefox DevTools User Docs</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Source Code Documentation</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/mots/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Governance</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/browser/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Firefox Front-end</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/dom/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">DOM</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/editor/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Editor</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/layout/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Style system (CSS) & Layout</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/gfx/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Graphics</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/ipc/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Processes, Threads and IPC</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/devtools/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Firefox DevTools Contributor Docs</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/toolkit/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Toolkit</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/js/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">SpiderMonkey</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/jsloader/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">JS Loader</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/mobile/android/geckoview/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">GeckoView</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/mobile/android/fenix/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Fenix</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/mobile/android/focus-android/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Focus for Android</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/dom/bindings/webidl/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">WebIDL</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/modules/libpref/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">libpref</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/networking/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Networking</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/remote/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Remote Protocols</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/services/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Services</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/permissions/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Permissions</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/uriloader/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">File Handling</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/widget/cocoa/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Firefox on macOS</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/widget/windows/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Firefox on Windows</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/toolkit/components/ml/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Firefox AI Runtime</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/accessible/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Accessibility</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/media/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Media Playback</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/code-quality/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Code quality</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/writing-rust-code/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Writing Rust Code</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/rust-components/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Rust Components</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/tools/profiler/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Gecko Profiler</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/performance/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Performance</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/storage/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Database bindings (SQLite, KV, …)</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/xpcom/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">XPCOM</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/nspr/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">NSPR</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/security/nss/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Network Security Services (NSS)</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/content-security/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Web Security Checks in Gecko</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">The Firefox Build System</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/mach/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Mach</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/tools/try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Pushing to Try</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/build/buildsystem/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Build System</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/taskcluster/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Firefox CI and Taskgraph</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/tools/moztreedocs/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Managing Documentation</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/mozbuild/vendor/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Vendoring Third Party Components</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Testing & Test Infrastructure</span></p> <ul class="current"> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/automated-testing/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Automated Testing</a></li> <li class="toctree-l1 current"><a class="current reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#">Understanding Treeherder Results</a> <ul> <li class="toctree-l2"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#confirm-failure-cf">Confirm Failure (CF)</a></li> <li class="toctree-l2"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#retrigger-r">Retrigger (R)</a></li> <li class="toctree-l2"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#new-annotations-new">NEW annotations (NEW)</a></li> <li class="toctree-l2"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#mitten-icon">MITTEN icon</a></li> <li class="toctree-l2"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#bug-suggestions">Bug Suggestions</a></li> </ul></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/sheriffed-intermittents/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Sheriffed intermittent failures</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/tests-for-new-config/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Turning on Firefox tests for a new configuration</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/intermittent/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Avoiding intermittent tests</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/testing-policy/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Testing Policy</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/ci-configs/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Configuration Changes</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/browser-chrome/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Browser chrome mochitests</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/chrome-tests/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Chrome Tests</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/marionette/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Marionette</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/geckodriver/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">geckodriver</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/test-verification/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Test Verification</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/webrender/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">WebRender Tests</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/mochitest-plain/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Mochitest</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/xpcshell/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">XPCShell tests</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/tps/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">TPS</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/web-platform/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">web-platform-tests</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/gtest/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">GTest</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/tools/fuzzing/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Fuzzing</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/tools/sanitizer/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Sanitizer</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/perfdocs/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Performance Testing</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/tools/code-coverage/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Code coverage</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/testing-rust-code/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Testing & Debugging Rust Code</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Releases & Updates</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/update-infrastructure/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Mozilla Update Infrastructure</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/update-infrastructure/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#watershed-updates">Watershed Updates</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/update-infrastructure/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#desupport-updates">Desupport Updates</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/tools/update-verify/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Update Verify</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Localization & Internationalization</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/intl/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Internationalization</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/l10n/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Localization</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Firefox and Python</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/mozbase/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">mozbase</a></li> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/python/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Using third-party Python packages</a></li> </ul> <p class="caption" role="heading"><span class="caption-text">Metrics Collected in Firefox</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/metrics/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Metrics</a></li> </ul> </div> </div> </nav> <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> <nav class="wy-nav-top" aria-label="Mobile navigation menu"><i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="https://firefox--source--docs-mozilla-org.translate.goog/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB">Firefox Source Docs</a> </nav> <div class="wy-nav-content"> <div class="rst-content"> <div role="navigation" aria-label="Page navigation"> <ul class="wy-breadcrumbs"> <li><a href="https://firefox--source--docs-mozilla-org.translate.goog/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB" class="icon icon-home" aria-label="Home"></a></li> <li class="breadcrumb-item active">Understanding Treeherder Results</li> <li class="wy-breadcrumbs-aside"><a href="https://translate.google.com/website?sl=pl&tl=iw&hl=en-GB&u=https://bugzilla.mozilla.org/enter_bug.cgi?product%3DDeveloper%2BInfrastructure%26component%3DFirefox%2BSource%2BDocs%253A%2BContent%26short_desc%3DDocumentation%2Bissue%2Bon%2Btesting/treeherder-try/index%26comment%3DURL%2B%3D%2Bhttps://firefox-source-docs.mozilla.org/testing/treeherder-try/index.html%26bug_file_loc%3Dhttps://firefox-source-docs.mozilla.org/testing/treeherder-try/index.html" rel="nofollow">Report an issue</a> / <a href="https://firefox--source--docs-mozilla-org.translate.goog/_sources/testing/treeherder-try/index.rst.txt?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB" rel="nofollow"> View page source</a></li> </ul> <hr> </div> <div role="main" class="document" itemscope itemtype="http://schema.org/Article"> <div itemprop="articleBody"> <section id="understanding-treeherder-results"> <h1>Understanding Treeherder Results<a class="headerlink" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#understanding-treeherder-results" title="Link to this heading"></a></h1> <p><a class="reference external" href="https://translate.google.com/website?sl=pl&tl=iw&hl=en-GB&u=https://treeherder.mozilla.org/userguide">Treeherder</a> serves as the primary dashboard for developers looking to view CI test results from their try pushes. The main purpose of the dashboard is to display all the tasks and their status along with tool to view logs, see failures, add jobs, cancel jobs, or connect to other tools.</p> <p>When a test fails, it is important to figure out if this is a regression or an intermittent failure. It is typically to see 3-10% of the jobs fail, although the majority of the failures are intermittent (infrastructure failed, timing is off, test is sort of flaky, etc.). Sometimes a regression occurs but the failure is assumed intermittent.</p> <dl class="simple"> <dt> There are a few tools to use when a test fails: </dt> <dd> <ul class="simple"> <li><p>Confirm Failure (CF)</p></li> <li><p>Retrigger (R)</p></li> <li><p>NEW annotation (NEW)</p></li> <li><p>MITTEN icon</p></li> <li><p>Bug Suggestions</p></li> </ul> </dd> </dl> <p>The quick answer is: use confirm failure for all failures and when all results are in, whichever tests are orange and do not have a mitten icon are what need further investigation.</p> <p>It is best to understand how each of the tools work and use them in combination to be most efficient with your time and to find a regression.</p> <section id="confirm-failure-cf"> <h2>Confirm Failure (CF)<a class="headerlink" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#confirm-failure-cf" title="Link to this heading"></a></h2> <dl class="simple"> <dt> This tool will give a strong signal when it is applicable. Confirm failure works by running the failing test path 10x in the same browser. We have found that this is the strongest signal compared to other methods, but there are limitations: </dt> <dd> <ul class="simple"> <li><p>This requires a test failure that is discoverable in the failure line.</p></li> <li><p>This requires the failure to be in a supported test harness (web-platform-test, mochitest*, reftest, xpcshell).</p></li> <li><p>Some exceptions exist around specific hardware (e.g. android reftests) or required machines.</p></li> <li><p>Running this can result in infrastructure failure.</p></li> <li><p>Some specific tests do not work well when run in this method.</p></li> <li><p>This launches a CF task for every failure (up to 10) discovered in the failing task, so often you end up with >1 CF job.</p></li> </ul> </dd> </dl> <p>When this runs, a new task is scheduled and typically only takes a few minutes to complete (a retrigger can take 15-90 minutes depending on the task). If you run confirm failure on a failure and there is a lack of test path, unsupported harness or other well known limitations, a retrigger will automatically be scheduled.</p> <p>To launch confirm failure you need to select the task</p> <blockquote> <div> <a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_select_task1.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_select_task1.png" src="../../_images/th_select_task1.png" style="width: 300px;"></a> </div> </blockquote> <p>then click the task action menu</p> <blockquote> <div> <a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_task_action.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_task_action.png" src="../../_images/th_task_action.png" style="width: 300px;"></a> </div> </blockquote> <p>and finally select “Confirm Test Failures”</p> <blockquote> <div> <a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_confirm_failures.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_confirm_failures.png" src="../../_images/th_confirm_failures.png" style="width: 300px;"></a> </div> </blockquote> <p>When the jobs are done, Treeherder will determine if the CF job passes and add a <a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_mitten.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="MITTEN" src="../../_images/th_mitten.png" style="width: 30px;"></a> icon to the original failure if it is intermittent.</p> <p>In the future we are planning to run confirm failure automatically on NEW failures only.</p> </section> <section id="retrigger-r"> <h2>Retrigger (R)<a class="headerlink" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#retrigger-r" title="Link to this heading"></a></h2> <dl class="simple"> <dt> When a retrigger happens the entire task is rerun. This sounds straightforward if a test fails once, rerun it and if it passes it is intermittent. In reality that isn’t the strongest signal, but sometimes it is the best signal. Here are some limitations: </dt> <dd> <ul class="simple"> <li><p>a retrigger has infrastructure failures</p></li> <li><p>a retrigger has a different set of test failures (maybe not even running your test)</p></li> <li><p>a retrigger runs other tests that could influence the state of the browser/machine causing uncertain results</p></li> <li><p>a retrigger can take a long time (many tasks run >30 minutes)</p></li> </ul> </dd> </dl> <p>Given these limitations, a retrigger is a useful tool for gathering more information, just be prepared to double check the logs and failures in more detail if the task doesn’t succeed.</p> <p>To launch a Retrigger, you can: select a task:</p> <blockquote> <div> <a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_select_task1.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_select_task1.png" src="../../_images/th_select_task1.png" style="width: 300px;"></a> </div> </blockquote> <p>click the rotating arrow icon in the task action bar:</p> <blockquote> <div> <a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_retrigger1.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_retrigger1.png" src="../../_images/th_retrigger1.png" style="width: 300px;"></a> <p>OR type the ‘r’ key on your keyboard</p> </div> </blockquote> <p>If a task is retriggered and it is green, the original failing task will have a <a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_mitten.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="MITTEN" src="../../_images/th_mitten.png" style="width: 30px;"></a> icon.</p> </section> <section id="new-annotations-new"> <h2>NEW annotations (NEW)<a class="headerlink" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#new-annotations-new" title="Link to this heading"></a></h2> <p>Treeherder keeps a cache of every failure line seen on Autoland and Mozilla-Central for the last 3 weeks. When a new failure line shows up it is flagged in the task failure summary with a NEW tag. The NEW tag is very successful at finding nearly all regressions, the downside is that many of the NEW tags are seen on intermittents (if an intermittent wasn’t seen recently, or the failure line is slightly different).</p> <p>NEW failures are for all tasks (build, lint, test, etc.) and for all failures (infra failures as well)</p><a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_new.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_new.png" src="../../_images/th_new.png" style="width: 400px;"></a> <dl class="simple"> <dt> On Try server, the NEW annotations are shown, and can act as a way to quickly filter a large number of failing tasks down to a more manageable number. It is best practice to run confirm failure or retrigger on the failing NEW task. To view only failing tasks, you can: </dt> <dd> <p>click on the filter icon for the entire page</p> </dd> </dl><a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_filter.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_filter.png" src="../../_images/th_filter.png" style="width: 300px;"></a> <p>select the field “failure classification” and select the value “new failure not classified”</p><a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_filter_classifications.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_filter_classifications.png" src="../../_images/th_filter_classifications.png" style="width: 300px;"></a> <p>then click “Add”</p><a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_filter_add.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_filter_add.png" src="../../_images/th_filter_add.png" style="width: 300px;"></a> <p>the above will add a <cite>&failure_classification=6</cite> to the url, you can add that manually if you wish.</p> </section> <section id="mitten-icon"> <h2>MITTEN icon<a class="headerlink" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#mitten-icon" title="Link to this heading"></a></h2> <p>Another feature in Treeherder is the <a class="reference internal" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_mitten.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="MITTEN" src="../../_images/th_mitten.png" style="width: 30px;"></a> icon. This is added to an orange job if the job had a retrigger or confirm failure and it is green. This is a great visual shortcut to ignore job failures.</p> </section> <section id="bug-suggestions"> <h2>Bug Suggestions<a class="headerlink" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/treeherder-try/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#bug-suggestions" title="Link to this heading"></a></h2> <p>Built into Treeherder originally is the feature when looking at the “Failure Summary” there will be bug suggestions showing you similar bugs that match the failure.</p><a class="reference internal image-reference" href="https://firefox--source--docs-mozilla-org.translate.goog/_images/th_bug_suggestions.png?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB"><img alt="../../_images/th_bug_suggestions.png" src="../../_images/th_bug_suggestions.png" style="width: 300px;"></a> <p>If there is a <a class="reference external" href="https://firefox--source--docs-mozilla-org.translate.goog/testing/sheriffed-intermittents/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB#single-tracking-bugs">Single Tracking Bug</a>, only that will be shown.</p> <dl class="simple"> <dt> Some caveats to keep in mind: </dt> <dd> <ul class="simple"> <li><p>It is easy to assume if there is a bug that the failure is an intermittent.</p></li> <li><p>this doesn’t tell you if you have made an intermittent a permanent failure.</p></li> <li><p>the bug could be for a different configuration (look at failure history and compare platform, build type, test variant to make sure this failure isn’t spreading)</p></li> <li><p>the bug could be inactive for months.</p></li> </ul> </dd> </dl> </section> </section> </div> </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="Footer"><a href="https://firefox--source--docs-mozilla-org.translate.goog/testing/automated-testing/manifest-sandbox.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB" class="btn btn-neutral float-left" title="Adding Context to manifestparser based Manifests" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a> <a href="https://firefox--source--docs-mozilla-org.translate.goog/testing/sheriffed-intermittents/index.html?_x_tr_sl=pl&_x_tr_tl=iw&_x_tr_hl=en-GB" class="btn btn-neutral float-right" title="Sheriffed intermittent failures" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a> </div> <hr> <div role="contentinfo"> <p></p> </div> Built with <a href="https://translate.google.com/website?sl=pl&tl=iw&hl=en-GB&u=https://www.sphinx-doc.org/">Sphinx</a> using a <a href="https://translate.google.com/website?sl=pl&tl=iw&hl=en-GB&u=https://github.com/readthedocs/sphinx_rtd_theme">theme</a> provided by <a href="https://translate.google.com/website?sl=pl&tl=iw&hl=en-GB&u=https://readthedocs.org">Read the Docs</a>. </footer> </div> </div> </section> </div> <script> jQuery(function () { SphinxRtdTheme.Navigation.enable(true); }); </script> <script>function gtElInit() {var lib = new google.translate.TranslateService();lib.translatePage('pl', 'iw', function () {});}</script> <script src="https://translate.google.com/translate_a/element.js?cb=gtElInit&hl=en-GB&client=wt" type="text/javascript"></script> </body> </html>

CINXE.COM

Understanding Treeherder Results — Firefox Source Docs documentation