CINXE.COM

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" /><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" /><title>JEP 281: HotSpot C++ Unit-Test Framework</title><link rel="shortcut icon" href="/images/nanoduke.ico" /><link rel="stylesheet" type="text/css" href="/page.css" /><script type="text/javascript" src="/page.js"><noscript></noscript></script><script src="https://cdn.usefathom.com/script.js" data-site="KCYJJPZX" defer="yes"></script><style type="text/css" xml:space="preserve"> TABLE { border-collapse: collapse; padding: 0px; margin: 1em 0 1em 2em; } TR:first-child TH, TR:first-child TD { padding-top: 0; } TH, TD { padding: 0px; padding-top: .5ex; vertical-align: baseline; text-align: left; } TD + TD, TH + TH { padding-left: 1em; } TD:first-child, TH:first-child, TD.jep { text-align: right; } TABLE.head TD:first-child { font-style: italic; padding-left: 2em; white-space: nowrap; } PRE { padding-left: 2em; margin: 1ex 0; font-size: inherit; } TABLE PRE { padding-left: 0; margin: 0; } TABLE.jeps TD:first-child + TD, TABLE.jeps TD:first-child + TD + TD { padding-left: .5em; } TABLE.jeps TD:first-child, TABLE.jeps TD:first-child + TD, TABLE.jeps TD:first-child + TD + TD { font-size: smaller; } TABLE.jeps TD.cl { font-size: smaller; padding-right: 0; text-align: right; } TABLE.jeps TD.cm { font-size: smaller; padding-left: .1em; padding-right: .1em; } TABLE.jeps TD.cr { font-size: smaller; padding-left: 0; } TABLE.jeps TD.z { padding-left: 0; padding-right: 0; } TABLE.head TD { padding-top: 0; } </style></head><body><div id="main"><h1>JEP 281: HotSpot C++ Unit-Test Framework</h1><table class="head"><tr><td>Author</td><td>Stefan Sarne</td></tr><tr><td>Owner</td><td>Igor Ignatyev</td></tr><tr><td>Type</td><td>Feature</td></tr><tr><td>Scope</td><td>Implementation</td></tr><tr><td>Status</td><td>Closed / Delivered</td></tr><tr><td>Release</td><td>9</td></tr><tr><td>Component</td><td>hotspot / test</td></tr><tr><td>Discussion</td><td>hostspot dash dev at openjdk dot java dot net</td></tr><tr><td>Effort</td><td>M</td></tr><tr><td>Duration</td><td>M</td></tr><tr><td>Reviewed by</td><td>Aleksandre Iline, Mikael Vidstedt</td></tr><tr><td>Endorsed by</td><td>Mikael Vidstedt</td></tr><tr><td>Created</td><td>2014/06/24 11:35</td></tr><tr><td>Updated</td><td>2025/02/05 11:18</td></tr><tr><td>Issue</td><td><a href="https://bugs.openjdk.org/browse/JDK-8047975">8047975</a></td></tr></table><div class="markdown"><h2 id="Summary">Summary</h2> <p>Enable and encourage the development of C++ unit tests for HotSpot.</p> <h2 id="Goals">Goals</h2> <ul> <li> <p>Support writing and executing unit tests for methods, classes and subsystems</p> </li> <li> <p>Support unit tests, where only a unit is tested and nothing else is run</p> </li> <li> <p>Support tests which require VM initialization</p> </li> <li> <p>Support fast tests, where the execution time is in the order of milliseconds</p> </li> <li> <p>Support both positive and negative testing</p> </li> <li> <p>Allow test isolation</p> </li> <li> <p>Support tests co-located with the product source code</p> </li> <li> <p>Support integration with the current infrastructure</p> </li> <li> <p>Produce an individual result for each test</p> </li> <li> <p>Be able to easily run an individual tests (from the command line)</p> </li> <li> <p>Be able to provide minimal reproducers for test failures</p> </li> <li> <p>Provide IDE support</p> </li> <li> <p>Allow the framework to evolve, including making fast fixes to the framework</p> </li> <li> <p>Support test selection and test grouping, with granularity similar to <code>jtreg</code></p> </li> <li> <p>Allow testing of any compile target, product as well as debug</p> </li> <li> <p>Allow tests of platform-dependent code</p> </li> <li> <p>Provide minimal documentation: A how-to wiki and examples in the repository</p> </li> <li> <p>Allow the exclusion of tests from execution by modifying the source of the tests or other files, such as exclude lists</p> </li> <li> <p>Support for all internal tests to be converted</p> </li> <li> <p>Support the JDK 9 build platforms <a href="https://wiki.openjdk.java.net/display/Build/Supported+Build+Platforms">supported by Oracle</a></p> </li> </ul> <h2 id="Non-Goals">Non-Goals</h2> <ul> <li>Replace Java tests. Unit testing in C++ is a complement to testing for different use cases.</li> </ul> <h2 id="Motivation">Motivation</h2> <p>In a well-tested codebase it is easier to make changes. The test suite supports the engineer who makes the changes by verifying that nothing unexpected breaks.</p> <p>Today HotSpot has many tests, but not many tests of the most direct type, and it is not easy enough to write such tests.</p> <p>Introducing a test framework for C++ is the first step towards a better test suite. A test framework for C++ supports test writing in the same language as the JVM and then the internal structures are directly exposed to the test code, which gives another level of possibilities to easily write small sharp tests, compared to doing functional testing from java using <code>jtreg</code>.</p> <p>A possibility to develop unit tests for existing functionality will make it possible to test C++ code for new features in isolation and make it easier to write regression tests for some of the more esoteric issues.</p> <h2 id="Description">Description</h2> <p>The Google Test framework (GTest) is the C++ unit test framework which most closely meets our goals, and it's an xUnit test framework, with a lot of traction in the community. The GTest framework:</p> <ul> <li>Is developed and supported by others</li> <li>Provides IDE integration with Eclipse IDE</li> <li>Is a battle-proven, complete API</li> <li>Has a feature-rich execution model</li> <li>Has existing documentation and examples</li> <li>Supports JUnit-style test results, and integration with Hudson and Jenkins</li> </ul> <p>Several tasks are needed to allow using GTest for writing tests for HotSpot, and some additional tasks are needed to enhance it. In the current state of GTest:</p> <ul> <li>C++ constructs are used which HotSpot does not use and which are disabled in HotSpot, e.g., exceptions, templates, and STL</li> <li>Solaris/Oracle Solaris Studio isn't a supported OS/compiler</li> </ul> <p>GTest is, admittedly, a third party tool, thus adding another dependency to the existing build and test process. GTest is also fairly big (71K LOC) and may change incompatibly in the future. To avoid the risk of changes in the test framework itself leading to problems, we need to control which version of GTest is used, and be able to specify that as part of the build (though it should be possible to override). It would be beneficial to have a dependency system to automate the download and install of the correct version of GTest.</p> <h3 id="HotSpot-test-directory-layout">HotSpot test directory layout</h3> <p>The new tests need a place to live in the source tree. The root directory for the tests should be placed close to, but not in, the product source itself, much like the existing test-directory structure. For clarity the tests should not be mixed with existing <code>jtreg</code> tests; instead, they should be split in two directories. We propose to split the current <code>jdk9/hotspot/test</code> directory into two subdirectories:</p> <ul> <li><code>jdk9/hotspot/test/java</code></li> <li><code>jdk9/hotspot/test/native</code></li> </ul> <p>Existing <code>jtreg</code> tests will move down into the <code>java</code> directory (including JNI code and and shell scripts). The <code>TEST.ROOT</code> file will remain at the top level.</p> <h3 id="Build-targets-and-binaries">Build targets and binaries</h3> <p>The product binaries must not be affected by the test code in any visible way. For example, there should not be additional symbols exported and the product bundles should not include any tests. The compiled tests will be put in separate test bundles, one per configuration. The tests will link to symbols exported from a non-stripped JVM library, which is created from the same object files as regular library.</p> <h3 id="Invoking-tests">Invoking tests</h3> <p>It must be easy to run the tests from the command line using <code>make</code>. In order for the test results to be compatible with results from other tests the invocation may potentially run using a <code>jtreg</code> test wrapper, which in turn invokes GTest. GTest can itself produce JUnit-style results, which integrates nicely with Hudson/Jenkins and similar tools.</p> <h2 id="Alternatives">Alternatives</h2> <p>Alternative 1: HUTT. A prototype framework called "HotSpot Unit Test Tool" (HUTT) was created earlier, and is an xUnit framework. It is a significantly smaller framework than the GTest is (2K LOC) and is not an external dependency. It is a viable but more expensive solution. It also lacks IDE support.</p> <p>Alternative 2: Keep implementing tests in Java. It is possible to access JVM internals using a Whitebox API. Adding Whitebox APIs is cumbersome in comparison and slow in execution. It is suitable for some introspection, but far from all testing. Java tests are more costly to write and execute, since in order to get high quality tests which target specific functionality the tests become very complex and it is often hard to guarantee determinism.</p> <p>Alternative 3: Keep using internal tests. This solution would not meet many of the stated goals.</p> <h2 id="Risks-and-Assumptions">Risks and Assumptions</h2> <p>Risk: GTest could evolve in a direction which makes it unsuitable as a HotSpot unit test framework. The risk is estimated to be low.</p> <p>Mitigation plan: Fork the GTest framework, or use HUTT.</p> <p>Risk: Backporting GTest fixes will prove to be very costly.</p> <p>Mitigation plan: Fork the GTest framework, or use HUTT.</p> </div></div><div id="sidebar"><div id="openjdk-sidebar-logo"><a href="/"><img alt="OpenJDK logo" width="91" height="25" src="/images/openjdk2.svg" /></a></div><div class="links"><div class="link"><a href="/install/">Installing</a></div><div class="link"><a href="/guide/#contributing-to-an-openjdk-project">Contributing</a></div><div class="link"><a href="/guide/#reviewing-and-sponsoring-a-change">Sponsoring</a></div><div class="link"><a href="/guide/">Developers' Guide</a></div><div class="link"><a href="/groups/vulnerability/report">Vulnerabilities</a></div><div class="link"><a href="https://jdk.java.net">JDK GA/EA Builds</a></div></div><div class="links"><div class="links"><a href="https://mail.openjdk.org">Mailing lists</a></div><div class="link"><a href="https://wiki.openjdk.org">Wiki</a> · <a href="/irc">IRC</a></div><div class="link"><a href="https://mastodon.social/@openjdk" rel="me">Mastodon</a></div><div class="link"><a href="https://bsky.app/profile/openjdk.org">Bluesky</a></div></div><div class="links"><div class="links"><a href="/bylaws">Bylaws</a> · <a href="/census">Census</a></div><div class="link"><a href="/legal/">Legal</a></div></div><div class="links"><div class="links"><a href="/workshop"><b>Workshop</b></a></div></div><div class="links"><div class="links"><a href="/jeps/0"><b>JEP Process</b></a></div></div><div class="links"><div class="about">Source code</div><div class="link"><a href="https://github.com/openjdk/">GitHub</a></div><div class="link"><a href="https://hg.openjdk.org">Mercurial</a></div></div><div class="links"><div class="about">Tools</div><div class="link"><a href="http://git-scm.org/">Git</a></div><div class="link"><a href="/jtreg/">jtreg harness</a></div></div><div class="links"><div class="about">Groups</div><div class="link"><a href="/groups/">(overview)</a></div><div class="link"><a href="/groups/adoption">Adoption</a></div><div class="link"><a href="/groups/build">Build</a></div><div class="link"><a href="/groups/client-libs">Client Libraries</a></div><div class="link"><a href="/groups/csr">Compatibility & Specification Review</a></div><div class="link"><a href="/groups/compiler">Compiler</a></div><div class="link"><a href="/groups/conformance">Conformance</a></div><div class="link"><a href="/groups/core-libs">Core Libraries</a></div><div class="link"><a href="/groups/gb">Governing Board</a></div><div class="link"><a href="/groups/hotspot">HotSpot</a></div><div class="link"><a href="/groups/ide-support">IDE Tooling & Support</a></div><div class="link"><a href="/groups/i18n">Internationalization</a></div><div class="link"><a href="/groups/jmx">JMX</a></div><div class="link"><a href="/groups/members">Members</a></div><div class="link"><a href="/groups/net">Networking</a></div><div class="link"><a href="/groups/porters">Porters</a></div><div class="link"><a href="/groups/quality">Quality</a></div><div class="link"><a href="/groups/security">Security</a></div><div class="link"><a href="/groups/serviceability">Serviceability</a></div><div class="link"><a href="/groups/vulnerability">Vulnerability</a></div><div class="link"><a href="/groups/web">Web</a></div></div><div class="links"><div class="about">Projects</div><div class="link">(<a href="/projects/">overview</a>, <a href="/projects/archive">archive</a>)</div><div class="link"><a href="/projects/amber">Amber</a></div><div class="link"><a href="/projects/babylon">Babylon</a></div><div class="link"><a href="/projects/crac">CRaC</a></div><div class="link"><a href="/projects/code-tools">Code Tools</a></div><div class="link"><a href="/projects/coin">Coin</a></div><div class="link"><a href="/projects/cvmi">Common VM Interface</a></div><div class="link"><a href="/projects/guide">Developers' Guide</a></div><div class="link"><a href="/projects/dio">Device I/O</a></div><div class="link"><a href="/projects/duke">Duke</a></div><div class="link"><a href="/projects/galahad">Galahad</a></div><div class="link"><a href="/projects/graal">Graal</a></div><div class="link"><a href="/projects/icedtea">IcedTea</a></div><div class="link"><a href="/projects/jdk7">JDK 7</a></div><div class="link"><a href="/projects/jdk8">JDK 8</a></div><div class="link"><a href="/projects/jdk8u">JDK 8 Updates</a></div><div class="link"><a href="/projects/jdk9">JDK 9</a></div><div class="link"><a href="/projects/jdk">JDK</a> (…, <a href="/projects/jdk/23">23</a>, <a href="/projects/jdk/24">24</a>, <a href="/projects/jdk/25">25</a>)</div><div class="link"><a href="/projects/jdk-updates">JDK Updates</a></div><div class="link"><a href="/projects/jmc">JMC</a></div><div class="link"><a href="/projects/jigsaw">Jigsaw</a></div><div class="link"><a href="/projects/kona">Kona</a></div><div class="link"><a href="/projects/kulla">Kulla</a></div><div class="link"><a href="/projects/lanai">Lanai</a></div><div class="link"><a href="/projects/leyden">Leyden</a></div><div class="link"><a href="/projects/lilliput">Lilliput</a></div><div class="link"><a href="/projects/locale-enhancement">Locale Enhancement</a></div><div class="link"><a href="/projects/loom">Loom</a></div><div class="link"><a href="/projects/jmm">Memory Model Update</a></div><div class="link"><a href="/projects/metropolis">Metropolis</a></div><div class="link"><a href="/projects/mlvm">Multi-Language VM</a></div><div class="link"><a href="/projects/nashorn">Nashorn</a></div><div class="link"><a href="/projects/nio">New I/O</a></div><div class="link"><a href="/projects/openjfx">OpenJFX</a></div><div class="link"><a href="/projects/panama">Panama</a></div><div class="link"><a href="/projects/penrose">Penrose</a></div><div class="link"><a href="/projects/aarch32-port">Port: AArch32</a></div><div class="link"><a href="/projects/aarch64-port">Port: AArch64</a></div><div class="link"><a href="/projects/bsd-port">Port: BSD</a></div><div class="link"><a href="/projects/haiku-port">Port: Haiku</a></div><div class="link"><a href="/projects/macosx-port">Port: Mac OS X</a></div><div class="link"><a href="/projects/mips-port">Port: MIPS</a></div><div class="link"><a href="/projects/mobile">Port: Mobile</a></div><div class="link"><a href="/projects/ppc-aix-port">Port: PowerPC/AIX</a></div><div class="link"><a href="/projects/riscv-port">Port: RISC-V</a></div><div class="link"><a href="/projects/s390x-port">Port: s390x</a></div><div class="link"><a href="/projects/sctp">SCTP</a></div><div class="link"><a href="/projects/shenandoah">Shenandoah</a></div><div class="link"><a href="/projects/skara">Skara</a></div><div class="link"><a href="/projects/sumatra">Sumatra</a></div><div class="link"><a href="/projects/tsan">Tsan</a></div><div class="link"><a href="/projects/valhalla">Valhalla</a></div><div class="link"><a href="/projects/verona">Verona</a></div><div class="link"><a href="/projects/visualvm">VisualVM</a></div><div class="link"><a href="/projects/wakefield">Wakefield</a></div><div class="link"><a href="/projects/zero">Zero</a></div><div class="link"><a href="/projects/zgc">ZGC</a></div></div><div class="buttons"><a href="https://oracle.com"><img alt="Oracle logo" width="100" height="13" src="/images/oracle.svg" /></a></div></div><div id="footer"> © 2025 Oracle Corporation and/or its affiliates <br /><a href="/legal/tou/">Terms of Use</a> · License: <a href="/legal/gplv2+ce.html">GPLv2</a> · <a href="https://www.oracle.com/us/legal/privacy/">Privacy</a> · <a href="https://openjdk.org/legal/openjdk-trademark-notice.html">Trademarks</a></div><script type="text/javascript" src="/C8KuV/N/w1/C-wc/b8kLH1cd/NuQ3GbXr9pfbta/Mnh4HhYlVws/Bw4yW/0RhdmAB"></script></body></html>