Visualization of CTest Test Times

For effective software development, the edit-compile-test cycle should run quickly.  In order to get a better understanding of ITK‘s test times, we created a Python script that visualizes CTest test times with VTK‘s TreeMapView.


ITK test times.

Tests that run longer have a proportionally larger area.  With the visualization, it is clear that there are some tests that take significantly longer than most others.  It is possible to zoom in and zoom out on tests and identify their names with a mouse hover or click.

After identifying the long running tests, we added a CTest label to them. 

ITK tests without RUNS_LONG label (top) and with RUNS_LONG label (bottom).

Tests with the RUNS_LONG label constituted less than 1% of the tests, but they take approximately half of the time to run the test suite!

It is possible to also organize the visualization by ITK module label.

ITK test times organized by module.

Most of the long running tests identified were integration tests, as opposed to unit tests, and tests for advanced algorithms on semi-realistic data.  The decrease in code coverage after excluding the tests was 0.25%.  To increase the quickness of the edit-build-test cycle, it is possible to exclude the long running tests locally with

  ctest -LE RUNS_LONG

For dashboard build platforms that take so long to run that their maintainers may prefer not to run them at all (we’re looking at you, Visual Studio in Debug!), the tests can be excluded with 

  set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)

Of course, these approaches should only be taken after it is determined that it is not feasible to refactor the code or the tests so they run more quickly.

The script is generally applicable on any system that has VTK Python wrapping installed and uses CTest.  For example, screen captures of VTK and CMake test suites are shown below.

VTK test times.

 

CMake test times.

3 Responses to Visualization of CTest Test Times

  1. Pingback: Jean-Christophe Fillion-Robin

  2. Pingback: Jeff Baumes

  3. Pingback: Matthew McCormick

Questions or comments are always welcome!

X