CTest & CDash add support for new dynamic analysis tools

CTest and CDash now support the new suite of "sanitizer" dynamic analysis tools that are available for gcc and clang!  Support for these tools will be included in the upcoming release of CMake 3.1.  They can be found in the first release candidate CMake 3.1-rc1, or you can checkout CMake master from git and build it from source.

These new features were added as part of the Google Project Tango effort to create mobile 3D scanning devices.  The sanitizer tools allow for low overhead run time error checking similar to what you can find in valgrind. However, instead of running a machine code level simulator to detect the errors, the compiler based sanitizers add extra error checks that are compiled into the code. This provides better performance. There is one price that is paid for that performance gain. Once the first error is detected, the program halts. 

This blog will provide examples of how to use CTest and CDash to run sanitized test programs and collect and display the results. In order to make it easier for you to replicate the examples below, all of the sample code is available as an attachment to this blog post.

Use of the sanitizer tools requires building your code with extra compile flags that inject the error checking into the final executable. There are several ways to do that. You can set CMAKE_C_FLAGS and CMAKE_CXX_FLAGS on the command line of cmake or by preloading the cache with values in a ctest script.

 

ThreadSanitizer

ThreadSanitizer (TSan) allows us to detect data races in our programs.

 

Example code

Here's a simple example of a program with a data race:

 

Building with TSan

All we need to do to test this program with ThreadSanitizer is add some flags to the C and CXX flags used to build the project. The flags required are:

 

Example Output

After we build this executable and run it, we see the following output from TSan:

 

Submitting to CDash

Here's a CTest script that you can use to run this example and submit the results to CDash's public dashboard.

Other than the compile flags mentioned earlier, of particular note here is the line

This is what tells CTest how to intrepret the output of the dynamic analysis (memcheck) step.

After you've submitted your results to CDash, you should see a new row in the Dynamic Analysis section of the page.  Here's an example of what this will look like:

When you click on the "1" (under Defect Count) you'll be taken to a page displaying the following information:

From here, if you click on the name of the test (simple_race) you will see the output of the sanitizer tool.

 

AddressSanitizer

AddressSanitizer (asan) allows us to detect buffer overflows and cases where memory is read after being freed.

 

Example code

Here's an example C program that attempts to read some memory after freeing it:

 

Building with ASan

Similarly to the TSan example, we enable ASan dynamic analysis by modifying the flags that we use to compile our program:

 

Example output

Here is an example of the output you should expect when ASan detects a defect in your program:

 

Submitting to CDash

To submit these results to CDash, you can use a script very similar to the one above in the TSan example.  The only important difference is that instead of:

you should specify:

and the cache should will be:

Here are a couple views of these results from CDash:

 

MemorySanitizer

MemorySanitizer (msan) allows us to detect reads of uninitialized memory.

 

Example code

In this example, our program reads from memory before it has been initialized:

 

Building with ASan

As before, we simply need to modify the compile flags for the executable that we wish to test.  In this case, we specify -fsanitize=memory instead of =thread or =address.

 

Example output

Here is the output you should expect if you run the example above through MSan:

 

Submitting to CDash

You can follow the pattern outlined above for TSan.  The important change is to specify

instead of:

and the cache should have:

 

Finally, here's what these results look like on CDash:

 

That's all there is to it!  I hope that you found these examples useful.  With the use of these new tools, you can be more confident that there aren't any nasty memory defects lurking in your codebase.  If you have any questions or comments about these new features of CTest & CDash, please contact us on the CMake users mailing list.

7 Responses to CTest & CDash add support for new dynamic analysis tools

  1. It does not work!! It just do not compile any thing. Any real example on github????

  2. Xavier says:

    In the CTest script, you call ctest_test() and ctest_memcheck(). Does it mean that the tests are run twice?

  3. Bill Hoffman says:

    Yes, it will call it once with memcheck and one without. Although now that you mention it this does not make as much sense with compiled in memory checking. If you leave out the ctest_test the test column will be empty on CDash, but it should work fine.

  4. Xavier says:

    Thanks for the clarification!

Questions or comments are always welcome!