What mutation-testing frameworks exist for C/C++?

Question

Mutation testing has been out there for a while now, and it seems there are at least one or two commercial mutation testing frameworks for C/C++. Have you used them? What are yo

Answer 1

A straight forward python script for mutating c programs is available at:

https://github.com/parunbabu/mutate.py

the author says it works better if the code under test is de-commented and indented.

and it is also free and opensource ... i think this is what you are looking for.

Answer 2

Notice that Parasoft's tool only generate equivalent mutations. That echoes the problem described on Wikipedia article about Mutation Testing - it is hard to distinguish between equivalent and non-equivalent mutations so they decided to stick with equivalent.

I tried another interesting tool that can automatically discover invariants in instrumented C and C++ code - it is called "Daikon". Essentially it is doing same thing as tool that generates equivalent mutations, but instead of identifying problematic code it gives you a set of invariants such as "A == B + 1". I think invariants are more useful because when you look at discovered invariant it gives you assurance that your code is correct if invariant make sense, and then you can convert invariants into asserts and that gives you more confidence when you change code.

Answer 3

• Mull is LLVM-based and seems to be actively developed and easy to use.
• dextool mutate also LLVM-based and actively developed, more complicated to use but has more features like re-running alive mutants and only mutate introduced changes based on a git diff

Answer 4

I have no experience with it but Mutate++ seems to be an option that is missing from the ones already mentioned.

Mutate++ - A C++ Mutation Test Environment

Answer 5

The existing frameworks where way too time-consuming to set up and use so I did my own implementation, a quick and easy solution that should work on any machine. There is binaries available for MacOSX, Windows and RaspberryPi (Linux):

https://github.com/RagnarDa/dumbmutate

Hope it helps anyone!

Answer 6

A brief search resulted in:

• PlexTest: http://www.itregister.com.au/products/plextest_detail.htm
• Insure++: http://www.parasoft.com/jsp/products/insure.jsp;jsessionid=baacpvbaDywLID?itemId=63
• MILU (may be only for C): http://www.dcs.kcl.ac.uk/pg/jiayue/milu/

With that said, you need to realize that mutation testing isn't particularly useful (at least from some stuff I've previously read). It's an interesting tool when faced with hard (metaphorically speaking) asserts and for making sure that data requirements are heeded to (when dealing with if and only if situations).

In my opinion, there are much more established ways of analyzing the robustness of code.