Testing for assert in the Boost Test framework

Question

I use the Boost Test framework to unit test my C++ code and wondered if it is possible to test if a function will assert? Yes, sounds a bit strange but bear with me! Many of m

Answer 1

Sorry, but you're attacking your problem the wrong way.

"assert" is the spawn of the devil (a.k.a. "C") and is useless with any language that has proper exceptions. It's waaaaaay better to reimplement an assert-like functionality with exceptions. This way you actually get a chance of handling errors the right way (incl proper cleanup procedures) or triggering them at will (for unit testing).

Besides, if your code ever runs in Windows, when you fail an assertion you get a useless popup offering you to debug/abort/retry. Nice for automated unit tests.

So do yourself a favor and re-code an assert function that throws exceptions. There's one here: How can I assert() without using abort()?

Wrap it in a macro so you get _ _FILE _ _ and _ _ LINE _ _ (useful for debug) and you're done.

Answer 2

There are two kinds of errors I like to check for: invariants and run-time errors.

Invariants are things that should always be true, no matter what. For those, I use asserts. Things like you shouldn't be passing me a zero pointer for the output buffer you're giving me. That's a bug in the code, plain and simple. In a debug build, it will assert and give me a chance to correct it. In a retail build, it will cause an access violation and generate a minidump (Windows, at least in my code) or a coredump (Mac/unix). There's no catch that I can do that makes sense to deal with dereferencing a zero pointer. On Windows catch (...) can suppress access violations and give the user a false sense of confidence that things are OK when they've already gone horribly, horribly wrong.

This is one reason why I've come to believe that catch (...) is generally a code smell in C++ and the only reasonable place where I can think of that being present is in main (or WinMain) right before you generate a core dump and politely exit the app.

Run-time errors are things like "I can't write this file because of permissions" or "I can't write this file because the disk is full". For these sorts of errors throwing an exception makes sense because the user can do something about it like change the permission on a directory, delete some files or choose an alternate location to save the file. These run-time errors are correctable by the user. A violation of an invariant can't be corrected by the user, only by a programmer. (Sometimes the two are the same, but typically they aren't.)

Your unit tests should force code to throw the run-time error exceptions that your code could generate. You might also want to force exceptions from your collaborators to ensure that your system under test is exception safe.

However, I don't believe there is value in trying to force your code to assert against invariants with unit tests.

Answer 3

I don't think so. You could always write your own assert which throws an exception and then use BOOST_CHECK_NOTHROW() for that exception.

Answer 4

At work I ran into the same problem. My solution is to use a compile flag. When my flag GROKUS_TESTABLE is on my GROKUS_ASSERT is turned into an exception and with Boost you can test code paths that throw exceptions. When GROKUS_TESTABLE is off, GROKUS_ASSERT is translated to c++ assert().

#if GROKUS_TESTABLE
#define GROKUS_ASSERT ... // exception
#define GROKUS_CHECK_THROW    BOOST_CHECK_THROW
#else
#define GROKUS_ASSERT ... // assert
#define GROKUS_CHECK_THROW(statement, exception)  {}  // no-op
#endif

My original motivation was to aid debugging, i.e. assert() can be debugged quickly and exceptions often are harder to debug in gdb. My compile flag seems to balance debuggability and testability pretty well.

Hope this helps

Answer 5

I think this question, and some of replies, confuse run-time errors detection with bug detection. They also confuse intent and mechanism.

Run-time error is something that can happen in a 100% correct program. It need detection, and it needs proper reporting and handling, and it should be tested. Bugs also happen, and for programmer's convenience it's better to catch them early using precondition checks or invariant checks or random assert. But this is programmer's tool. The error message will make no sense for ordinary user, and it does not seem reasonable to test function behaviour on the data that properly written program will never pass to it.

As for intent and mechanism, it should be noted that exception is nothing magic. Some time ago, Peter Dimov said on Boost mailing list (approximately) that "exceptions are just non-local jump mechanism". And this is very true. If you have application where it's possible to continue after some internal error, without the risk that something will be corrupted before repair, you can implement custom assert that throws C++ exception. But it would not change the intent, and won't make testing for asserts much more reasonable.

Answer 6

Having the same problem, I digged through the documentation (and code) and found a "solution".

The Boost UTF uses boost::execution_monitor (in <boost/test/execution_monitor.hpp>). This is designed with the aim to catch everything that could happen during test execution. When an assert is found execution_monitor intercepts it and throws boost::execution_exception. Thus, by using BOOST_REQUIRE_THROW you may assert the failure of an assert.

so:

#include <boost/test/unit_test.hpp>
#include <boost/test/execution_monitor.hpp>  // for execution_exception

BOOST_AUTO_TEST_CASE(case_1)
{
  BOOST_REQUIRE_THROW(function_w_failing_assert(),
                      boost::execution_exception);
}

Should do the trick. (It works for me.)

However (or disclaimers):

• It works for me. That is, on Windows XP, MSVC 7.1, boost 1.41.0. It might be unsuitable or broken on your setup.

• It might not be the intention of the author of Boost Test. (although it seem to be the purpose of execution_monitor).

• It will treat every form of fatal error the same way. I e it could be that something other than your assert is failing. In this case you could miss e g a memory corruption bug, and/or miss a failed failed assert.

• It might break on future boost versions.

• I expect it would fail if run in Release config, since the assert will be disabled and the code that the assert was set to prevent will run. Resulting in very undefined behavior.

• If, in Release config for msvc, some assert-like or other fatal error would occur anyway it would not be caught. (see execution_monitor docs).

• If you use assert or not is up to you. I like them.

See:

• http://www.boost.org/doc/libs/1_41_0/libs/test/doc/html/execution-monitor/reference.html#boost.execution_exception

• the execution-monitor user-guide.

Also, thanks to Gennadiy Rozental (Author of Boost Test), if you happen to read this, Great Work!!