Catch test case order

问题

Can I guarantee the order of execution with multiple TEST_CASEs with Catch? I am testing some code using LLVM, and they have some despicable global state that I need to explicitly initialize.

Right now I have one test case that's like this:

TEST_CASE("", "") {
    // Initialize really shitty LLVM global variables.
    llvm::InitializeAllTargets();
    llvm::InitializeAllTargetMCs();
    llvm::InitializeAllAsmPrinters();
    llvm::InitializeNativeTarget();
    llvm::InitializeAllAsmParsers();
    // Some per-test setup I can make into its own function
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile(...));
    CHECK_NOTHROW(Compile...));
    CHECK_NOTHROW(Interpret(...));
    CHECK_THROWS(Compile(...));
    CHECK_THROWS(Compile(...));
}

What I want is to refactor it into three TEST_CASE,

• one for tests that should pass compilation,
• one for tests that should fail, and
• one for tests that should pass interpretation (and in the future, further such divisions, perhaps).

But I can't simply move the test contents into another TEST_CASE because if that TEST_CASE is called before the one that sets up the inconvenient globals, then they won't be initialized and the testing will spuriously fail.

回答1:

I'm a bit late to this as I only just saw it - sorry (in future you can post Catch related questions to the Catch forum or issues list on GitHub, if appropriate.

Anyway - I don't know what you did in the end but in this case it sounds like you just want to group each set of assertions into SECTIONs.

TEST_CASE() {
    // Initialize really shitty LLVM global variables.
    llvm::InitializeAllTargets();
    llvm::InitializeAllTargetMCs();
    llvm::InitializeAllAsmPrinters();
    llvm::InitializeNativeTarget();
    llvm::InitializeAllAsmParsers();

    SECTION( "should pass compilation" ) {
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile(...));
      CHECK_NOTHROW(Compile...));
    }
    SECTION( "should pass interpretation" ) {
      CHECK_NOTHROW(Interpret(...));
    }
    SECTION( "Should fail compilation" ) {
      CHECK_THROWS(Compile(...));
      CHECK_THROWS(Compile(...));
    }
}

Each section then acts like an embedded test case (the whole test case is executed from the start - through all the initialisation - for each section). So if one of the no-throws throws it will not prevent the other sections from executing.

... unless the initialisation code should only be executed once - in which case you could either put in a static initialiser, as @paddy suggested (a class that calls the initialisers in its constructor - then just create a global instance) - or you could protect the block of initialisation code with an if on a static bool.

回答2:

If Phil's solution doesn't suit for some reason, here's an alternative:

struct TestFixture {
  static bool _initialised;
  TestFixture() {
    if (!_initialised) {
      llvm::InitializeAllTargets();
      llvm::InitializeAllTargetMCs();
      llvm::InitializeAllAsmPrinters();
      llvm::InitializeNativeTarget();
      llvm::InitializeAllAsmParsers();
      _initialised = true;
    }
  }
};

bool TestFixture::_initialised = false;

TEST_CASE_METHOD(TestFixture, "should pass compilation" ) {
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
  CHECK_NOTHROW(Compile(...));
}

TEST_CASE_METHOD(TestFixture, "should pass interpretation" ) {
  CHECK_NOTHROW(Interpret(...));
}

TEST_CASE_METHOD(TestFixture, "Should fail compilation" ) {
  CHECK_THROWS(Compile(...));
  CHECK_THROWS(Compile(...));
}

In this sample code, it doesn't matter which TEST_CASE runs first because the first-run one will call the llvm initialisation functions and the others will skip this due to the bool.

This code makes use of Catch's Test Fixture support, which we use extensively in my day job: https://github.com/philsquared/Catch/blob/master/docs/test-fixtures.md

来源：https://stackoverflow.com/questions/19694391/catch-test-case-order