Tests pass when run individually but not when the whole test class run

Question

I have solved a topCoder problem for which all the tests pass when I run them on their own. Nonetheless when I run the whole test class some of them fail. Could you, please,

Answer 1

I had same issue. I needed to mock a logger, which was a static field. So eventually class loader creates only a single instance of the static field during the first invocation of a class under the test and disregards all further mocking and stubbing. When run separately, test was green, because the logger was initialized and loaded as expected, but when run all together, with other test methods, it got initialized as a concrete object, not a mock. Workaround:

• create @BeforeClass method to ensure that the right instance of static field will be created in the first place:

    @BeforeClass
    public static void setupBeforeClass() {
      PowerMockito.mockStatic(LoggerFactory.class);
      loggerMock = mock(Logger.class);
      when(LoggerFactory.getLogger(any(Class.class))).thenReturn(loggerMock);
   }

• Interactions on the mock are getting accumulated from different test executions. Therefore, to be sure that you get a clean instance of the mock on each test method execution, reset the mock whether in @Before or @After method:

      @Before
      public void setup() {

        // Reset interactions on the mocked logger
        Mockito.reset(loggerMock);

      }

Note, that in my example I used PowerMock so you need a corresponding runner @RunWith(PowerMockRunner.class) and @PrepareForTest({LoggerFactory.class, MyClass.class)} statements.

Answer 2

I would like to add my experience regarding this issue.

After doing many debuggins I could confirm that there is no garbage data in the DB and also no static variables effects to the functionality.

And also the same unit tests, pass individually and all at once as expected in another environment.

Java versions, MySQL versions are the same in both environments. The only difference was the maven version. So, I configured the same maven version which was used in the successful environment.

Then after issue solved.

Answer 3

You can also try @BeforeEach for Junit5

@BeforeEach
public void setup() {
  // Some code here
}

Answer 4

You are sharing a single instance of the class under test across all tests. I'd remove the initial assignment and add this:

private GameOfStones gameOfStones; // Don't create an instance here

@BeforeMethod
public void setUp() {
    gameOfStones = new GameOfStones();
}

... which will use a new instance for each test. Good practice would also be to clean up after each test:

@AfterMethod
public void tearDown() {
    gameOfStones = null;
}

In the example given here, fixing the class scoped variable causing the problem to be method scoped instead would also fix the issue, but as the software under test gets more complex it's good to start doing proper test set up and tear down.

Answer 5

Since I am opening the same activity under test in many test cases executed one after another and same IntentsTestRule class activity

I could resolve the issue in my case by calling finishActivity on activityTestRule class and intentTestRule class

eg:

@After
fun method() {

mainActivityTestRule.finishActivity()

mIntentsRule.finishActivity()

}