Find commit that broke a test without running every test on every commit

问题

I am trying to find a tool that can solve the following problem:

Our entire test suite takes hours to runs which often makes it difficult or at least very time consuming to find out which commit broke a specific test since there may be 50 to 200 commits in between test runs. At any given time there are only very few broken tests, so rerunning only the broken tests is very fast compared to running the entire test suite.

Is there a tool, e.g. a continuous integration server, that can rerun failed tests with a couple of revisions in between the last revision where the test was OK and the first revision where the test was not OK and therefore automatically find out on which specific commit a test switched from successful to broken.

For example:

Test A and B are ok in revision 100. Test A and B are broken in revision 200.

The tool should now run both tests with revision 150. Then if e.g. test A was broken and test B was OK in revision 150, it could continue to check test A with revision 125 and test B with revision 175 and so on until every broken test can be explained by some specific commit.

For a single test, I could probably hack something together with git bisect. But for multiple failed test this is probably not sufficient, since we need to search in both directions for many revisions.

回答1:

git

Are you using git or mercurial (see: What is Mercurial bisect good for?)?

Suppose version 2.6.18 of your project worked, but the version at "master" crashes. Sometimes the best way to find the cause of such a regression is to perform a brute-force search through the project's history to find the particular commit that caused the problem. The git bisect command can help you do this:[...]

From: Finding Issues - Git Bisect.

Essentially you mark two revisions as start and end and hit:

$ git bisect

Then run the test and depending on whether it passed or failled call

$ git bisect good

or

$ git bisect bad

Respectively. git will do binary search, always cutting remaining revisions in half. I guess you can script it easily. If you are using svn, git can easily import the whole repository.

Building single revision at a time

This is not an answer but an advice - just test every single commit! Today you can cluster continuous integration servers easily, with a farm of servers it is not that hard to test all commits.

来源：https://stackoverflow.com/questions/10043816/find-commit-that-broke-a-test-without-running-every-test-on-every-commit