Dependencies analysis tool - updating regression test cases

Question

Problem

Its quite a common problem I would like to think. Adding new code translates into regression - existing test cases become obsolete. Dependencie

Answer 1

You can figure out which tests are relevant by tracking what code they touch. You can track what code they touch by using test coverage tools.

Most test coverage tools build an explicit set of locations that a running test executes; that's the point of "coverage". If you organize your test running to execute one unit test at a time, and then take a snapshot of the coverage data, then you know for each test what code it covers.

When code is modified, you can determine the intersection of the modified code and what an individual test covers. If the intersection is non-empty, you surely need to run that test again, and likely you'll need to update it.

There are a several practical problems with making this work.

First, it is often hard to figure out how the test coverage tools record this positioning data. Second, you have to get the testing mechanism to capture it on per test basis; that may be awkward to organize, and/or the coverage data may be clumsy to extract and store. Third, you need to compute the intersection of "code modified" with "code covered" by a test; this is abstractly largely a problem in intersecting big bit vectors. Finally, capturing "code modified" is bit tricky because sometimes code moves; if location 100 in a file was tested, and the line moves in the file, you may not get comparable position data. That will lead to false positives and false negatives.

There are test coverage tools which record the coverage data in an easily captured form, and can do the computations. Determine code changes is trickier; you can use diff but the moved code will confuse the issue somewhat. You can find diff tools that compare code structures, which identify such moves, so you can get better answers.

If you have source code slicers, you could compute how the output tested is (backward slice-)dependent on the input; all code in that slice affects the test, obviously. I think the bad news here is that such slicers are not easy to get.