i-think Twenty-Two

This is an interesting question that I’ve been pondering over recently. My initial opinion was that if you have some sort of complex logic you should create unit tests to verify the code is sound. I still feel that complex logic in any code warrants unit testing, but I am beginning to wonder whether the need for unit tests may indicate a larger problem: your automation may be too complex.

But the complexity gets even more complex than that. Large suites of integrated tests are quite capable of getting complicated without your help. More abstraction and code reuse certainly makes things easier, but as you add more code paths you increase the likelihood of creating new bugs. Perhaps subtle bugs like using > when you should use >=. Maybe a regular expression isn’t quite right.

Some of these bugs may result in a “false fail” when the test is executed. At this point you need to begin failure analysis tracing the issue which could be either in your tests or in the application under test. Constant failures due to bugs in the test suite erodes confidence in the tests and I have seen failure analysis put off because it is believed to be an issue with the tests.

If you can’t trust your tests, who can you trust?

If you can’t be confident that a failure in a test represents a failure in the product under test your tests lose value. But what if your test passes when it should fail? In this case your test has no value as it has failed its primary purpose, to accurately confirm the application conforms to the test.

A passing test stays off the radar. Its functionality is assumed to work so may be overlooked during manual testing. Eventually the problem may be found, but possibly further down the line than is desirable such as during UAT or worse yet, in production.

By creating unit tests around our more complicated code we can improve our confidence in our own tests. It also sets a good example for the developers working on the application. We can’t expect them to write unit tests for their code if we don’t do the same.

We don’t need to have a unit test around every test (where would it stop?) but we certainly should be looking to at least verify our more complex bits of code.