97 Things Every Programmer Should Know

Chapter 80. Test for Required Behavior, Not Incidental Behavior

Kevlin Henney

A COMMON PITFALL IN TESTING is to assume that exactly what an implementation does is precisely what you want to test for. At first glance, this sounds more like a virtue than a pitfall. Phrased another way, however, the issue becomes more obvious: a common pitfall in testing is to hardwire tests to the specifics of an implementation, where those specifics are incidental and have no bearing on the desired functionality.

When tests are hardwired to implementation incidentals, changes to the implementation that are actually compatible with the required behavior may cause tests to fail, leading to false positives. Programmers typically respond either by rewriting the test or by rewriting the code. Assuming that a false positive is actually a true positive is often a consequence of fear, uncertainty, or doubt. It has the effect of raising the status of incidental behavior to required behavior. In rewriting a test, programmers either refocus the test on the required behavior (good) or simply hardwire it to the new implementation (not good). Tests need to be sufficiently precise, but they also need to be accurate.

For example, in a three-way comparison, such as Java’s String.compareTo or C’s strcmp, the requirements on the result are that it is negative if the lefthand side is less than the right, positive if ...