Beautiful Testing

Regression and Process Improvement

The stress on a software testing team comes from many directions. If we tried to rerun every test idea every two weeks, the weight would compound. Within a half-dozen iterations the team would be perennially behind. If we tried to automate every test idea, we’d have the same problem—and as I discussed earlier, trying to do automation can lead to blinders about entire categories of defects.

So as odd at it seems, the critical questions in this area are: What do we not test? What tests can we skip? What tests always seem to pass, we expect to always pass, and provide us only confirmatory value?

These are philosophical questions about truth, the classic problem of induction. As the saying goes, if we are trying to determine the truth of the statement “all swans are white,” a million white swans give us less information than a single black one.^[90]

Over the past year, we have changed our tactics several times. It’s impossible to know anything for sure, but if we spent eight hours per iteration testing features that always worked—and involved a well-isolated piece of the code—well, we had plenty of other techniques to use that could deliver results. We added reviews earlier in the process, changed the iteration schedule by a day or two here or there, and changed the format of our team standup meetings.

Perhaps it’s time I told you about our staging tests.

^[90] The best book about testing I’ve read this year is, by far, The Black Swan by ...