We’re Writing Too Many Tests
Creating and maintaining a test automation suite is a time-consuming endeavor, yet the value of those efforts is often questionable.
--
Creating and maintaining a test automation suite is a time-consuming endeavor, often rivaling the time spent developing the code changes they are there to support. Yet the value of those efforts is often questionable. It’s hard to judge the value of test automation, and it’s surprisingly easy to invest a lot of time for no, or even negative, return.
Now, I don’t want to suggest that we shouldn’t write tests. Done carefully, they are a vital part of the development process; they increase confidence, help us catch bugs earlier in the development process, and make future improvements easier. Instead, I propose that we be more careful about how we write tests and what tests we choose to write. We need to ensure the effort in creating tests is balanced against the value we get in return.
Last year, I deleted our team’s end-to-end test suite. This suite had all the same problems as the last two e2e test suites I deleted. It wasn’t finding any valuable regressions. It was burning time investigating and fixing failures that all turned out to be flakes and had become difficult to add new tests to. A few months later, we started transitioning the app to a continuous delivery model as part of an extensive modernization project. But now we had a problem — did we have sufficient test automation to give us confidence in our releases? While the e2e tests had provided little confidence, it was hard to gauge whether we could rely on our unit tests either. Sure, we had a lot of tests and a high code coverage number, but many of the tests didn’t appear to actually be testing anything. Without being able to trust the existing tests, we retrofitted a suite of highly reliable integration tests that covered the scenarios most important to the business and customer. Even better, we were able to write these tests in a tiny amount of time.
Three anti-patterns plague the test suites of almost every application I have worked on: tests that aren’t actually testing anything, tests that repeat the same coverage across locations and layers of tests, and tests that ignore the context of the code they’re testing…
