Sun Jun 20 2021

Don't be afraid to change your tests

Tests. You write them. You forget about them. You keep writing code. You write more tests. And more tests. Now they take three minutes to run. You've stopped running them as often, only on CI pipelines, crossing your fingers and holding your breath, hoping for that little green checkmark. You make a change and a dozen tests break. Where do you start looking to make the fix? How did you get here? There has to be a better way!

It might sound like a contrived infomercial scenario, but I've seen people shudder at the memory of tests that made the whole development experience painful.

infomercial fail
I've seen it happen

As I've said before, you should change your code often. Tests are code too. They should change less often and for different reasons, but if they're causing you pain then do something about it.

If they're not useful anymore, delete them. If they're getting in the way, figure out better tests that will help you work quickly and make updates safely. Use coverage metrics to make sure you're not leaving any gaps when you make changes.

Most importantly, pay attention to them. Tests (and your reaction to them) are valuable feedback. Don't just write a bunch of them, assume they're good forever and forget about them. If they're causing you friction, figure out why. Make changes. Tests should reduce friction, so if yours aren't then figure out why not. Good tests will not slow you down.

So... how should you change them? Well, consider the tradeoffs.

For example, if you have dozens of tests that break often or run slowly (anything more than a couple hundred milliseconds per test is slow), can you reduce the number of tests by putting more assertions in one test? Then do it! Gasp! I know! That's why it's a tradeoff: you're trading granularity for ergonomics.

Ergonomics: When you see a single test fail instead of twenty, it's easier to know where to start. When tests only take a few seconds (or less) to run, you'll run them much more often. These things make testing a better experience.

Granularity: When you have multiple assertions in one test, you'll only see one failure at a time. You might fix something only to have the test fail on the next line. But modern testing frameworks like Jest give us so much information when a test fails that we can often afford to cover more ground in a single test. We live in exciting times.

Maybe this example doesn't fit your case, or maybe you're skeptical. You're more comfortable keeping them separate. I get it. Is there something else causing a bottleneck? Other changes you could make? Test friction can creep up slowly. An extra 100ms sneaks in here and there, not a big deal when you have 10 tests but now you have 1000 and before you know it you've boiled the proverbial frog with 1000 proverbial cuts.

I'll say it again: pay attention to your tests. Pay attention to how you feel about your tests. Update them when they get in the way, or remove them when they become redundant. Use a code coverage tool to do it confidently. If you've written them well they should stay out of the way unless something actually breaks, but sometimes the headache sneaks up on you. Don't ignore the headache. Change your tests.