I’ve been talking to a few people about Test-Driven Development, which has me thinking about what important factors for success might be overlooked. Here’s a big one: you have to use the unit tests. Otherwise, they’ll rot on the vine and the whole TDD effort will be in jeopardy.
Here’s what I look for – you want to have a CI pipeline that automatically runs all of the unit tests and fails the build if any test case fails. You shouldn’t be able to release the software (without an emergency override from senior management, perhaps) if the unit tests aren’t all passing. If you don’t have a CI pipeline yet, you can still put this safeguard into your build process, and ideally into your version control system so you can’t make a code change if the tests aren’t all passing.
If you have failing unit tests already, the first step is to get them to pass (you might say “the tests are red”). Maybe this means fixing bugs in your production code, or more likely, fixing broken tests. But in a lot of cases, the most practical thing to do is to disable or delete the troublesome tests. You can put tasks in your backlog to fix them if you want to, but if they’re failing now, they’re not helping you, and you should get your tests back to “green” quickly. Only then is it feasible to introduce a mandate that they have to stay green.
Do you see a failed unit test run and instinctively hit the button to re-run them because they’re likely to pass after a few tries? Does your CI already do this automatically? Flakes in unit tests are less likely than with higher order tests, but it can happen. It’s also fairly common that tests that are called unit tests are really higher order tests (a topic we can explore later). This is a tough problem to deal with. I recommend that you take measures to identify which tests are flaky, and also keep metrics on which builds encountered one or more flaky tests. Though you might choose to remove flaky tests, when you do, you’re also removing the possibility that they might start failing consistently and thus demand the investigation of a bug in production code. It might make sense to take a long-term look at improving the design of your troublesome test suites, because flaky tests have a cost that you have to bear over time.
Do you have no unit tests at all? It’s time to get started! You only need one in order to update your build process and get people used to having green tests. Really, just one.
Why is all of this important? As soon as failing tests become normalized, you lose track of how much bit rot has happened in the tests. The technical debt quickly builds to the point that the most dedicated TDD adherents end up spending a large fraction of their time fixing problems introduced by other developers so that they can get the tests back to green and get back to their work. The pressure to ship the next code change will soon overwhelm everyone, and then your developers aren’t doing TDD at all.
If you’re not going to use the tests, don’t add them to your code base.
Here’s what I look for – you want to have a CI pipeline that automatically runs all of the unit tests and fails the build if any test case fails. You shouldn’t be able to release the software (without an emergency override from senior management, perhaps) if the unit tests aren’t all passing. If you don’t have a CI pipeline yet, you can still put this safeguard into your build process, and ideally into your version control system so you can’t make a code change if the tests aren’t all passing.
If you have failing unit tests already, the first step is to get them to pass (you might say “the tests are red”). Maybe this means fixing bugs in your production code, or more likely, fixing broken tests. But in a lot of cases, the most practical thing to do is to disable or delete the troublesome tests. You can put tasks in your backlog to fix them if you want to, but if they’re failing now, they’re not helping you, and you should get your tests back to “green” quickly. Only then is it feasible to introduce a mandate that they have to stay green.
Do you see a failed unit test run and instinctively hit the button to re-run them because they’re likely to pass after a few tries? Does your CI already do this automatically? Flakes in unit tests are less likely than with higher order tests, but it can happen. It’s also fairly common that tests that are called unit tests are really higher order tests (a topic we can explore later). This is a tough problem to deal with. I recommend that you take measures to identify which tests are flaky, and also keep metrics on which builds encountered one or more flaky tests. Though you might choose to remove flaky tests, when you do, you’re also removing the possibility that they might start failing consistently and thus demand the investigation of a bug in production code. It might make sense to take a long-term look at improving the design of your troublesome test suites, because flaky tests have a cost that you have to bear over time.
Do you have no unit tests at all? It’s time to get started! You only need one in order to update your build process and get people used to having green tests. Really, just one.
Why is all of this important? As soon as failing tests become normalized, you lose track of how much bit rot has happened in the tests. The technical debt quickly builds to the point that the most dedicated TDD adherents end up spending a large fraction of their time fixing problems introduced by other developers so that they can get the tests back to green and get back to their work. The pressure to ship the next code change will soon overwhelm everyone, and then your developers aren’t doing TDD at all.
If you’re not going to use the tests, don’t add them to your code base.
swalchemist 