Your gut tells you that if you could just implement continuous testing in your software development process, you’d easily reduce risk, decrease costs, and probably even increase revenue. The challenge is, your CFO doesn’t think your gut is a voice of authority. So, how do you convince the higher-ups to give you the budget you need?
You’ve been in the trenches. During your career, you’ve written your share of code. Depending on your management style, you might still write code. Either way, you’ve done your time and know that when coding runs smoothly, it’s a thing of beauty. But, when does it do that? Some days are better than others, but, by and large, development feels like a stop and start process. Getting “into the Flow” never seems to happen because of all the urgent bugs that need to be squashed and the fires that need to be put out.
It’s clear to you that if you could systematize the testing process a bit more and give your attention to the right things at the right time, you and the team would be much more productive and have fewer surprises. And you know that would end in real impact to the bottom line.
But how do you prove it? You know you need to speak in terms of dollars and cents, but how do you convert what your gut’s telling you into hard numbers?
There are several great articles on the internet proposing ways continuous testing can financially benefit an organization. You and I, and anyone else who has been in the thick of it, read those articles and know what they say is true. The problem is, most of the arguments don’t end up with real numbers at the end. Not much more convincing to a CFO than your gut was. That’s why I’ve decided to focus this writeup on just a few strategies that you can literally run math on. Specifically, we’ll look at Catching a Bug Early, Automating Regression Testing, Reducing Churn, and Improving Reputation.
Catching a Bug Early
You know when you’re coding along, making happy progress and then you test your code and, say, the cool math you wrote doesn’t pencil out? No problem, you flip right back to that code, with that function still fresh in your mind, and, maybe even before you actually see the problem, you realize your mistake, tweak the calculation, and you’re done.
Coders are constantly fixing bugs. I’m pretty sure I never wrote a complete function of any real size without fixing a half dozen bugs before even committing it to the code base. It’s just part of the process. The problem is, these bugs aren’t thought of as bugs. They’re usually not tracked or measured or recorded. We don’t think of bugs as bugs until someone other than the original developer finds them. And why do I say this is a problem? Because this kind of bug is the best kind of bug. We want to turn all other bugs into this kind of bug. But until we acknowledge its existence, the other bugs can’t aspire to be like it.
You already know why this is the best kind of bug. You and everyone on your team would rather squash this kind of bug all day long over those ugly bugs. You know the ones. They are the bugs in that code you handed to QA last week. You’re still pretty sure you remember your strategy with it, but it takes you a bit to get back in as you mentally and literally get out of your current job and back into last week. They’re not too hard to fix, though they will take longer than if you’d caught it yourself. Last week.
There are worse bugs though; namely, the ones caught in Beta testing. Now you’ve moved way beyond that code and can’t even remember what you were doing with it. You must reread the functional spec just to understand what the original goal was. Getting your head back into that code takes significantly longer than if the bug had been found earlier in the process.
But even those aren’t the horrifying bugs. The nightmare bugs have been sitting all comfy out in Production for like 10 years and have grown all sorts of appendages. Once someone discovers them, they’re so entangled in the system, removal requires major surgery. Plus, the guy who created them is long gone and no one has looked at that code for ages. As the manager, you get to decide which member of your team you’re going to owe donuts.
Think of all the time you would save if every bug created was caught by the creator before it left her desk? You know there would be major time (and, therefore, money) saved. The good news is, now you can prove it to the person holding the purse strings. And this, thanks to a nice little study done way back in 2002 for the National Institute of Standards and Technology called, “The Economic Impacts of Inadequate Infrastructure for Software Testing” where they got real numbers to attach to those four kinds of bugs we just discussed.
Turns out, your gut was right. According to the study, when a bug is caught right away (during coding or unit testing), it takes on average 3.2 hours to fix. If a bug is caught by QA, it takes almost three times as long, or 9.7 hours. If a bug makes it to Beta testing, it is squashed in 12.2 hours and, if it makes it all the way to Production, it takes 14.8 hours.
Now we have the beginnings of some real math. To complete the formula, all we need to know is two things:
- Exactly how much continuous testing shifts the catching of these bugs to the left along the cycle, and
- How many total defects we’re talking about?
Fortunately for this conversation, we have the first set of numbers and you have the second.
As we have implemented our Continuous Testing Framework with our clients, we have found the following average reduction in bugs found (by percentage):
- First Quarter: 88% (of the bug count before implementation)
- Second Quarter: 53%
- Third Quarter: 27%
- Fourth Quarter: 19%
- Fifth Quarter: 9%
- Sixth Quarter: 2%
- Seventh Quarter: 1%
- Eighth Quarter: 0%
As time continuous on, bugs do slip by, but, in our experience, after a solid continuous testing strategy is in place, you really are bouncing around 0 and 1 percent of your pre-continuous testing defect count.
The final set of numbers comes from your own environment. How many defects do you catch over the period you’re tracking? And, when are they caught in each of the 4 stages? Unless your tracking is slicker than most, you likely don’t have those last numbers, but you can estimate pretty easily. We estimate that 20% are caught by QA, 60% during Beta testing, and 20% are caught in Production, but you can play with those numbers however you like. The trickiest number is the count of bugs caught during coding. We suggest you ask your developers what they think. When we have made similar surveys, we’ve found that a conservative estimate is about 200%. In other words, for every bug that slips by them, they caught 2 others.
Once you have all the pieces, you just need to plug it into a spreadsheet. Create your own, or, you’re welcome to use ours (ask us for it, using the link below).
Spoiler alert: the numbers add up fast. For a team of only 10 developers, in just the first year, you can save 1,200 to 5,000 hours or up to 25% of total available dev time. Over 3 years, those numbers can climb as high as 22,000 and 35%. If those 10 developers are being paid on average $75,000, you save more than $1,000,000 in those 3 years. That’s a number your CFO can appreciate.
Automating Regression Testing
It makes intuitive sense that the sooner you catch a bug, the less it will cost to fix. And now you have numbers to back that up. Similarly, it makes intuitive sense that automating testing generally should save you money. The challenge is that there are a lot of variables that go into test automation and it’s not always straight forward which ones are having what effect. Possibly the biggest unknown is whether the human test is more, or less effective than an automated one. Let’s face it, a human can test in ways we just haven’t (yet) figured out how to get a computer to do. And we’re all pretty sure those tests are worth what we pay for them.
There is, however, one area of automated testing that is easy to quantify: Regression Testing. With regression testing, you are going to run the same set of tests every single time. No “thinking outside the box” necessary. In fact, if you are currently doing manual regression testing, you are probably following a defined set of instructions. And, it turns out, computers are exceptional at following defined sets of instructions.
Therefore, we recommend when looking for ROI from automated testing, you focus on just your regression testing. We’re not saying automating all the other tests won’t save you money; we’re just saying the financial story is much clearer with regression testing. If a computer can run the same scripted test that is currently being run by a human and get the same outcome, the cost of that human is the savings you’ll find.
The math is simple. First, figure out how much of all your testing is spent on regression testing. If you’re not doing any automated regression testing yet, and if your code base is larger than a peanut, we estimate that you are spending between 50% and 90% of your time doing regression tests. Next, decide how much of that regression testing you think you can get automated by the end of years one, two, and three. Finally, plug the salaries of your full-time testers into a spreadsheet, multiply by the first number and the second set (once for each year), and you’ll have your cost savings.
As a simple example, if you’re spending $10 per year right now on testing and 50% of your testing time is spent on regression tests and you think you can automate half of your tests by the end of the first year, you’ll save an incredible 2 dollars and 50 cents by the end of the first year! (Note: If you want to be totally honest with your CFO, you can add some complicated math to factor in when the tests are ported throughout the year. But really, it’s easier to think of the percent of tests you’ll automate by the end of the year as an average of all tests throughout the year. So, 50% by the end of the year probably really means that you’ll have 20% done very quickly, 50% by the middle, and 80% by the end)
As with the calculations for ROI from Catching a Bug Early, you can create your own spreadsheet to figure this regression testing ROI out. Or, you can ask us for ours.
But I already automated my regression testing.
That’s great! Are you still doing it? What processes do you have in place to ensure these tests remain meaningful or to ensure you’re expanding that regression test suite to encompass the ever-growing list of new features? To keep the value that you gain by automating regression testing you must maintain it.
Yes, I have automated regression testing and I’m maintaining it.
Excellent, then this part should be more of a review for you. You’ve already experienced the value of automating your regression suite. You must be here to learn more about how the rest of this can help you. Let’s continue then…
Reducing Churn
Now let’s look at how we can show continuous testing can not only save money but can produce money. This is the holy grail of every development shop manager. For most of our careers, we’ve been treated as a cost center, meaning we cost the company money. It’s the sales guys who are the revenue center. They’re the ones that make the company money. Yeah. Whatever. They wouldn’t have anything to sell if we didn’t make it for them! The problem is, we’re generally not very good at attaching our work directly to revenue. We’re too far removed from the sale itself. This section and the next are attempts to change that a bit for you.
First, a definition just in case you haven’t run into this term: Churn. Churn is how many users (or dollars) are lost during a period. It’s a little deceptive because you could be increasing total user count and still have churn. For example, you might have 5% user growth over a quarter, but you in fact, added 8% more users and lost 3%. In other words, if you had 100 users last month and you now have 105 users, you can’t just assume you added 5 users. In our example, you added 8 users, but lost 3 others. Those 3 users are your churn.
Adding users (and therefore, revenue) is something you and I know comes down to our work as software developers, but that is very difficult to prove. What we can prove is that our work is responsible for a lot of the churn in a software product. If a product is buggy, people will stop using it. Sure, bad features, design, and usability are also all reasons for churn, but bugs are a huge deal.
Now, this sounds like a bad part of our financial story, but it turns out that it is the opposite. Stick with me here. Churn happens. Yes, you can think of it as our fault, but you can also think of it as our opportunity. If we can produce less buggy software, then we can reduce churn. Unlike finding the customer or landing the sale, the quality of the software is something we can directly influence. And we influence it positively by improving our testing.
So, this is very simple math. First find out how many users are lost each year based on low product quality. This is going to be a guess, unless you have very good customer exit surveys, but I’m betting your financial people are willing to put a percentage on it for you. Now, multiply that number by the revenue an average user gives the company. That is the total revenue lost due to bugs. Finally, throw that against the calculations you made from the Catching a Bug Early section, and you will see how much revenue you can create for the company! Notice, based on real world experience our clients have seen, within only a few cycles, you could be down to zero or one percent of current bugs. That means between zero and one percent churn. That, all by itself, is a good ROI.
Improving Reputation
The final bit of ROI we think you can rationally use as an argument for implementing continuous testing is based on a number you probably don’t have access to. But your marketing or product people probably do. This is the idea that some products in your space enjoy higher revenue based on their reputation as a Quality Product Leader. To find this number, just start asking people this question: “How many additional new users could we acquire annually if we developed a reputation in [our space] as the Quality Product Leader?”
Once you have a number people are at least willing to use for argument’s sake, simply apply it in the same way we did in the previous section. Look at where on your product horizon you think you will be down to basically no bugs, add a couple cycles for people to talk about it (with the help of your marketing department), and boom: new users.
Again, simple math. And, again, based on the very quantifiable calculations of fixing bugs earlier in the cycle.
Putting it all Together
Although it might take you some time to put together a spreadsheet to pull this all together, the good news is that this really is all just math. Dollars and cents. The language of the folks holding your budget. The value of implementing continuous testing is a no-brainer. You know it and I know it. And now, by speaking their language, you can help your finance people know it.
To save you the headache of putting this all together (and because we have at least one spreadsheet nerd on the team), we have done the work for you. We’d love to walk you through the process and answer any questions specific to your situation. Just ask by clicking the link below.