How is static analysis a productivity tool for engineering teams?
The earliest accessible verification activity is static analysis. Learn why Coverity static analysis is the top productivity tool for engineering teams.
“I lost my keys. How long will it take to find them?”
This is a laughable question, but it’s analogous to “How long will it take to debug this?” Developers scoff at this question as if it were an unreasonable demand, just as inexperienced project managers are shocked that a simple answer isn’t forthcoming. But this interaction says more about the problem we face than about the people involved. Occasionally, an extended debugging session can be fun, and we enjoy telling those “war stories.” But in reality, debugging is one of the most frustrating and inefficient things that we do. By some estimates, we spend about half of our time on it.
Debugging code
Although we never know how long it will take to debug a particular problem, we do have some idea of the relative difficulty of resolving various issues. If there is a repeatable null pointer dereference, for example, we can often knock it out in a few minutes. The debugging portion is rarely hard. The code fix isn’t anything to write home about either. In this case, the hard part is deciding what the correct behavior for a particular input should be.
On the other hand, a race condition that occurs infrequently may be nearly impossible to find. Memory corruptions aren’t much prettier, although we have at least some basic tools to help. I once visited a medical device manufacturer who knew that they had a race condition and had spent months debugging, only to reach an impasse. They were trying to take a new piece of equipment to market and needed to have some sort of time estimate. So as part of their solution, they actually planned to rewrite entire subsystems until the problem was eliminated.
If debugging code that we just wrote is hard, revisiting something that was done months (or even years ago) is even harder. Imagine if the question were “I lost my keys a month ago. I don’t know where. How long will it take to find them?” That’s why we like solutions that give us fast feedback. Type checking by the compiler is a great example. Some of those warnings that get spit out are also useful. But we probably get the most value from the static analyzer.
Early SDLC verification
In the software development life cycle (SDLC), the earliest accessible verification activity is static analysis. It’s possible to run a static analyzer as soon as there is one parsable function, even if no code can be executed yet. Modern static analyzers are incredibly fast. They can produce results in less time than it takes to get a cup of coffee or answer your boss’s email (although I still rarely find time for the latter when I’m heads-down on a project).
We all know that finding defects early in the SDLC makes us more efficient. (Ideally this would happen in the construction phase if you’re using a waterfall methodology, or during the same sprint when the defect was introduced if you are using more agile techniques.) An entire cottage industry has sprung up around measuring the efficiency of early defect detection, starting with some data that NIST published over a decade ago. The consensus is that the overall savings of finding a defect early versus late is around 8 hours of work.
Enabling engineering velocity
Engineering teams are usually considered the long pole in the tent. Saving 8 hours of work overall is great, and we like to be team players, but we can’t take on additional work to save other groups time. We have to look at how much of the savings accrues to us and whether our decisions accelerate or slow down the development pace.
When looking to decide how much time can be saved through better software development processes, everybody looks at the same NIST study from 2002. As a reasonable approximation, we can say that the savings from finding a defect during development (as opposed to during a testing phase) accrue to the engineering team. The incremental cost of defects escaping tests will fall on the rest of the organization (support, testing, sales, etc.).
In the NIST study, the average fix time for a defect found during coding / unit testing was 3.4 hours versus 7.0 hours for integration / system testing. In other words, if the static analyzer finds one defect per week that would have otherwise escaped, we become 9% more efficient if working 40-hour weeks (4.5% more efficient if we’re already working 80-hour weeks). This number is pretty consistent with the numbers reported by Synopsys customers. The static analyzer typically finds about three new defects per developer per week, but only one of them represents a defect that would have otherwise escaped (as opposed to being caught during automated testing such as unit testing).
Reaping the benefits
This brings us to the final question of how quickly the benefits will accrue. Investing 8 hours today to save 20 hours next month makes sense on the surface, but not if my deadline is tomorrow. Good static analyzers such as Coverity static analysis can track defects over time and distinguish new defects from legacy / technical debt issues. If code has been in production for years and hasn’t caused a problem, whatever “defects” are revealed are unlikely to be worth fixing.
On the other hand, if I just wrote some new code today and didn’t realize that an API I called sometimes returns existing memory and at other times returns a new object that needs freed (or closed or disposed) memory, resolving that issue before checking in the code is likely to pay dividends within the existing development sprint.
So how long will it take to find your keys? It won’t take any time at all, if you stop losing them. Don’t be afraid to introduce static analysis into your SDLC. The boost in productivity will come quickly enough that it makes sense to start sooner rather than later.
