One thing that I constantly encounter in big software companies are well-intentioned attempts to ensure bug fixes are properly ported into releases, that, in the end, simply fail to deliver.

It’s no wonder. The problem is much bigger than one might think. It costs hundreds of thousands of dollars to buy static analysis tools to find unknown bugs in large codebases. It costs even more to fix the bugs found in QA, development or production. But, in spite of all that effort, companies consistently fail to eliminate those bugs from releases.

Why? Our research shows about 10%!! of changesets in development branches end up missing in some of the releases. Given the devastating consequences of failing to eliminate security or other important types of bugs (recalls, hot fixes, etc.), we thought a list of the culprits we consistently see might be helpful. Here they are:

The failure of SCM Systems

The code always lives in an SCM system – Subversion, Git, Perforce, CVS, ClearCase etc. It’s natural that developers turn to these tools to ensure when a bug is fixed, it’s properly propagated to all the target branches. Modern SCM systems like Perforce and Subversion have built-in mechanisms to integrate one changeset from one branch to another. If an integration action is taken, it leaves an entry in the SCM log. Based on the history in the SCM log, companies build in-house tools to track the target branches where a bug fix has been applied. Sounds good, right?

Well, not really. SCM systems rely on manual input. It always comes down to developers taking the “right” action during merge or keying in the “right” information. But developers don’t always use the integration command to port bug fixes. Instead, the bugs are manually fixed in other branches. And that leaves no trace in the SCM log.

Second, SCM systems don’t understand code and treat it as text. For example, SCM systems cannot track duplicated content. And if a file is removed and added back in, SCM systems cannot track the changes between these two files. Also, if a change has occurred to the one branch, even if the change is superficial (e.g. a variable rename), the SCM will fail to do the integration and the developer will have to manually decide what to do.

Third, legacy SCM systems cannot even do basic changeset tracking. Regularly used in development shops (you’ve probably seen them too – CVS, Borland StarTeam, and Rational ClearCase Base). These don’t support the notion of a changeset (or changelist). They were designed to do one thing, track revisions in files. Naturally, they fail at tracking changesets. How do you even know what the complete change is that needs to be applied – especially if it spans multiple segments of code in multiple files? And this makes tracking a bug fix nearly impossible. Welcome to more manual workaround rodeos.

Grep to the rescue? Not really…

Another method I see companies attempt is keyword search or regular expression based tools. Sadly, these are really lacking as well. Here’s why:

Have you ever tried writing a regular expression to capture a bug across multiple segments of code? It’s difficult to impossible to determine the right query to get the results you want. A bug tends to involve many snippets in different files. A simple keyword search or regex-based search cannot capture this complexity.

Second, the results are usually very noisy, since the search lacks context. Search results contain extraordinary volumes of irrelevant results. The overhead of processing them and narrowing down the candidate code snippets is prohibitive.

Finally, it is impossible to be certain that the results are complete. For instance, cases including examples where buggy code has been copy-pasted into another area, and then modified, are nearly always missed.

The solution?

Any robust solution to the problem must:

• Parse the code: Unless a tool can parse and understand the code it’s impossible to identify copy-pasted segments or files, tolerate superficial changes and make structural and semantic matches
• Not rely on manual input: If a human is involved and the process is not fully automated, it’s impossible to avoid operator mistakes. This is the reason we are looking for tools to ensure that bug fixes are in the release in the first place.
• Track replication: Based on our studies of open source and commercial code, 20-40% of the code within a single code base is replicated (for the Linux kernel for example the percentage is in the low 20s). Furthermore, about 2/3 of these replicated code segments are modified. If a tool cannot account for 20-40% of a codebase it’s useless.
• Be automated: The only way for a solution to scale is to be fully automatable. I fixed a bug, now give me a report, or, take all the bugs from a development branch and tell me if they are in the release. Otherwise, there is no way that there is going to be wide adoption.
• Identify complete bugs (not just single snippets of code): The vast majority of bugs in mature codebases involve multiple snippets of code and usually multiple files. If a solution does not take the complete set of changes into consideration or fails to consider the context it will overwhelm users with results that are mostly noise

At Pattern Insight, we spent the last few years thinking about this problem. We think Code Assurance is the solution to it, so do some of companies with the biggest codebases of our industry, like Cisco, Motorola, Qualcomm, Wind River, etc.

I will be happy to provide a trial version to anyone who thinks that his SCM system has him covered. Just shoot me an email (spiros@patterninsight.com).

--Spiros