A lot of really smart people in the software business – including people with decades of experience – ask us the same question: How do these bugs we’ve already fixed get back into our releases?

The simple answer:  We in the software industry still rely on manual processes to assure code quality. More specifically though, the underlying cause for previously fixed bugs being released to customers can be chalked up to the proliferation of components, software releases, and product variants in modern development methods.

We can thank two innovations for this:

Component and software reuse is a fine practice, and it can lead to lower development costs. BUT, it creates a huge number of variants. We see mind-numbing product variants all sharing software components in the software used by telecom, mobile, aerospace, automotive, medical device businesses.

Parallel branching sure can help streamline software development. It is common for software companies to maintain many old releases in the field even while working on a new release. And some organizations maintain multiple customized versions, each specialized for specific customers. But all that comes with a HIGH price tag.  Developers make constant code changes – including bug fixes – and keeping these versions in sync is a real nightmare.

So, what does this have to do with those magical bugs insinuating themselves back into releases?

Companies usually impose a policy that if a bug is fixed in BRANCH A, the fix needs to be ported to BRANCH X, Y, and Z. But, the ever-increasing number of branches and bug fixes brings manual processes to a grinding halt.  Add in the constant pressure to deliver releases on time… and patching quickly become a secondary priority.

Often it’s tough to simply determine which branches a bug fix should be ported to because developers often don’t know when a bug was originally introduced.

When bug fixes are integrated into another branch, it’s common for conflicts to arise with other changes made to the same snippets. At that moment, the developer only sees the conflict at the snippet level. And because the purpose of conflict resolution is to make the build clean, the developer is likely to unintentionally break another developers’ bug fix.

In other cases, a bug fix consists of several change sets in the SCM system. We see cases where another developer comes into that situation and mistakenly reverts one of the numerous change sets, breaking the fix.

Copy-paste is also common – we’ve seen estimates that 15-50% of all software is duplicated code.  Entire components are commonly reused and modified.  But when a component is reused, buggy code in it goes right into the release.

These are all completely understandable situations – no developer we’ve met intends to cause bugs. But it happens all the time. Why? Because managers aren’t always aware of the new work being done to automate tracking these duplicated bugs.

-- Spiros