Managing Technical Debt

Among the services we offer is an assessment that can span everything from mobile and cloud strategy, to enterprise architecture, process and methodology, design, testing strategy, software tools, and even an in-depth analysis of existing codebases. After conducting a number of these assessments, we noticed a trend. Many of the IT directors who had until recently been hailed as heroes for delivering low-cost mobile apps seem increasingly alarmed about their future. Why? Their projects incurred significant technical debt as a consequence of cost-cutting measures. (See also: wiki article on technical debt by Martin Fowler)

In short, technical debt is the accrued balance of the shortcuts taken and compromises made to get a project out the door on time and on (or under) budget.

While initially, an app whose development team cut one too many corners may seem to work just fine, as the app evolves the code base can quickly degrade into an impenetrable mass of spaghetti. Unfortunately, development teams rarely consider (much less track) the amount of technical debt added from one revision to the next.

Signs and Symptoms

One of the symptoms we typically see when a project is over-leveraged (i.e., has accrued too much debt), is a growing rift between Product (the business) and IT. The mobile development team may be perceived as losing velocity over time, struggling to meet deadlines, delivering increasingly buggy code, and being seemingly incapable of implementing a growing subset of new feature types.

The latter issue tends to be particularly frustrating to Product. It would naturally be hard to reconcile development’s claims that certain features are ‘impossible’ to implement with the presence of the selfsame bright, shiny, organic, grass-fed, free-range features in the latest versions of competitors’s apps.

When developers claim that new features are impossible to implement as specified, and plead for requirements changes and simplifications, often what they’re really saying is, “Uh, we can’t get there from here — the current state of our application’s design and architecture (or lack thereof) and codebase simply won’t support it.”

Fingers get pointed. Things get said. Chairs get thrown.

The Young and the Reckless

While it can be argued that there’s always some degree of technical debt in a given codebase, a clear distinction can be drawn between prudent technical debt and reckless technical debt.

For example, reckless technical debt might be incurred by a junior developer writing poor code simply through lack of experience. (This could also be characterized as unintentional technical debt.) Prudent technical debt, on the other hand, would be incurred if an experienced developer decided to go with a temporary, quick-and-dirty implementation to meet time-to-market constraints for a new feature, while planning to incorporate a better, more permanent solution later on.

In general, inexperienced developers tend to introduce a great deal more technical debt — particularly of the reckless variety — than do their more seasoned colleagues. And sadly, they’re rarely aware of any technical debt they may have introduced.

Ultimately, you get what you pay for. The initial cost saving achieved through cheaper developer rates will often be offset to a great degree by greater technical debt. Unfortunately, the degree can be very large, and is hard — if not impossible — to measure. Veteran developers, on the other hand, have bloodied their foreheads enough to know what a reasonable level of debt is, which items to address in the short term, and which to defer to future releases.

The Real Culprits

We often see codebases with unhealthy levels of reckless technical debt. Mostly, we see this in code that was outsourced to low-cost providers. When new features need to be added, they’re simply bolted on — often with the coding equivalent of duct tape — and pushed out the door. As they rotate inexperienced, generalist developers through projects, any consistency in the codebase quickly evaporates, code and component-level reuse goes from negligible to non-existent, and application architecture and design don’t evolve with the codebase.

Surprisingly though, we sometimes also see similar levels of reckless technical debt in codebases developed by in-house teams. The sources are many, including flawed architecture decisions; overly simplistic or sloppy design; failure to leverage available platform resources, patterns, and best practices; failure to keep the codebase current — but I’ll leave the details for another post.

Paying the Piper

Then comes the inevitable shakeup, either thru a new hire, a merger, or a re-organization and new management is now drawing conclusions about all this disfunction in the organization. Hopefully you, the savvy IT director, were already promoted and left this mess to someone else! But often, you’re the one who ends up in the cross-hairs. What you failed to manage effectively is the inverse relationship between cost and risk. As a result the project accrued enough technical debt to serve as a significant drag on performance.

At a certain point, the ‘interest cost’ (the extra effort required to fix bugs and implement new features in a pathological codebase) became crippling, while simultaneously the debt grew too large to repay. (In the worst case scenario, paying the debt might entail throwing away essentially the entire codebase and starting over from scratch.) We know its not all your fault — you were in the trenches making these decisions in real time, and doing your best given scope, time, and money (It’s a tough racket!). But the new management will tend to focus on the risk first (as they were not around to reap any of the benefits of your low-cost strategy).

1. Negotiate with the product team on a regular basis to ensure that scope is manageable in the first place so that your team doesn’t feel pressured to cut corners again and again.
2. Make sure project risks are fully captured in writing and communicated to stakeholders on a regular (ideally weekly) basis.
3. A small team of experts will run rings around a larger but more junior team. Ensure that your budget supports at least seeding — if not fully populating — your team with experts. The work should get done much faster, and as a result, the net cost may actually be less, in spite of the higher hourly rates. But more importantly, the project won’t incur hidden costs in the form of technical debt that can quickly spiral out of control.