Key Takeaways
As software systems evolve, they tend to become less flexible and more difficult to work with. We often attribute this to rampant "technical debt", but fail to talk about what causes it.
Technical debt is not primarily caused by clumsy programming, and hence we cannot hope to fix it by more skilled programming alone. Rather, technical debt is a third-order effect of poor communication. It is a symptom of an underlying lack of appropriate abstractions, which in turn stems from insufficient modelling of the problem domain. This means that adequate communication has not taken place; discussions and decisions to resolve ambiguity and make informed trade-offs have been swept under the rug.
What we observe and label "technical debt" is the by-product of this dysfunctional process: the reification of this lack of resolution in code. To fix the problem of accumulating technical debt, we need to fix this broken process.
The technical debt metaphor was introduced by Ward Cunningham to describe a process where developers make a conscious decision to ship code with known limitations into production. The purpose of shipping early is two-fold: to get quickly to market and to enable a feedback loop from production back to further development and refinement. It quickly caught on, since it allowed developers to communicate "invisible" problems with the technical solutions to management and other stakeholders.
In the process of becoming widespread and popular, however, the meaning of the technical debt metaphor has also become diluted. Any code running in production that has limitations or quality problems may find itself labelled as technical debt. This is unfortunate, since it undermines the usefulness and the richness of the metaphor. Much of what is considered technical debt is incurred inadvertently over time, and with no clear strategy for paying it off.
It is regrettable that the meaning of the technical debt metaphor has been diluted in this way, but in language as in life in general, pragmatics trump intentions. This is where we are: what counts as "technical debt" is largely just the by-product of normal software development. Of course, no-one wants code problems to accumulate in this way, so the question becomes: why do we seem to incur so much inadvertent technical debt? What is it about the way we do software development that leads to this unwanted result?
These questions are important, since if we can go into technical debt, then it follows that we can become technically insolvent and go technically bankrupt. In fact, this is exactly what seems to be happening to many software development efforts. Ward Cunningham notes that "entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation". That stand-still is technical bankruptcy. It means that your organization can no longer move forward.
Reasonable changes to the software take an unreasonable amount of time to implement. Quality problems become permanent; you can't fix bugs without a very high chance of introducing new ones, leading to a sort of oscillation between problems.
If we are to understand the forces that drive the accumulation of inadvertent technical debt, we must look at the code and see how "technical debt" manifests itself.
My observation is that there tends to be a lot of ifs and buts, and little to communicate intent and aid understanding. By that I mean literally that there are many if and else branches in the code, and a great number of boolean flags to control execution flow between those branches. Whats missing are useful abstractions and boundaries to make sense of it all. This makes it difficult to isolate the code related to a single feature, since the code for that feature isnt isolated in any meaningful or obvious sense. It is difficult to predict the effects of changes, since changing a single boolean flag can have rippling effects throughout the codebase.
Code ends up like this when we have an underdeveloped, inadequate mental model of the problem were trying to solve with software. Softwares dirty secret is that we can implement solutions to problems we cant articulate clearly. If our software is "wrong", the correct behavior is always only an if-branch away. We just need a way to inject the right flag so that we can take the correct turn in the flow of execution instead of the wrong one. We can and do compensate for our poor domain models by using if-branches as epicycles. But this is exactly the cause of our problem with inadvertent technical debt: over time, we paint ourselves into a corner. It leads to incomprehensible software - technical bankruptcy.
We can no longer add functionality this way without breaking existing functionality.
It's very hard to write simple and precise code if we don't have the proper concepts in our heads. We need those concepts not only to structure our solution, but to think clearly about the problem in the first place. As long as we lack the proper concepts, both our thinking and our communication with others will be clumsy and roundabout. Imagine trying to tell someone a story about a dog without knowing the word dog or even the word animal. "It's one of those eager, four-legged living things that wag their tails". It sounds silly, but I've been in that situation many times on projects.
On one project I was on, we struggled with our credit card module. The code was complex and difficult to understand, and we had very inefficient and frustrating discussions whenever we talked about that module, but we couldn't really figure out why. It wasn't until we realized that we lacked a concept to describe how credit cards were associated with credit card deals (an "association mechanism"), that everything fell into place. Suddenly our heads cleared up, our discussions cleared up, and it was possible to implement very straightforwardly. We deleted all the clumsy code we had written up to that point, and replaced it with something that was trivial to understand.
This experience points toward a heuristic for tackling complex code - code that tends to get labelled "technical debt" after a while: look for missing or awkward concepts. Look for patterns of frustration in the design discussions in the team. Its probably the domain trying to tell you something. Trying to "fix" the code without the right concepts is likely to fail, since there is no elegant or clean organization of the wrong concepts.
I want to argue that our problems stem from an underdeveloped, inadequate mental model of the problem we're trying to solve with software. For software that is developed as group efforts, which is most software, that mental model needs to be shared among the people working on the software. Otherwise, it's no wonder that inconsistencies and corner cases come to bite us. If were not aligned on what the problem and the proposed solution is, then we should expect to see the consequences of those failures of alignment manifest themselves in the code. And we do.
The key to developing a sufficiently rich and flexible shared mental model is communication and collaboration. When software is weighed down by technical debt, that's a symptom that the organization developing the software probably needs to look at its communication and collaboration patterns.
Ward Cunningham invented the technical debt metaphor to enable developers to communicate to business people something that is visible to the former, but hidden from the latter; that while we shipped code now that meets the business requirement, we overstretched in doing so. Having done so has left us off-balance, and we need to spend some time regaining our balance. Otherwise we will eventually fall down and it's going to be hard to get up. But in a sense, that's an easy problem to fix: generously give the developers a bit of time every now and then to clean up in their own house, so to speak. All that's required from the business people is a little patience while the developers catch up.
Unfortunately, I dont think its going to work. If I'm right that much of what we label technical debt really stems from inadequate modelling of the business domain and ultimately is caused by communication and collaboration issues, it's not a problem that developers can solve on their own. In fact, thinking that the developers can and should handle technical debt alone is a symptom of the kind of thinking that leads to technical debt. Thats an uncomfortable insight for both developers and business people. It is convenient for business people to think of technical debt as something for IT to handle, and it is more comfortable for developers to think that all they need is a little time to get things right. But its a convenience and a comfort that we cannot afford if were going to address the root causes of technical debt.
The main problem for a software organization that finds itself close to technical bankruptcy is not the amount of debt itself, but rather that the organization in its current state produces unmanageable amounts of complex code inadvertently. There is little to be gained by chipping away at the incurred debt if we continue to produce new debt at the same rate as before. It can be very costly, time-consuming and risky to try to untangle code that is near bankruptcy. It is often better to find some way of replacing debt-heavy code with other code that has been produced in a healthier way. The best advice I can give is to try to incur less debt than we currently do, that is, reduce the amount of technical debt we have to reduce in the first place.
Over time, the best approach to reducing what we label "technical debt" is by addressing the root cause, which is how we work together. It can be difficult to change the culture of a software organization. Top-down initiatives will often struggle, since they fail to address the problems seen on the ground. Perhaps the best top-down initiative is to give leeway and autonomy to those on the bottom, since I believe it is possible to bring about positive change bottom-up.
My experience has been that doing software development as a group (i.e. ensemble programming) not only produces better designed solutions to problems quicker, but also creates a cultural shift towards more open, empathic and candid communication. This in turn means that teams doing ensemble programming are less likely to find themselves bogged-down in technical debt as time moves on. Moreover, having experienced improved communication within the team, ensembles are less likely to settle for poor communication across team boundaries as well, or between people with different roles in the organization. If this is true, then working in ensembles can have a positive rippling effect on the communication patterns of the organization.
The uncontrolled accumulation of inadvertently incurred technical debt is endemic in the software industry. The underlying cause for this tendency is that our communication patterns are inadequate. This leads to underdeveloped mental models, and developers approximating solutions to poorly articulated and understood problems by heaping on boolean flags and branching control flows.
Over time, software built this way becomes incomprehensible. The way to break this tendency is to change the way we build software: by collaborating and communicating better. Working in ensembles can be a step in the right direction, since it places collaboration and communication at the core of software development.
Einar W. Hst is a software developer at NRK, the Norwegian public broadcaster, where he helps build the TV streaming service. His main interests are domain modelling, API design and computer programming. He has a PhD in Computer Science from the University of Oslo. You can find him on Twitter as @einarwh or read his blog.
Read more:
Avoiding Technical Bankruptcy: a Whole-Organization Perspective on Technical Debt - InfoQ.com
- Boy Scouts' bankruptcy judge approves nearly $250 million in fees - Reuters - October 22nd, 2023 [October 22nd, 2023]
- What CARD's Clean Slate Post-Bankruptcy Could Mean For Autism ... - Behavioral Health Business - October 22nd, 2023 [October 22nd, 2023]
- 8 Senators Ask U.S. Treasury to Consider 'Going Concern' Bid to ... - International Brotherhood of Teamsters - October 22nd, 2023 [October 22nd, 2023]
- A Burger King franchisee gets sold out of bankruptcy, and a big buyer is Burger King - Restaurant Business Online - September 28th, 2023 [September 28th, 2023]
- Furniture-maker Noble House falls into bankruptcy, owes overseas ... - Retail Dive - September 28th, 2023 [September 28th, 2023]
- How to Get Your Own "Free" Private Investigator in Bankruptcy - Ward and Smith, PA - September 28th, 2023 [September 28th, 2023]
- Large Corporate Bankruptcy Filings Surged in First Half of 2023 ... - ABL Advisor - September 28th, 2023 [September 28th, 2023]
- MV Realty Files For Chapter 11 Bankruptcy - The Real Deal - September 28th, 2023 [September 28th, 2023]
- Vesttoo case: Time to "stop litigating and start cooperating" says ... - Artemis.bm - September 28th, 2023 [September 28th, 2023]
- BlockFi Bankruptcy Plan Receives Approval - Watcher Guru - September 28th, 2023 [September 28th, 2023]
- From Bank To Bankruptcy: How This Crypto Scam Toppled A US ... - Bitcoinist - September 28th, 2023 [September 28th, 2023]
- Mainstream Media Objects to Further Customer Data Redactions in ... - Cryptonews - September 28th, 2023 [September 28th, 2023]
- CARD's Founding CEO Returns to the Helm: What's Next for the ... - Behavioral Health Business - September 28th, 2023 [September 28th, 2023]
- "Calvin Klein was once on the verge of bankruptcy": BTS' Jungkook's ... - Sportskeeda - September 28th, 2023 [September 28th, 2023]
- Lerner Family Fight: Father's Actions led to Bankruptcy, son Says - The Real Deal - September 28th, 2023 [September 28th, 2023]
- Mercy Foundation asks court to clarify its role in bankruptcy - The Gazette - September 28th, 2023 [September 28th, 2023]
- Celsius bankruptcy auction nears end, with Fahrenheit in the lead - Reuters - May 18th, 2023 [May 18th, 2023]
- Surge in corporate bankruptcies just the start for debt-loaded firms - Business Insider - May 18th, 2023 [May 18th, 2023]
- 7 Companies Declare Bankruptcy Over Weekend in 15-Year Record - PYMNTS.com - May 18th, 2023 [May 18th, 2023]
- Credit Crunch Fuels 48-Hour Bankruptcy Rush With Seven Filings - Yahoo Finance - May 18th, 2023 [May 18th, 2023]
- Private equity-backed Envision Healthcare files for bankruptcy - Financial Times - May 18th, 2023 [May 18th, 2023]
- Drobo, having stopped sales and support, reportedly files Chapter 7 bankruptcy - Ars Technica - May 18th, 2023 [May 18th, 2023]
- Roxby Bankruptcy Cases Dismissed, Foreclosure Action Expected ... - Wheeling Intelligencer - May 18th, 2023 [May 18th, 2023]
- Voyager bankruptcy plan approved, customers may recover 35.7% of claims initially - Cointelegraph - May 18th, 2023 [May 18th, 2023]
- Bed Bath and Beyond's bankruptcy could help TJ Maxx owner TJX - Yahoo Finance - May 18th, 2023 [May 18th, 2023]
- IronNet: Near-Term Bankruptcy Filing Might Be In The Cards - Sell ... - Seeking Alpha - May 18th, 2023 [May 18th, 2023]
- Filing For Bankruptcy Protection Due To PFAS Litigation Costs - The National Law Review - May 18th, 2023 [May 18th, 2023]
- Full list of retailers that have filed for bankruptcy as shoppers go online to find deals... - The US Sun - May 18th, 2023 [May 18th, 2023]
- Quadriga CX Bankruptcy Claimants to Get 13% on the Dollar - CoinDesk - May 18th, 2023 [May 18th, 2023]
- Sorrento Therapeutics, Inc.'s Bankruptcy Court Orders Brokerage Firms to Credit Dividended Scilex Stock to Customers' Accounts and Sorrento Advises... - May 18th, 2023 [May 18th, 2023]
- Failing ventures and Twitter hiatus spark Tai Lopez bankruptcy ... - Protos - May 18th, 2023 [May 18th, 2023]
- Facing bankruptcy, Highland Park settles lawsuit over seized ... - WXYZ 7 Action News Detroit - May 18th, 2023 [May 18th, 2023]
- Judge in archdiocese bankruptcy case recuses himself over donations scandal - The Guardian US - April 30th, 2023 [April 30th, 2023]
- Revlon Taps New Directors as Lenders Take Control in Bankruptcy - The Wall Street Journal - April 30th, 2023 [April 30th, 2023]
- Opinion: Highland Park shouldn't declare bankruptcy over water debt - Detroit Free Press - April 30th, 2023 [April 30th, 2023]
- In Kolkata, India's new bankruptcy law is put to the test - Financial Times - April 30th, 2023 [April 30th, 2023]
- Judge in Catholic bankruptcy recuses over church donations - The Associated Press - April 30th, 2023 [April 30th, 2023]
- Judge in New Orleans Roman Catholic bankruptcy recuses himself over church donations - PBS NewsHour - April 30th, 2023 [April 30th, 2023]
- Four companies indirectly owned by Berkshire Hathaway file for bankruptcy (NYSE:BRK.B) - Seeking Alpha - April 30th, 2023 [April 30th, 2023]
- Pfizer Buys Lucira Health for $36.4M Through Bankruptcy Auction - GenomeWeb - April 30th, 2023 [April 30th, 2023]
- Bed Bath & Beyond shares drop after it files for bankruptcy protection - Detroit Free Press - April 30th, 2023 [April 30th, 2023]
- Bankruptcy Battle Breaks Out Over Greenwich Village Dev Site - The Real Deal - April 30th, 2023 [April 30th, 2023]
- US companies in distress turn to debt exchanges to dodge bankruptcy - Financial Times - April 30th, 2023 [April 30th, 2023]
- National retailer sold out of bankruptcy, will close remaining South ... - columbiabusinessreport.com - April 30th, 2023 [April 30th, 2023]
- Companies facing bankruptcy do this: Internet irate after Elon Musk led-Twitter reportedly slashed parental leaves - Business Today - April 30th, 2023 [April 30th, 2023]
- Tim Ziss Files For Bankruptcy To Extend Contract For Midtown Office - The Real Deal - April 30th, 2023 [April 30th, 2023]
- Future at Conway plant unclear after bankruptcy - Arkansas Online - April 30th, 2023 [April 30th, 2023]
- J&J Uses Cajoling, $9 Billion Offer to Sell New Talc Bankruptcy - Bloomberg Law - April 30th, 2023 [April 30th, 2023]
- Donald Trump and 5 Other Famous People Who Filed for Bankruptcy and What Happened Next - Yahoo Finance - April 30th, 2023 [April 30th, 2023]
- Beloved retailer and Michaels rival to close nine more stores amid bankruptcy fears see the full list of... - The US Sun - April 30th, 2023 [April 30th, 2023]
- Process - Bankruptcy Basics | United States Courts - March 31st, 2023 [March 31st, 2023]
- 10 years after bankruptcy scare, village is paying $2.5M in cash for new Village Hall - MLive.com - March 31st, 2023 [March 31st, 2023]
- Bankruptcy: What You Need to Know - NerdWallet - February 7th, 2023 [February 7th, 2023]
- DCG Creditor Pact Revealed With Plan to Sell Genesis Trading Unit as Part of Bankruptcy - CoinDesk - February 7th, 2023 [February 7th, 2023]
- Staring Down Bankruptcy, Bed Bath & Beyond Says It Will Sell Stock - The New York Times - February 7th, 2023 [February 7th, 2023]
- Pet store with several Mass. locations files for bankruptcy, will cease operations at end of month - Boston 25 News - February 7th, 2023 [February 7th, 2023]
- What Is Bankruptcy? Forbes Advisor - January 19th, 2023 [January 19th, 2023]
- Filing for Bankruptcy: What to Know | Consumer Advice - January 19th, 2023 [January 19th, 2023]
- Bankruptcy | California Courts | Self Help Guide - January 19th, 2023 [January 19th, 2023]
- Bed Bath & Beyond could file for bankruptcy 'as early as this weekend': Macco CEO - Yahoo Finance - January 6th, 2023 [January 6th, 2023]
- Bankruptcy Basics Glossary | United States Courts - October 23rd, 2022 [October 23rd, 2022]
- Cash crunch caused Thrifty Propane to file for bankruptcy: statement - WJW FOX 8 News Cleveland - October 19th, 2022 [October 19th, 2022]
- Chinese Steel Manufacturers On The Brink Of Bankruptcy - OilPrice.com - October 19th, 2022 [October 19th, 2022]
- Pennsylvania Bankruptcy Court Finds That Consent and Release Do Not Protect Against Wrongful Foreclosure Claim - JD Supra - October 19th, 2022 [October 19th, 2022]
- Congressional Candidate Critical of Loan Forgiveness Sought to Have Student Debt Cleared Via Bankruptcy - NBC4 Washington - October 19th, 2022 [October 19th, 2022]
- Bloomberg Tax Introduces New Portfolio Addressing Employee Benefits and Bankruptcy - PR Newswire - October 19th, 2022 [October 19th, 2022]
- Alex Jones claims bankruptcy: How will Sandy Hook families get paid? - Grid - October 19th, 2022 [October 19th, 2022]
- Bankruptcy - Michigan - October 17th, 2022 [October 17th, 2022]
- PFC to take RattanIndia plant to bankruptcy court this year | Mint - Mint - October 17th, 2022 [October 17th, 2022]
- Significant Fee increases for Bankruptcy and Winding-Up Petitions on the way - Lexology - October 17th, 2022 [October 17th, 2022]
- US Bankruptcy Court NJ - Live Database - October 15th, 2022 [October 15th, 2022]
- It Is Time To Change Bankruptcy Forum Laws - Patent - United States - Mondaq - October 13th, 2022 [October 13th, 2022]
- Judge in Archdiocese of Santa Fe bankruptcy sets proposed timeline to pay abuse victims - Yahoo News - October 13th, 2022 [October 13th, 2022]
- CMS Protests QHC Bankruptcy Plan Will Not Cover $1M Owed to Agency - Skilled Nursing News - October 13th, 2022 [October 13th, 2022]
- As Jury Deliberates In CT, Alex Jones And His Lawyers Are Setting Fire To Each Other In TX Bankruptcy Court - Above the Law - October 13th, 2022 [October 13th, 2022]
- Student loans are notoriously difficult to discharge in bankruptcy. A new bill could change that - Fortune - October 13th, 2022 [October 13th, 2022]
- Pioneering computer science alumnus battles bankruptcy and more on road to graduation - Oklahoma State University - October 13th, 2022 [October 13th, 2022]
- J&J Talc Bankruptcy 'Bad Faith' Claims Go Before Third Circuit - Bloomberg Law - September 20th, 2022 [September 20th, 2022]
- New amendments to bankruptcy code can maximise recovery of stressed assets - DNA India - September 20th, 2022 [September 20th, 2022]
- Judge reports threats, harassment over J&J talc bankruptcy - Reuters - September 17th, 2022 [September 17th, 2022]