Technical Debt – The # 1 Concept That Manufacturing Executives Must Understand To Secure Agility And Competitiveness.
Arguably the single most important software concept for manufacturers to understand as they move towards an advanced manufacturing future is that of “Technical Debt”. Operating without a core understanding of Technical Debt, how it is incurred and appropriate mitigation strategies is risky business.
Without insight into Technical Debt there is every chance that instead of the intended productivity and agility outcomes, manufacturers will find themselves becoming slower and slower to respond to change. Manufacturers can become anchored to their installed technologies while being blind to the reasons why.
This article provides an overview of Technical Debt as it pertains to manufacturing automation and controls. Its purpose is to raise awareness and stimulate discussion on the topic within the manufacturing industry. We will:
- Put forward a definition of Technical Debt that makes sense to us based on our experience of 30 years of practice.
- List some typical signs and symptoms of Technical Debt.
- Show how using a financial analogy leads to better management decision on technical matters.
Definition of technical debt
In the early 1980s at ICT we were developing custom designed hardware and software products for industry. As a way of branding, all our own developed products were housed in distinct “black boxes” or cabinets. This was a reflection that many of our end customers were only concerned about the inputs and outputs to our product and had no interest its internal workings.
Today, even though our customers are much more educated about computer technology a “cone of silence” exists around the software development task. A common language that allows for effective discussions between business and engineering software professionals is missing.
Consider the following situation:
A Business Manager sees that his project is in danger of going over time. He decides to bring forward the completion deadline by 4 weeks and instructs his software engineer accordingly.
The software engineer explains that shortening the timeframe will affect the end quality of the product but is unable to make a persuasive argument and he leaves the meeting agreeing to the new time frame. Later with his own team he begins to study what activities they don’t have to do right away. They decide they must leave the code in its prototype state. There is no time to review and update the code to make it more robust. Nor is there time to tidy the structure or add the comments to make the code more readable by others, and time only for scant supporting documentation.
The project is finished on time to the new schedule and the field tests are completed.
The business manager is happy because his schedule is met. The software engineer is happy because his code “worked”. Project closed.
Gradually however for operations staff, problems begin to emerge. The impact of the deferred activities is now being felt. For example it is taking more hours that expected to fault diagnose due to coding and documentation issues and thus unanticipated maintenance/support costs are accumulating.
In this situation the consequences of the actions of the business manager and the engineer ultimately affect the end customer productivity. The absence of meaningful discussion on software quality allowed short term factors to influence decisions with a detrimental consequence to the project’s overall business objective.
In 1992 Ward Cunningham was the first to attempt to break down this cone of silence when he used the language from the finance industry to describe the delayed impact/costs of decisions made during the software development process.
Using a financial analogy he created the term Technical Debt to describe the trade-off between expedient decisions made in the short term which generate a legacy debt and the resultant need for future essential expenditure in order to preserve system functionality.
Meaning that management’s choice is either to invest in the robust final solution up front (no technical loan albeit maybe a financial loan) or defer engineering effort to a later time and thus creating a technical loan and interest which is paid when the deferred work is done.
More recently Ted Theodoropoulos* offered a broader holistic definition, and one that we feel is more appropriate to the manufacturing sector. Consider his definition below:
“Technical Debt is ANY gap within the technology infrastructure, or its implementation, which has a material impact on the required level of quality.” With regards to software, quality is measured in terms of its functionality, reliability, usability, efficiency, maintainability and portability.
The use of “Any gap” includes both
- A deficit caused by work that should have been done but has not been done, thus accruing a need for future expenditure.
- A surplus where extra functionality than required is provided and thus bringing forward expenditure that may never have been needed.
Both cases impact operational cash flows.
It is important to note that the analogy origins stem from a large software development system environment however as automation and control systems adopt more and more IT software attributes the term is very relevant to the automation and control environment.
The analogy of Technical Debt brings to light that not all software is created equal. Software quality matters. Clean, robust, well-structured code, written with thought to the future is the key to an agile manufacturing environment. Quality design builds flexibility and speed into your systems allowing a faster uptake of a new business opportunity.
We emphasise that a lack of transparency of the software development activity means that the impact of the software decisions made in the course of a project is not visible to the customer. These shortcomings tend to be hidden as they are not revealed by functional testing.
Do you have technical Debt?
I am sure that you will recognise the signs and symptoms of technical debt even if not directly expressed under these terms.
For example consider the following:
- You are living by good luck rather than good management; you have a legacy system where not only are the hardware components unavailable but no one is left who really understands the code.
- A System fails and only the “As Commissioned” software backups can be found. All modifications and enhancements since commissioning are lost.
- Unexplained system faults occur even though the system passed all test conditions.
- A project suffers software commissioning delays and overruns with commissioning engineers performing many on site software modifications.
- A machine was purchased from overseas and requires significant subsequent modification to perform as intended and meet Australian Standards.
- A new system is selected at a cost premium due to the many extra functions and features provided over and above the core project requirements, however it is found that operational staff never progress to adopt any of the new functionality.
- Customisation of existing systems has made upgrading or migration to new products or systems either difficult or impossible without major expenditure of time and money.
The business link
Technical Debt is a useful analogy because it puts a business context around the software task.
Technical Debt has three major components: principal, interest and probability.
- Principal is the cost of the complete solution, the “proper job” – a “clean” interest free implementation.
- Interest is the cost of required remedial works, including the loss of productivity and other business cost impacts where part of the Principal is deferred for whatever reason.
- Probability is the risk that the interest will need to be unexpectedly repaid.
A prototype is developed quickly and installed on one machine in manufacturing company. All stakeholders were pleased with the results of the trial. The software engineer then presents a time, cost proposal to develop the production version which includes time for software review, optimisation of code and extension of core functionality to allow for individual machine variables. Management do not understand why this time and cost should be spent as to them the code already works. The prototype version is rolled out across the other machines in the plant. Each time it is implemented the prototype code requires modification and accordingly the code becomes more complex than needed, more fragile and less stable.
The Principal: The original budget was for 80 hours of engineering labour to move to a robust and more “portable” production version before roll out.
The Interest: The reality was that for every machine the penalty of modifications was an extra 8 hours.
For 12 machines the actual cost was 96 hours.
The extra cost of maintaining slightly different versions of the software 1 hour per machine compounding every year.
The probability: In this case 1.
If time was taken to convey an understanding of the consequences of roll out of a prototype version to management in terms of the Technical Debt analogy, rather than just the raw costs of the production version then a different result may have occurred.
It is also worthwhile to note that not all Technical Debt is unjustified, nor will all Technical Debt necessarily need to be repaid. Sometimes the level of risk associated with the debt is preferable to the consequences of a delay in the project.
Software is becoming more and more entrenched in nearly every product that we use and so too are the problems that sit alongside the software development process.
The project management concept of the time – cost – quality relationship forms part of a project manager’s tool kit. We advocate that the Technical Debt concept should also form part of this toolkit when applied to technology projects.
Even if you don’t use the above financial formula for calculating your Technical Debt, thinking about the analogy will give you insight into the software development decision making process. To better question and evaluate your various choices; building greater awareness of the implications of your decisions.
That thinking will:
- Provide a more accurate view of the risks relating to technology in your plant.
- Promote wise investment decisions as the economic value of different choices becomes clearer e.g. adopting new technology or remediating what is already in place.
- Bring to light the value of expert input early in a technology implementation project.
- Make your cash flows more predictable and manageable through the identification and management of your Technical Debt.
We have applied the Technical Debt analogy derived from the software industry and applied it to the manufacturing automation and controls environment.
In essence Technical Debt is about understanding and being able to guide the decisions made in the industrial software development process; to know that a system’s quality limitations reveal themselves not necessarily through performance testing or during commissioning but instead over time while being used and modified. Identification and reduction of Technical Debt allows for greater future flexibility and agility, as systems are more robust, and easier to upgrade, migrate, operate and maintain.
Spending time understanding how Technical Debt impacts on you and your organisation will enable you to improve the quality of your technology solutions, better forecast the return on your investments and increase your organisation’s productivity and agility.
*Ted Theodoropoulos, President: Acrowire Technology Consulting, www.acrowire.com
What has been your experience? We welcome your comments. Feel free to drop us an email on firstname.lastname@example.org or give us a call.