By Michael D. Stark.
Most of us have had exposure to the remains of some system replacement that didn’t quite finish decommissioning the legacy system. Perhaps it still performs some current functions, or is simply used to access historic or run off data.
We often feel this is problematic and the result of a lack of commitment, but is it really inevitable, and perhaps even logical?
the analogy
A good friend of mine told me a story, or perhaps a legend, told to him at the shipping company he was working for. It just completely captures the essence of the decommissioning problem.
Like all good technology analogies it starts with a consignment of frozen Mackerel.
As the story goes there was a shipping container full of frozen Mackerel (Fish) that was being shipped from Ghana, West Africa, to a customer in Europe, passing through some very hot areas including Dubai on the way. At the Port of Jebel Ali, Dubai, someone notice a rather unpleasant smell, and upon inspection found that the refrigerator had been turned off. This was potentially a major headache deal with, so they simply turned it back on and sent it on its way to Europe.
Arriving at the Port of Amsterdam where it was scheduled to transfer to a truck, it first needed inspection. Customs crack the container to check it prior to it being cleared for entry into the Netherlands. The container was one massive block of frozen rotten fish. A blend of water from what was the packing ice, Mackerel, and mangled cardboard. The door was quickly closed again so it could be returned to sender, as the regulations for disposing of such cargo meant it would be an expensive, and very messy, task for the port.
During the elapsed time of this journey the fishery company in Ghana that had sent the container of Mackerel had gone bankrupt, and so the port could neither send the rotten frozen fish back to West Africa, nor let it into the country to send it forward to its intended recipient. This was a massive headache for the manager responsible for dealing with it; whose targets didn’t accommodate the time and cost of this cleanup. Then one of the team pointed out that they could put it out back in an unused corner of the port and just leave it plugged in. No cost, and no real downside. Job done. A couple of years later the manager had moved on, and then one hot summers day a couple of year after that a worker noticed a rather foul smell coming out of the container. The refrigerator motor had seized. Nobody knew what the container was even there for at first, but one of the cynical old boys did remember it.
Opening the container only to be hit with a wall of foul smell, the new manager quickly closed the door and it took them less than a minute to determine it was much quicker and cheaper to just fix the refrigerator motor. The same exact thing happened 5 years later to a different manager, with the same outcome, just fix the motor and leave it alone, but with one difference. He was a more diligent manager and wanted to do the right thing, so he organised for regular maintenance of the refrigeration unit.
“Never put off until tomorrow what you can avoid altogether” – Mark Twain
According to the legend the container is still humming away some 25 years later. True or not does anybody doubt that this is a perfect analogy for the non-decommissioned legacy system?
picking the eyes out
This strange expression is quite common in New Zealand and Australia, but apparently not so common elsewhere. It basically means picking out the good bit and leaving a carcass of little use.
The analogy about the rotten Mackerel in the container needs no further explanation in regards to how it applies to the psychology of what people do in such situations. It is all quite logical with rotten fish and hanger-on legacy systems. If we can kick the can down the road then why wouldn’t we. But how do these systems get like that.
Well typically it is all about bring the value forward during system replacement programs. It makes good sense. Aim for the high value first, and pick the eyes out. What you are left with is a few functions that typically don’t have a very strong ROV (Return of Value) for a replacement business case. With a lot of old mainframe systems it is common to move them to a low cost Windows platform, which make sense. But this further reduces the business case for completely decommissioning it. Few of us are able to get a case for a 20 year ROV horizon approved.
As architects, we get frustrated with these old systems hanging around, particularly from a systems thinking viewpoint when we can see the accumulative impact on the overall enterprise. But assuming people act logically given their current conditions then it largely makes sense, and is almost inevitable when you add it all up. So what is the answer? What are we to do?
The best way to deal with this situation would be to ensure the original program has a governance gate for success that includes the full decommission of the legacy system. But to be honest in the heat of battle when sponsors are under pressure this is likely to get traded away with perhaps a risk assessment and the good old sponsor risk acceptance trick.
Alternatively if you have lots of these old containers full of rotten Mackerel out back you can push to get a strategy endorsed at a high enough level that individual business cases to decommission are not needed. So typically making it a strategic business priority if you can.
I don’t know the answer, but perhaps the logical thing to do is to find a cool dark corner out back, plug in the refrigerator, and let it be. Just don’t forget a bit of maintenance if you do want to stink the place up. Sorry, but in the end, given the condition we operate in, it often makes sense.
thinking humanistically about enterprise architecture 
Enterprise Architecture is about the total cost of ownership. If you leave your fish lying around long enough eventually it’s going to cost you more than biting the bullet at the start would have. But how much greater isn’t always immediately obvious. But it may not just be the cost of running the freeze; IT is much more complex and interconnected than a freezer container.
Some years ago I conducted a budgetary analysis of a government department’s IT projects. They had been saving money for years by not decommissioning. But they couldn’t figure out why more recent projects seemed to be more expensive.
It turned out that about 33% of the budget of every new project was being consumed “integrating” and “accommodating” systems that should have been decommissioned long before the current projects were even an idea. It wasn’t just capital technical debt it was operational technical debt.
What was even more alarming was the that this proportion was rising by about 2-3% a year and seemed to be accelerating!
LikeLike
Thank you for your insight Dr Hope. I fully agree, and having some hard numbers to back up the accepted intuition is useful. The challenge is misaligned incentives for decision makers, who are often being judged by in year TCE (total controllable expenses). This of course comes from the top down, driven ultimately by the need for Black Rock and other institutional investors, or government budgets, or whoever else pulls the purse strings.
I’m not prone to quote Hillary Clinton, but I do recall her observing that [Western] companies will always struggle to compete with China while China focus on 5+ year outcomes and [Western] companies focus on the next quarterly results.
Unpacking ‘technical debt’ is key. IT think it is a key decision driver, and the business side tend to role their eyes. We need to find a way to speak the same language, in a way that actually motivates both sides to action. If we are telling a senior manager that it will cost them in 3 years time, yet they know they will get shuffled out long before the benefits or costs are realized, so it is understandable that when the trade-off concerns tech debt doesn’t make it.
The other problem is that often we they do listen to IT about reducing costs the overall cost still seem to increase overall. But we do need to find better ways to tell the story and build trust.
LikeLike
Nothing truer was ever said than you are perfectly aligned for the results you are getting and books like Freakonomics and The Tyranny of Metrics demonstrate the peril of that. I believe Walmart once compiled monthly sales figures, but stopped after the stock market got wind of it and demanded their release! I wonder what they would make of the Swedish practice of four monthly rather than quarterly reporting. The logic is simple a quarter year is too short and a half too long.
You touch on two issues that affect the legitimacy of architecture. The first is cost. The most stupid, but unfortunately all too common, business statement is “that’s too expensive”. I watch architects accept this all the time and it is nonsense. This shouldn’t be a surprise when dealing with accountants but it seems to floor most architects! The response is “too expensive compared with what?” There’s a lot to be said for the proposition that without an alternative you don’t have a problem.
Accountants see the world from a very limited viewpoint to quote Nietzsche it is the nature of the beast. A viewpoint that is accepted even by the profession as inadequate, where do you think the Balanced Score Card came from? And sitting right at the heart of the of this profession are two ambiguities that undermine all claim to rationality, depreciation and intangible value, both huge balance sheet numbers that are completely made up. Accountancy can be fairly described as incredibly precise and totally inaccurate.
To use an accounting term the bottom line is Technical Debt is stealing or at the very least fraud. That’s something that accountants understand. When you intentionally incur Technical Debt you are stealing from the rest of the organization; you might benefit but you’re expecting everyone else to pay for it; and pay over the odds to add salt to the wound. You’re stealing from the shareholders, you’re stealing from the other managers, the organization in general and you are stealing from the future because sooner or later it’s going to have to be fixed. It’s like not servicing your car and hoping it doesn’t break down. And like all crimes there are always unseen side effects (Unintentional Technical Debt) just ask corporate risk. The largest Technical Debt I ever encountered ($400m and still accumulating at around $20m a year) was clearly the unintentional consequence of saving $20m.
Enterprise Architecture is supposed to be holistic; the problem is that typically it’s not. It’s often the end point of an architect’s technical career. And they carry a similarly limited viewpoint often devoid of any business knowledge. Sure they know their business, but that is process knowledge mistaken for business knowledge. They don’t know who Michael Porter is. They absorbed a few business terms and go about sprouting them, but few understand them. Don’t believe me? Then test them, ask to them explain the difference between competitive and comparative advantage. Alarmingly that’s all too commonly often enough to shut the business up too. But that’s another topic.
That brings me to the second issue authority. Where the Enterprise Architecture team sits in the organization is critical and all too often it sits in the IT department. The research is clear EA teams should report to the CEO or the board, it’s supposed to be holistic right? The performance of corporations with this structure is clearly superior. So why doesn’t that happen? Because EA is perceived as IT centric and the architects through no fault of their own and by virtue of their career progression confirm that. In many organizations EA is simply not fit for purpose and the fault lies in EA as a discipline. So every three years or so: according to Gartner 40% of all EA practices are abolished, restructured, defunded or otherwise disposed of.
LikeLike
Fish is not the worst things that have been stuffed in containers, at times not even paying for a reefer (freezer container) . . . . .
The continuity in temperature, is one of the reasons why containers are being equipped with sensors, mostly important for reefers but also for standard dry equipment used to transport chemicals etc.
Back to decommissioning, splendid article!!!
Having that reefer humming away carries more than deference of cost; there is a constant drain
It is not much different from legacy decommissioning, it is difficult to get to full decommissioning; there will be a remainder that gets exponentially difficult/expensive to get out of:
Deciding to decommission, must be formalized in a strategy->roadmap where the objective is the complete annihilation of the legacy and first then can the benefits be harvested. Business and IT needs to be fully aligned, because I will almost guarantee that it will be an awfully poor business case to use technology only to clear out the legacy.
E.g. it can be sensible to give clients/customers/partners a (positive) incentive to move to products or services offered on the new platform, avoiding a costly and potentially unsatisfied clients/customers/partners.
Getting data transformed and achieved, is one of the final steps in decommissioning and here it is important to remember compliance and semantics.
Debt is not just cost that is being deferred; it is a constant drag on velocity and a constant drain on budget. Most decommissioning do not have a good ROI before completely annihilated and the longer it takes the worse the ROI will be. eg. MIPS/VCORES etc the less that the enterprise commits to, the more expensive they will be and the people maintaining and operating the legacy will have to be retained until until system or their EOL.
A little ranting on migration: https://www.linkedin.com/pulse/data-migration-opportunity-improved-quality-kristian-leth-jvkde
And a little on simplicity: https://www.linkedin.com/pulse/from-surviving-winning-keeping-landscape-lean-fit-kristian-leth-3kmue
LikeLiked by 1 person
Thanks for such a well considered response Kristian. It does seem like we have seen the same thing in companies, and drawn the same conclusions.
It is important that architects and other technologists get their heads around the ‘alignment’ of interests challenges here (even if short term).
LikeLike