The Cost of Technical Debt in Analytics

In the world of data and analytics, technical debt is what happens when organizations make conscious decisions to solve short-term problems, even when they know there could be long-term and potentially negative implications around their actions. It exists in every organization, including yours. Not all technical debt is bad, as long as it is strategically planned and tactically paid off.

Taking a shortcut and implementing a cheap, short-term solution to a problem is usually how technical debt starts. These shortcuts are often necessary for business continuity, but letting a temporary fix become a permanent or long-term solution results in other major problems and expenses down the road. Much like regular debt in our lives, technical debt must be paid off at some point.

Symptoms of Technical Debt in Analytics

• Enterprise systems are in serious need of upgrading
• Spreadsheets are being used to share or manage data
• Highly customized assets exist (SQL, JavaScript, etc.)
• There are overly engineered semantic/metadata layers in place

Here’s a common analytics example that illustrates how technical debt starts building. You are tasked with building out complex reporting against an operational data source. You realize the strategic path to pursue is to get this data into your warehouse.  However, given the short turnaround time that is being requested, you build a functioning metadata model directly against the source data. Unit testing does not expose many issues. When you roll it out to 20 analysts, report run-times become unwieldy, highlighting a performance issue.  After some time, analysts begin exporting data and manipulating it on their own, claiming it is faster. Not only did you create an unusable solution, you also amassed technical debt.  It’s time to pay things back and build that data mart.

However….

Because of the lackluster sentiment created by slow response, few are likely to support your initiative, not wanting to spend the time and money to fix the performance issues.  They are getting by with their spreadsheet-based solution, further adding to the debt you incurred with your original metadata-based reporting solution. Think of this as compounding interest on your initial debt. The longer you “deal” with the problem, the more you pay these costs.

Source: http://commadot.com/the-ux-of-technical-debt/

How Technical Debt in Analytics Affects You

• Convenience: Temporary solutions that were devised as a means of convenience become increasingly more inconvenient as they pile on top of each other (think of the prior example – the metadata model that gave way to the spreadsheet).
• Performance: Lots of one-off, short-term solutions make for fragmented systems that don’t integrate well and provide competing versions of data, causing unnecessary rework and affecting both employee and system performance.
• Slow Response: Your ability to react to the pace of business is handicapped when you’re forced to deal with repaying debt.
• Cost of Change (CoC): Optimal CoC of a system increases as it becomes more complex (increased data sources, data volumes, and added capabilities). CoC of a system with technical debt increases exponentially as the system becomes more complex or time goes on.
• Loss of Business Opportunity: Data is the backbone of a company, and if you lack the well-planned/integrated infrastructure needed to access it efficiently and make clear decisions in a timely fashion, everything downstream gets affected from your employees to customer-facing activities to revenue. This is what’s called an opportunity cost, and it often accompanies technical debt.

 

Source: http://www.industrialmarketer.com/marketers-technical-debt/

Your Options

Do Nothing: Kick the can down the road and hope for the best. You may be fine for a few years, but at some point this snowball of technical debt will require a massive effort (cost) to rectify. Disruption in services, questions about preparedness and effectiveness of data analytics, internal frustration, and loss of business are all risks that arise from neglecting to paying off technical debt.

Explore Options for Tactical Enhancements: Begin identifying your technical debt, cataloging it, and developing a prioritized plan to “pay it off” in an efficient manner. Be tactical about your plan. Most projects aren’t likely to fix things in a “big bang” method. Get help. Consulting with experts ensures that proven designs are implemented in the first attempt.

 

There are plenty of organizations who are dealing with similar issues. At Ironside we’ve been working with our customers to assess, manage, and unify solutions for decades. If your organization is preparing to implement new technology to curb technical debt, the goal will be to have fewer manual workarounds and more focus on a unified process. Keep in mind that implementing an analytical tool is only the beginning. The process often requires end users to evolve their role which will put your company in a much better position to achieve successful user adoption.

 

About Ironside

Ironside was founded in 1999 as an enterprise data and analytics solution provider and system integrator. Our clients hire us to acquire, enrich and measure their data so they can make smarter, better decisions about their business. No matter your industry or specific business challenges, Ironside has the experience, perspective and agility to help transform your analytic environment.

References

1. Matthews, Brian. “Why Marketers Need to Start Thinking about Technical Debt.” Industrial Marketer. Oct. 4, 2016. http://www.industrialmarketer.com/marketers-technical-debt/

2. Lipka, Glen. “The UX of Technical Debt.” commadot.com. Feb. 15, 2011. http://commadot.com/the-ux-of-technical-debt/