Here be (data) dragons – or why your data sucks

Me: “Did you know you have customer information in 17 places?”

Manager: “What?! Are you sure?”

Me: “Yeah. See I made this heat map…”

Manager: “Oh s**t…”

That’s my first memory of making a real impact at an organisation. So, I’d held roles of database responsibility in IT for years, but being a data geek, I frequently felt like I was spitting in the wind. “We have data quality issues”, “You need a better platform if you want to keep scaling up”, “You can’t integrate these two databases without a lot of work” — all of these statements had fallen on deaf ears. One visualisation circulated an organisation and suddenly there was realisation: our data sucks.

We were in a mess. My name is Edafe and I slay data dragons. This is as hard as you’d think but not for the reasons you’d imagine. Very rarely is a data dragon due to the infrastructure or technology. In my experience of small and medium enterprises, data dragons manifest due to people and processes. They are nudged to glory by organic architectural growth, technical debt and a lack of data strategy.

This is what happens: organisation X is growing. They started with a spreadsheet or maybe even an Access database (I know, I know!). Then they start to grow. “We need to get to market!” goes the cry. So we build an independent system (sort of). Then another. Then another. Everyone’s busy, busy, busy! Who has time to think about data modelling, integration and alignment? Who has time to plan this puppy? “Get it out the door!”.

Sometimes, it goes like this: “I just want to make a small change” which morphs into another change and another and another. Change management fails to spot that you’ve just changed the entire nature of a product or maybe it just wasn’t fit for purpose in the first place. Even worse: “Let’s reuse that field cos we can’t change the underlying database” uh oh…

Months, maybe years later, the high level estimates for a tiny change are huge. Stuff is all over the place. No-one knows what this system does, it breaks constantly and all we can do is feed the beast some support staff. Shall we rewrite? Oh hell no! We must keep moving forward! (Also, who the on earth knows what’s does under the hood?) We can’t afford to rewrite systems! C’mon, you’re in IT, why can’t it just work?

It got me thinking on my feet: there has to be a better way. A better way than declaring “So, here be data dragons, feed them support and development staff. and leave them be”. The solution wasn’t magic either; prod them with some whizzy methodologies: Agile will save us! Buy moar Visual Studio! Use BIML! It also wasn’t the nuclear option: Raze it to the ground boys! I love the smell of wholesale rewrites in the morning!

Slaying a data dragon needs a clear data strategy aligned to your organisation’s strategy. It needs people committed to resolving data quality and architecture issues. Why? You can’t big data, data science or analytics successfully on a poor platform. If your data quality and architecture suck, so will any solutions you build on them.

Bottom line? Data dragons don’t just manifest, they’re born in every decision made, by people and about processes: in a lack of strategy, communication, architecture and growth without direction or alignment.

Congratulations, you have data dragons.

Well… looks like you need a data strategy.

Published by


Consultant: Data Science & Data Analysis – Making the complex, simple

2 thoughts on “Here be (data) dragons – or why your data sucks”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s