Data Sharing is a Responsibility

I was recently asked to sit in on a client meeting focused on data architecture strategy. As you can probably guess, there was a diverse audience of technology folks – each with their pet projects and agendas.  Application developers wanted to add data services to the existing SOA initiative; the ETL team wanted improved source system data access; the Big Data junkies wanted to increase Hadoop visibility; and the data administration team was pounding the metadata drum (yet again). Everyone seemed to be in attendance and it appeared that everyone was focused on a variation of the same need: data sharing.

When I asked if there were any data sharing policies or data provisioning standards, I got a bunch of weird looks.  Apparently no one perceived that data sharing was their responsibility. The ETL folks said they only processed source data for the data warehouse (and no one else).  The ESB folks said they didn’t own the data, they just moved it; the Big Data guys said sending out copies of their data wasn’t a priority. It was interesting – everyone believed data sharing was someone else’s problem.  “I’m measured on my SLAs. I don’t have the budget or staff to support everyone that wants data”

Wow!

The concept of data sharing seems to have taken a weird path over the past few years.  While everyone claims that data is a corporate asset, when it comes to getting data, everyone is on their own.  Every project team that relies on data created within another application (a system of creation) has to bite off the entire development effort to search, find, extract, transform, and deliver data from the system of creation to support their own needs. (So, if there are 5 teams that need customer data, they each build their own code to retrieve customer data from the source system.) This is like telling every grocery store manager that they each need to have their own purchasing, logistics, delivery, and warehouse team in order to stock their shelves. No grocer handles product delivery in this manner – logistics and distribution is very costly and inefficient when implemented as a distributed process. It’s well understood that if you have a limited number of suppliers and a defined inventory, it’s better to leverage an economies-of-scale approach.

It seems to me that if multiple teams need access to the same data, shouldn’t there be a central solution that is shared by everyone?  Instead of forcing every application to own data sharing, why not move the responsibility into the hands of the folks that produce the data? It’s accepted that data sourcing can consume 30%-40% of new application development (operational, analytical, or even packages). The cost is staggering because this distributed approach forces dozens (if not hundreds) of developers to learn the data storage intricacies within an application that they neither own nor manage. (The most popular ERP package contains more than 10,000 tables – all represented in German)   Why not steal a page from the world of reengineering (or common sense) and gain the benefits of centralizing a highly distributed, repetitive, and costly process?  Data sharing is highly suited to an economies-of-scale approach.

Maybe it’s time to expand the responsibilities of application development teams to include both application functionality and data sharing.  We’re not really insisting on an IT paradigm shift; we’re talking about shifting data sharing responsibilities into the hands of the folks that actually generate the data.  Then again, if your desire is to train lots of developers on the German language – there’s certainly a lucrative opportunity for you.

Photo “Olympic Provisions” courtesy of jenarr via Flickr (Creative Commons license).