I presented a webinar a few weeks back that challenged the popular opinion that the only way to be successful with data science was to hire an individual that has a swiss army knife of data skills and business acumen. (The archived webinar link is http://goo.gl/Ka1H2I )
While I can’t argue on the value of such abilities, my belief is that these types of individuals are very rare, and the benefits of data science is something that can be valued by every company. Consequently, my belief is that you can approach data science successfully through building a team of focused staff members, providing they cover 5 role areas: Data Services, Data Engineer, Data Manager, Production Development, and the Data Scientist.
I received quite a few questions during and after the August 12th webinar, so I thought I would devote this week’s blog to those questions (and answers). As is always the case with a blog, feel free to comment, respond, or disagree – I’ll gladly post the feedback below.
Q: In terms of benefits and costs, do you have any words of wisdom in building a business case that can be taken to business leadership for funding
A: Business case constructs vary by company. What I encourage folks to focus on is the opportunity value in supporting a new initiative. Justifying an initial data science initiative shouldn’t be difficult if your company already supports individuals analyzing data on their desktops. We often find collecting the existing investment numbers and the results of your advanced analytics team (SAS, R, SPSS, etc.) often justifies delving into the world of Data Science
Q: One problem is that many business leaders do not have a concept of what goes into a scientific discovery process. They are not schooled as scientists.
A: You’re absolutely correct. Most managers are focused on establishing business process, measuring progress, and delivering results. Discovery and exploration isn’t always a predictable process. We often find that initial Data Science initiatives are more likely to be successful if the environment has already adopted the value of reporting and advanced analytics (numerical analysis, data mining, prediction, etc.) If your organization hasn’t fully adopted business intelligence and desktop analysis, you may not be ready for Data Science. If your organization already understands the value of detailed data and analysis – you might want to begin with a more focused analytic effort (e.g. identifying trend indicators, predictive details, or other modeling activities.) We’ve seen data science deliver significant business value, but it also requires a manager that understands the complexities and issues of data exploration and discovery.
Q: One of the challenges that we’ve seen in my company is the desire to force fit Data Science into a traditional IT waterfall development method instead of realizing the benefits of taking a more iterative or agile approach. Is there danger in this approach?
A: We find that the when organizations already have an existing (and robust) business intelligence and analytics environments, there’s a tendency to follow the tried and true practices of defined requirements, documented project plans, managed development, and scheduled delivery. One thing to keep in mind is that the whole premise of Data Science is analyzing data to uncover new patterns or knowledge. When you first undertake a Data Science initiative, it’s about exploration and discovery, not structured deliverables. It’s reasonable to spin up a project team (preferably using an iterative or agile methodology) once the discovery has been identified and there’s tangible business value to build and deploy a solution using the discovery. However, it’s important to allow the discovery to happen first.
You might consider reading an article from DJ Patil (“Building Data Science Teams“) that discusses the importance of having a Production Development role that I mentioned. This is the role that takes on the creation of a production deliverable from the raw artifacts and discoveries made by the Data Science team
Q: It seems like your Data Engineer has a similar role and responsibility set as a Data Warehouse architect or ETL developer
A: The Data Engineers are a hybrid of sorts. They handle all of the data transformation and integration activities and they are also deeply knowledgeable of the underlying data sources and the content. We often find that the Data Warehouse Architect and ETL Developer are very knowledgeable about the data structures of source and target systems, but they aren’t typically knowledgeable on social media content, external sources, unstructured data, and the lower details of the specific data attributes. Obviously, these skills vary from organization to organization. If the individuals in your organization are intimate with this level of knowledge, they may be able to cover the activities associated with a Data Engineer.
Q : What is the difference between the Data Engineers and Data Management team members?
A: Data Engineers focus on retrieving and manipulation data from the various data stores (external and internal). They deal with data transformation, correction, and integration. The Data Management folks support the Data Engineers (thus the skill overlap) but focus more on managing and tracking the actual data assets that are going to be used by data scientists and other analysts within the company (understanding the content, the values, the formats, and the idiosyncrasies).
Q: Isn’t there a risk in building a team of folks with specialized skills (instead of having individuals with a broad set of knowledge). With specialists, don’t we risk freezing the current state of the art, making the organization inflexible to change? Doesn’t it also reduce everyone’s overall understanding of the goal (e.g. the technicians focus on their tools’ functions, not the actual results they’re being expected to deliver)
A: While I see your perspective, I’d suggest a slightly different view. The premise of defining the various roles is to identify the work activities (and skills) necessary to complete a body of work. Each role should still evolve with skill growth — to ensure individuals can handle more and more complex activities. There will continue to be enormous growth and evolution in the world of Data Science in the variety of external data sources, number of data interfaces, and the variety of data integration tools. Establishing different roles ensures there’s an awareness of the breadth of skills required to complete the body of work. It’s entirely reasonable for an individual to cover multiple roles; however, as the workload increases, it’s very likely that specialization will be necessary to support the added work effort. Henry Ford used the assembly line to revolutionize manufacturing. He was able to utilize less skilled workers to handle the less sophisticated tasks so he could ensure his craftsmen continued to focus on more and more specialized (and complex) activities. Data integration and management activities support (and enable) Data Science. Specialization should be focused on the less complex (and more easily staffed) roles that will free up the Data Scientist’s time to allow them to focus on their core strengths.
Q: : Is this intended to be an Enterprise wide team?
A: We’ve seen Data Science teams be positioned as an organizational resource (e.g. specific to support marketing analytics); we’ve also seen teams set up as an enterprise resource. The decision is typically driven by the culture and needs of your company.
Q: Where is the business orientation in the data team? Do you need someone that knows what questions to ask and then take all of the data and distill it down to insights that a CEO can implement.
A: The “business orientation” usually resides with the Data Scientist role. The Data Science team isn’t typically setup to respond to business user requests (like a traditional BI team); they are usually driven by the Data Scientist that understands and is tasked with addressing the priority needs of the company. The Data Scientist doesn’t work in a vacuum; they have to interact with key business stakeholders on a regular basis. However, Data Science shouldn’t be structured like a traditional applications development team either. The teams is focused on discovery and exploration – not core IT development. Take a look at one of the more popular articles on the topic, “Data Scientist: the sexiest job of the 21st century” by Tom Davenport and DJ Patil http://goo.gl/CmCtv9
Photo courtesy of National Archive via Flickr (Creative Commons license).
I’ve been intrigued with all of the attention that the world of Data Science has received. It seems that every popular business magazine has published several articles and it’s become a mainstream topic at most industry conferences. One of the things that struck me as odd is that there’s a group of folks that actually believe that all of the activities necessary to deliver new business discoveries with data science can be reasonably addressed by finding individuals that have a cornucopia of technical and business skills. One popular belief is that a Data Scientist should be able to address all of the business and technical activities necessary to identify, qualify, prove, and explain a business idea with detailed data.
If you can find individuals that comprehend the peculiarities of source data extraction, have mastered data integration techniques, understand parallel algorithms to process tens of billions of records, have worked with specialized data preparation tools, and can debate your company’s business strategy and priorities – Cool! Hire these folks and chain their leg to the desk as soon as possible.
If you can’t, you might consider building a team that can cover the various roles that are necessary to support a Data Science initiative. There’s a lot more to Data Science than simply processing a pile of data with the latest open source framework. The roles that you should consider include:
Manages the various data repositories that feed data to the analytics effort. This includes understanding the schemas, tracking the data content, and making sure the platforms are maintained. Companies with existing data warehouses, data marts, or reporting systems typically have a group of folks focused on these activities (DBAs, administrators, etc.).
Responsible for developing and implementing tools to gather, move, process, and manage data. In most analytics environments, these activities are handled by the data integration team. In the world of Big Data or Data Science, this isn’t just ETL development for batch files; it also includes processing data streams and handling the cleansing and standardization of numerous structured and unstructured data sources.
Handles the traditional data management or source data stewardship role; the focus is supporting development access and manipulation of data content. This includes tracking the available data sources (internal and external), understanding the location and underlying details of specific attributes, and supporting developers’ code construction efforts.
Responsible for packaging the Data Scientist discoveries into a production ready deliverable. This may include (one or) many components: new data attributes, new algorithms, a new data processing method, or an entirely new end-user tool. The goal is to ensure that the discoveries deliver business value.
The team leader and the individual that excels at analyzing data to help a business gain a competitive edge. They are adept at technical activities and equally qualified to lead a business discussion as to the benefits of a new business strategy or approach. They can tackle all aspects of a problem and often lead the interdisciplinary team to construct an analytics solution.
There’s no shortage of success stories about the amazing data discoveries uncovered by Data Scientists. In many of those companies, the Data Scientist didn’t have an incumbent data warehousing or analytics environment; they couldn’t pick up the phone to call a data architect, there wasn’t any metadata documentation, and their company didn’t have a standard set of data management tools. They were on their own. So, the Data Scientist became “chief cook and bottle washer” for everything that is big data and analytics.
Most companies today have institutionalized data analysis; there are multiple data warehouses, lots of dashboards, and even a query support desk. And while there’s a big difference between desktop reporting and processing social media feedback, much of the “behind the scenes” data management and data integration work is the same. If your company already has an incumbent data and analytics environment, it makes sense to leverage existing methods, practices, and staff skills. Let the Data Scientists focus on identifying the next big idea and the heavy analytics; let the rest of the team deal with all of the other work.