In this part of “How to let your data science team succeed (or why your data science team will fail)” Part 2 of 2, we will go through some more real life scenarios of the data science challenge.
Company culture – embracing or resisting change
Let’s start with the “internal elements”. Is your organisation an old-school big corporate? Or are you a lean start-up? Needless to say, bigger corporates often find it harder to embrace technologies; whilst start-ups are usually more flexible in technological adaptation and trying out new things.
True story #1
Take an old-school corporate for example: – they were relying on revenue forecast models that had been developed in Excel back in the 90’s. The intricacies of the model were so complex that not one of the current employees knew what all the underlying macros did; there were many product verticals unknown to anyone in living organisational memory; the person who developed it had long gone to greener pastures; and the finance team who had been relying on said model for years (and still counting) was extremely reluctant to change.
In this scenario it is important to have leadership support from the top level (eg the C-levels/directors). I don’t mean by just air-dropping several data scientists into the management ranks, as that is part 1 scenario on how to fail. Your leaders need to influence the wider company to embrace change by identifying barriers for change, and play their part to guide and encourage the organisation as a whole to develop a shared sense of purpose. Publishing early wins is essential, and the leadership team needs to ensure the results are being utilised in day-to-day operations. It’s simple: positive examples and encouragement will work to get your data transformation direction adapted and executed.
Infrastructure and data
Another common problem for data science, as the name suggests, is that data science requires data. Data science without sufficient/quality data is just pure magical voodoo. Data and infrastructure need to go hand-in-hand – you can’t have big data without storage solutions; nor can you have excellent architecture without any data.
True story #2
This brings us to another example – a large corporation relying on traditional on-premise IT infrastructures aiming to match modern cloud-based technologies.
Their data were in silos and sometimes they even had to beg their suppliers to send them their own data in csv files. Their IT was outsourced to an offshore company, and no one from the offshore company had sufficient knowledge to support modern Data Ops, let alone productionising data science models.
In this scenario, it is necessary to establish who’s accountable and who’s responsible for delivering the infrastructure surrounding the core data science development. There needs to be a common platform where the data science team and the infrastructure team can communicate and agree on their requirements. This will make sure data science outputs are not going to be stuck in some form of production bottleneck and waiting for eternity. Sometimes you simply cannot find the necessary skill sets to enable all these to happen, which brings us onto the final point.
Finding the right talent mix – for the right problem
Most data scientists can not write productionised (or even productionisable: lisable code (there, I said it). Conversely, data engineers most of the time are not able to build your ML models. There are those rare people who can do both tasks equally well (and attain unicorn status because of it), but as unicorns they are hard to find.
True story #3
The final story happens in every organisation embarking on a data science journey. How do you find the right talent for the right project? Do you take on permanent staff or contractors? An organisation I once worked at took on two data scientists, hoping they would solve all their data-related problems…. upon finding that they did not have the necessary infrastructure to support data science functions, they hired seven more data scientists, but still no engineering power. In the end, nothing made it into production other than – drum roll – KPI dashboards, because that was literally all they could push into production and KPI insight was all that could be delivered.
This final example touches upon the challenges of company cultural change, inadequate infrastructure, as well as difficulties in identifying the root cause of the problem. A data scientist is not best placed to produce beautiful dashboards. A data scientist is not going to build your data architecture. A data scientist cannot single-handedly tackle all the data challenges you are facing. The right mix of data science and engineering skills, however, can get you to the moon (literally, if that’s where you really want to go and you can afford the fuel).
Key point to take away
If your organisation is embarking on a data science journey, it is important to refine and measure the vision periodically. Remember, it may be better to adopt a “Fail Fast” approach (also known as don’t be like the concorde fallacy), and bring in the cavalry who can help provide the vision with a direction and set realistic goals.
If you are not sure how to correctly identify your problem and needs, or to find the right mix of skills, we are here to help!