In October of last year, The New York Times published a piece commenting on the arms race to sign top AI talent among the world’s tech giants. The biggest players in the space are placing huge bets on AI. In addition, they are offering initial salaries of $300,000 to $500,000+. As a result, the competition to acquire talent is fiercer than ever.
But the challenge to hire top data scientists isn’t just one that businesses in tech face. AI and ML are being applied with growing ubiquity. As a result, businesses across all industries recognize that AI is an essential part of their future. According to the Boston Consulting Group and MIT Sloan Management Review 2017 Study, ‘Reshaping Business with Artificial Intelligence’:
- 84% of businesses believe that AI will help them build or sustain a competitive advantage
- 72% of businesses believe AI will have a significant impact on the products they offer within the next 5 years
- 61% of businesses see developing an AI strategy as an urgent need
Despite this, less than a quarter of businesses surveyed incorporated AI into their processes or product offerings. This can be partly attributed to the relatively nascent adoption of AI and ML. However, the problem also lies in businesses lacking the resources and talent to implement AI at scale. And as long as the arms race for top data scientists continues to grow more competitive, bringing the best talent into an organization will remain an increasing challenge.
So how does a business overcome these challenges, and ensure they’re getting the best data scientists they can find?
The most in-demand data science skill sets
Data scientists play a variety of roles in an organization. These roles range from analysis, to designing algorithms, to deploying algorithms in production. Businesses need enough data scientists to cover all these roles in order to develop an effective data science strategy.
But with talent in short supply and demand greater than ever, hiring data scientists is a major challenge. Two of the key areas where hiring data science talent is particularly difficult are:
- Machine Learning / Algorithm Design – Engineers typically require a strong background in applied math, statistics, and algorithms. The best engineers in this area can develop new models and algorithms unique to each organization. But engineers who lack sufficient mathematical training can be shackled. This is especially so when when they facenew data science problems or poorly performing machine learning algorithms.
- Production Machine Learning Engineers – Working on models in an isolated, non-real world situation is vastly different to deploying models at scale on real-world data. Engineers in this area must have a strong background in scalable big data technology and operations. They need to know how to deploy technology that can process real-time data, build systems with high up-time, and deploy the models in production environments.
Even if you can find and hire data scientists with the requisite qualities to accomplish your organization’s goals, holding onto talent for enough time and consistency to develop the AI systems you need still poses a major challenge. But making those investments early will set your business up for the long term, as the landscape of AI continues to evolve and the role of the data scientist continues to shift.
How the role of the data scientist will change
Over time, an increasing subset of work traditionally done by data-scientists will be augmented by a variety of visualization and automation tools. We already have early versions of technology focused on making progress in areas such as:
- Visualization of large scale data
- Cleaning and normalization of data
- Feature engineering technology
- Model selection and hyper-parameter tuning
A small subset of machine learning problems will eventually be automated using Automated Machine Learning platforms. These platforms will be able to read raw data from data lakes and then successively clean data, engineer features, and pick the best models for specific machine learning problems. The subset of problems which are solvable using automation will increase over time. This is because algorithms will become more sophisticated and systems learn to accommodate an increasing set of problems.
The role of the data scientist will shift with the advent of these automation and visualization tools. Data scientists will increasingly work on more complex problems unique to the organization. These problems require a great deal of organizational knowledge. They also require tuning algorithms to tackle those organizational problems.
This shift will only increase the need for Machine Learning and Algorithm Design engineers capable of understanding and adapting machine learning technology to each business’ unique challenges. And while acquiring and holding onto these data scientists is no simple task, being able to do so holds the key to having a competitive edge, as AI continues to reshape how we do business.
