The AI Hype Cycle, Talent Wars, And Internet Comment Threads

Last week, somebody on Hacker News posted a link to a Bloomberg article titled “Sky-High Salaries Are the Weapons in the AI Talent War”. The article itself was good, and predictably discussed how the shortage of AI talent is driving up salaries. But what really stood out for me were the Hacker News comments on this article. While reading it, I silently golf clapped. Here’s a link to the full comment thread.

 

While I’m incredibly bullish on the medium to long term impact of AI, I’ve made no secret about being skeptical of the short term hype surrounding AI. My personal experience – having directly or indirectly witnessed data science efforts in a several dozen companies – has been that a lot of companies aren’t ready for AI and don’t yet have real use cases. But these companies are feeling pressure to hire expensive data scientists. The pressure may come from executives and their board who want a higher valuation, FOMO, a misguided desire to “do data science” without a clear plan, or other motivations. Regardless of where the pressure comes from, plenty of these companies will be disappointed.

Today’s AI hype reminds me a lot of the big data hype from several years ago. A lot of technology was misapplied against nonexistent problems. There were companies spinning up Hadoop clusters to house a few GB of structured, tabular data. A simple relational database would’ve worked just fine. A lot of these companies now have big data graveyards, and are reluctant to invest in new data initiatives.

Similarly, there are companies implementing machine AI without clear use cases. Also never mind the data used in these models may not be useful or good quality. But hey, they’re “doing AI”. Anecdotally, I’m hearing about Math PhD’s hired as data scientists at large tech companies, only to end up writing SQL queries and writing data engineering scripts. This is clearly a misuse of their talents and the company’s resources.

The AI hype creates a hiring binge that dries up the very limited supply of excellent data scientists; and their salaries to skyrocket. Of course, this is great for data scientists like me. But, I predict the lack of real use cases for AI will inevitably have a backlash when companies don’t see success or ROI on these expensive investments.