Per a16z, 2024 is the year when enterprises are getting serious about putting generative AI in production. And as I learned a few months ago via an MIT Sloan publication, it’ll be the majority of enterprises. Indeed, according to their published survey results, only about 5% of companies had any sort of GenAI in production in 2023. That figure was surprisingly low to me, given how much the world talks about LLMs. Nonetheless, it derives that many organizations are just now getting started on their AI transformation. As I did years ago about Business Agility, I want to share a few experiential learnings in case they help others. Also, the sooner I create a contemporary body of written work, the sooner I can train an AI agent to write in my stead.
Preamble
For my/our work experience to make sense, I believe some background is needed. If you disagree, feel free to skip over (link) this rather lengthy preamble.
Another potential bone of contention is my credibility on the subject. That’s in the eye of the beholder, I suppose. So here’s some context so you can assess. Today at JustAnswer, our GenAI-powered chatbots help our customers find professional services experts – the living, breathing kind – via 7M interactions per month. Moreover, we have deployed GenAI just about everywhere in our operations, from online ad management to the aforementioned chatbot infrastructure to a more personalized customer experience. Finally, employees across all departments use AI creatively, we leverage LLMs from all the big vendors, and our corporate strategy has had an AI pillar for over 15 months.
As far as my credentials are concerned, it’s admittedly less quantitative. However, I am the company CTO and the executive sponsor for a group of machine learning specialists that drive broad education for all employees, joint experimentation with all functions, and purpose-built systems and services for others to use. So you know that I’m surrounded by early adopters, that I’ve had a first-row seat to the process, and even, perhaps, a little influence.
Before I describe the steps we took, it might be useful to understand JustAnswer’s impetus for action when the power of LLMs became obvious in late 2022 because your situation might call for a different approach.
By now, the market consensus is that some companies will ride the GenAI wave and benefit from it while others will have that wave crash upon them to their chagrin. If there are companies altogether outside the path of the wave, I don’t know of them. But we can agree that not all organizations are strategically equidistant from it (again, context matters), so the urgency for action may differ.
JustAnswer is pretty close to that wave. See, the company seeks to democratize access to professional services by enabling consumers to have their issues resolved by qualified pros across a wide range of domains like medical, auto mechanic, veterinary, antique appraisal, legal, home improvement, legal, and so forth. JustAnswer broadens that access via online matchmaking and lowers the cost by primarily offering a lightweight chat-based interaction.
Questions. Answers. Chat interface. See where I’m going?
ChatGPT coming online was a very loud confirmation of what our data scientists had been saying: AI is getting pretty good at answering written questions. Given the nature of our product, should we not be paying attention? They had been poking around the OpenAI Playground since inception but hadn’t yet been able to convince the managerial class I represent to dial up our investments in AI. But from that point on, it wasn’t a matter of if we were going to embrace LLMs, but rather how.
Enough with potatoes, here’s the meat
When ChatGPT said “Hello world”, we had low double-digit employees sufficiently familiar with generative AI to understand its potential and fewer with any hands-on experience. I cannot claim to have been part of that group. Rather, at the time, I was with the majority of our 1,000 employees. So if we were going to take this seriously, we were going to have to grow our number of employees skilled in AI by two orders of magnitude, starting with yours truly.
The first thing we did was an internal education campaign on what GenAI is and why it matters. Thought leaders and titled leaders started posting regularly on our internal channels. We wrote about what GenAI is, what it can do, and how it could apply to us. Our goal was to create interest. We then created a few dedicated internal channels where technical and non-technical communities could go deeper into their area of interest. This was important because a publication bar set where a post must be of interest to all employees would be discouraging to authors. A bit like I’m feeling now writing this.
After the “tell”, came the “show”. Our most advanced users hosted demonstrations. We started with the basics, such as how to use ChatGPT, what prompt engineering is, what sort of tasks LLMs can readily do, etc. JustAnswer’s employees are a smart and curious bunch, so these demos triggered demand for more hands-on workshops, which our early experts – many in our ML group – so adroitly setup. We owe these folks a debt of gratitude for bootstrapping our GenAI adoption.
By early 2023, the potential of the LLM wave was clear enough that we declared learning to surf it a corporate priority. That declaration trickled down through the management operating system: goals and objectives, investment choices, rewards and recognition, etc. I believe this executive support was essential in overcoming our internal inertia. After all, our leaders and teams were already busy working on plans they liked and which we’d previously approved. Also, it was now clear that AI was going to be more transformative than the run-of-the-mill adoption of a new tool or business process. So we’d be faced with our own internal customer adoption curve, like any transformation. As I wrote years ago (ref), I believe that real transformations require exec support. Luckily, my colleagues on the exec team saw the same need so we were able to move swiftly.
Of particular importance was the decision we made to dedicate an agile team to creating a generative AI “paved path” for others to use. We did not mandate the usage of this emerging corporate capability, but we estimated that while a proof-of-concept is trivial for someone to create with, say, the OpenAI API, production systems are much more complex, what with all the ilities (ref). I may go into detail another time, but in short, we created an LLM proxy service anyone could use. That proxy service became a major enabler. I don’t always make great decisions, but this one has paid off in spades.
Just as important and not much less time-consuming was sorting out the Legal governance for the utilization of LLMs. At the time, it wasn’t immediately clear whether data sent to the big LLM vendors would be used to train future models. Some companies chose to to be cautious as a result (ref Samsung). Being in very close proximity to the wave, we decided to paddle hard instead so we tasked our legal team and ML leaders to generate guidance for the company. I wish I had been more closely involved earlier on because this turned out to be harder than I anticipated. In hindsight, I should have known that two tribes with very different skillset collaborating to clarify an emerging domain was going to require a little get-to-know-you.
Today, LLM API providers have much clearer terms of service and data agreements. Back then, we had to spelunk into each website, proactively request said ToS, and consult with external counsel to pressure-test our conclusions. R&D teams spent quite a bit of time articulating our use cases in terms that made sense to the legal team so they could provide nuanced risk assessments. Indeed, there are different concerns between a marketer creating online content, an engineering refactoring code, and a chatbot asking customers intake questions. I’m not sure why I had the expectation that I had, but it utlimately took several weeks for us to have an articulation that was both agreed by legal and understandable by employees.
So, short story long, by mid-2023, we had LLMs used by employees to refine their craft and deployed in production in a customer-facing manner to great effect. We were now beginner surfers. End-to-end, or rather, from seeing the wave to standing up on the board in public, it took about six months.