Listen to this Predibase episode on Spotify, Apple, Amazon, and Podcast Addict | Watch on YouTube.

Today, Madrona Partner Vivek Ramaswami hosts Dev Rishi and Travis Addair, co-founders of 2023 IA40 Rising Star, Predibase. Predibase is a platform for developers to easily fine-tune and serve any open-source model. They recently released LoRA Land, a collection of 25 highly-performant open-source fine-tuned models, and launched open-source LoRA eXchange, or LoRAX, which allows users to pack hundreds of task-specific models into a single GPU.

In this episode, Dev and Travis share their perspective on what it takes to commercialize open-source projects, how to build a product your customers actually want, and how to differentiate in a very noisy AI infrastructure market. This is a must-listen for anyone building an AI.

This transcript was automatically generated and edited for clarity.

Vivek: To kick us off, maybe we can just start off by telling us a little bit about the founding story at Predibase. Dev, you were a PM at Google in the Bay for about five years. Travis, you were a senior software engineer at Uber in Seattle for about four years. How did you meet and co-found Predibase in 2021?

Travis: The company originally started out of Uber. Our co-founder, Piero, was working at the time as an AI researcher for Uber’s AI team on a project called Ludwig. He and I worked together. He came to me in March of 2020 saying that he thought that there was an interesting opportunity to productize a lot of the ideas from Ludwig around making deep learning accessible and thinking about bringing that to other industries, other organizations, other personas beyond the data scientists, but folks like analysts and engineers as well.

It started getting a lot more serious in the summer of 2020. We started saying, “We definitely need someone who’s not completely technical and engineer. Let’s try to find someone who knows how to build good products and can help think about some of the go-to-market issues, and that’s how we ultimately came to meet Dev. Every other person that I had talked to up to that point was very bidirectional conversation like, “What are you guys doing? What do you think the opportunity is?” It was a lot of selling on our side. Dev came in with a presentation like, “Here’s how I think that you can turn this into a business.” I was like, “Okay, this guy’s pretty serious.”

Dev: On my side, I had spent time previous to Predibase as PM at Google, and worked on a number of different teams. I saw how Google did machine learning internally. For a while I was jaded on this idea of making deep learning and machine learning a lot easier. I’d seen it and we had tried to do it at Google, and it hadn’t really gone as well as we wanted. I originally got in touch with Piero and Travis through a mutual contact because I got introduced as a skeptic on the space. I think the thing that I had said was, “I’ve seen a lot of people die on this hill of trying to make machine learning a lot more accessible.”

I remember the first meeting I had with Piero and Travis, they didn’t understand what I think the vision directly was, but then I tried the projects that they had tried to be open source and they walked me through a presentation of how they were thinking about it. That’s when it clicked and I knew that there was something here that could be more differentiated approach. We all got to know each other over the summer of 2020, and we officially started the company at the beginning of 2021.

Vivek: It’s funny how different the deep learning and machine learning space was three or four years ago compared to where we are today. It’s almost hard for people to think, “Oh, people were jaded about it when you see all of the optimism today.” I’m curious: what was the aha moment when things really clicked? You mentioned, Travis, that the original vision was around Ludwig and these open-source projects. Often, there’s a lot of difficulty in going from a cool open-source project to Is this a business? So what was that aha moment, when did you start to see things really click?

Travis: What convinced me about the idea of making deep learning accessible was an extrapolation of what I perceived to be a trend happening in the industry, which was on two different fronts. One was the consolidation of data infrastructure from data swamps and data lakes and unstructured towards more canonical systems of record for data. Data was getting better and getting more organized. The other was on the modeling side, that model architectures were consolidating as well towards transformer-based architectures. There were only a handful of models that people were using in production and fine-tuning as opposed to having to build novel model architectures from scratch with low-level tools like PyTorch.

My bet was that eventually, these things were going to converge, and the idea of training or fine-tuning models on proprietary data would be like a one-click type operation in a lot of cases where you’re not having to think very critically about how many layers the model has or having to do a lot of intense manual data engineering. That was going to be much more about the business use case, the kind of level of the problem that someone like an analyst or someone like an engineer would work at as opposed to a data scientist.

I definitely believe this has been proven out to some extent over time. Of course, I’m sure we’ll get into it, but I think with large language models, it happened in a bit of more of a lightning flash as opposed to a slow burn. But definitely I think that that trend is continuing.

Vivek: It’s interesting, Predibase was founded after you all started talking in the summer of 2020 and officially formed the company. When we think about the start of this most recent AI moment or AI era, it’s really when ChatGPT was released at the end of 2022, and everyone started talking about large language models and AI. It’s so interesting talking to all these companies that were founded before that moment. How has the original vision changed or not changed since ChatGPT was released in this pre-LLM era? What do you think has changed and hasn’t changed at Predibase in terms of how you’re thinking about the future and the vision for the company?

Dev: It’s funny because I think a cheeky answer to this question could be that our vision hasn’t necessarily changed that much, but our tactics have changed entirely. Our vision initially was as a platform for anyone to be able to start to build deep learning models. We initially started with data analysts as an audience in 2021. We had an entire interface built around a SQL-like engine that allowed them to use deep learning models immediately. That’s where the name Predibase came from. Predibase is short for predictive database, and the idea was let’s take the same types of constructs that we’ve brought for database systems and bring it towards deep learning and machine learning.

At the end of that year, we found that analysts weren’t our niche. The people who wanted to use us were more like developers, engineers, the types of people that were along the lines of, “Hey, just give me some good technical documentation and I’ll figure this out.” That’s where we started to shift from an audience standpoint. Our vision was, how do we make deep learning accessible toward this type of audience? When we started pitching deep learning in 2021 to a lot of organizations, we came up with the value propositions of deep learning as, “Hey, you can start to work with unstructured data like text, and images. You get better performance at scale. And oh yeah, you can use some of them pre-trained, and that way you don’t need as much data to be able to start to get your initial results.”

The biggest change that happened is that last value proposition, which was third on our list, has become the most important thing that people have started to care about. Which, if I think about what was popular in our platform in 2022, people’s eyes sort of glazed over when we said deep learning is cool, but what they liked was we had this dropdown menu where you could select different text models, and one of them was a heavyweight model called BERT. You could start to use it pre-trained. They didn’t need as much data to be able to get started. They loved the idea that they could actually maybe fine-tune it inside of that platform. At the time, it was just one feature among many of the things that we had done on the platform, among many other value props.

In 2023, when large language models came out in a large way, we started to think about what we wanted our platform to be. One of our very first takes and maybe one of my very first takes specifically was LLMs are another dropdown item in the menu. You had BERT and all these other menus of deep learning models, and now we’re going to add Llama as an example. We needed to recognize that the market had changed how it thought about machine learning. It was no longer thinking about training models first and getting results after it. It was thinking about prompting, fine-tuning, and then re-prompting a fine-tuned model overall. Our tactics significantly shifted. We considered a product pivot that we did in 2023 to be able to actually better support large language models. Funnily enough, it’s still in service of the vision of how do we make deep learning accessible for developers.

Vivek: It’s almost as if the vision has stayed the same, but the market has come to you in some ways like, “Hey, we were talking about deep learning and machine learning before it was “cool” in the current context.” Large language models have opened up a whole new sphere of who you can market to.

Dev: We definitely got lucky there. We had to meet the market halfway. We had to make sure that we also were responsive and not trying to do or meet the market using our old form factors, or our old tactics. The market did come to us in essentially figuring out, “Hey, there are three value propositions you mentioned. One or two of these really matter. How do you center your offering around that?” That’s been one of the most helpful things for a business. As a startup, I find one of the biggest challenges is getting people to care. How do you get anyone from another startup organization to spend the 45 minutes with you or a large enterprise? One of the nice things that’s happened over the last year and a half is we no longer have to explain why deep learning matters. We frame it as LLMs and being able to fine-tune those.

Vivek: You both talk about the big thing for startups, and a lot of the founders who listen to this podcast will attest to the same, is you have this great idea in your head, and you see the tech maybe before anybody else does, but then there’s a question of, well, why should we care? How do you go from figuring out, “This is a cool open-source project,” to “This is something we can commercialize, and people and businesses are willing to pay for this.”?

Dev: A lot of open-source projects are very popular in GitHub, and I do think a subset of those are probably best as open-source software. They’re a framework that makes something easy but doesn’t necessarily need the full depth of a commercial or enterprise already product for it. No one knows more about these types of challenges of actually taking some of these open-source frameworks and running of the infrastructure than Travis. He’s been working directly on it both for Predibase as well as with users who have tried to use this independently. Travis can talk about the challenges we solve, translating open-source frameworks without a commercial offering, and why we thought there was a real commercial business to be built around these frameworks.

Travis: When it comes to open source and particularly open core models, I think the easiest argument to make is that at Uber we had a team of 50 to 100 engineers that were working on building infrastructure for training and serving models. The cost of that is quite significant, even for a company like Uber. For companies that don’t consider this part of their core business, maybe they consider it core infrastructure but it’s not differentiated IP for them, you could invest in building an entire team around it or you could just pay a company like Predibase to help solve those challenges. With our most recent project, LoRAX for example, there’s a good open-source piece of software that can be productized and productionized and used in situations where you need high availability, low latency, and all those sorts of things. That’s sort of the layer on top. We have internal software that’s been running on top of Kubernetes and running across multiple data centers to optimize the availability, latency, and throughput of the system that goes above what’s in the open source.

That’s inevitable when you’re talking about something that’s going to be used day-in and day-out thousands of times a day, that what starts off seeming like long-tail issues like, oh, this request failed or this service was down for some period of time, become mission-critical at certain points. That’s, where there’s a good opportunity to appeal to organizations that need that. There’s a good synergy, I think, where they need those particular levels of service, and we’re able to offer it, and that’s something that they’re willing to pay for at that point.

Vivek: It sounds like, unlike many other open-source projects where you start the open-source project and then you say, “Hey, let’s see if people are willing to pay or not,” this is almost where at the very start of it you knew that there is a willingness to pay given your time at Uber and having seen this scale. Let’s start with this open-source project and start with this product that’s out there, get people to try it, and there’s definitely a willingness to pay. So it’s a different angle. We see many open-source projects that are out there, you wait for a lot of people to use it and say, “Hey, are people willing to pay or not?” and then it’s a different debate you have at that point.

Travis: Dev actually has a good analogy about the front of the kitchen, back of the kitchen when it comes to this sort of thing. I think that serving definitely is this very front-facing thing where you have to get every detail right, and those minor differences make a huge difference in terms of the overall value of what you’re offering. So yeah, I’m not sure, Dev, if you wanted to maybe speak more to that.

Dev: I have two analogies that I’m going to throw out. The first analogy I think about with open-source projects is there a commercially viable business around this? The way I think about it is for us, Ludwig and LoRAX are sort of the engine, and what we’re trying to do is sell the car. There are some people, maybe advanced auto manufacturers, who just want an engine and they want to put it in their tractor or some other kind of setup. Most people want to be able to buy the fully functioning car, something that’s going to allow them to unlock the doors and have a steering wheel and other things along those lines, which in our world is the ability to connect to enterprise data sources, deploy into virtual private cloud, give you observability and deployments that you don’t necessarily get if you’re just to be able to run the open-source projects directly, and finally connect that engine to a gas line, which I think in our world will again be the underlying GPUs, cloud infrastructure that this is going to run on.

The second analogy is for how we think about the product, and then the other piece you always have to figure out is how much the core problem that the open-source product is solving really matters to that end customer and what the visibility around it. That’s where I think about there’s some things that can be done in the back of the house as a kitchen, where maybe someone has an internal pipeline for doing something. It doesn’t need to be pretty, doesn’t need to be production-ready. It could be so much better if they had this commercial product that was built on top of the open source, but it’s not necessarily mission-critical, or they’re not going to lose customers and users because it doesn’t work flawlessly. Think about these as especially those internal cell pipelines.

I think about the front of the house, which to me is taking, for example, fine-tuned models and being able to serve them very well, things that are going to go in front of customers and user traffic. This is the part of the restaurant that you want to make sure services folks really, really well. So, I’ll need to figure out how to combine the car analogy and the restaurant analogy, but to me, the car analogy is how we figure out what’s the commercial viability around the open-source projects, and the restaurant analogy is a little bit of how you think about if an open-source project is going to be important enough to be able to justify some of that commercial viability.

Vivek: I love it. Don’t be surprised if we steal both of those analogies for some of our own companies because it’s really important distinction, like selling the car versus the engine, and the front of the house versus back house. At the end of the day, all of these things roll up to what’s most important for customers.

One of the things that I love when we talk about the front of the house or even what people see is on your website you have a great tagline, which is, “Bigger isn’t always better.” With the explosion of GPT and everything we’ve been hearing from them, we’ve been hearing about these models with billions of parameters. For a while, it was a bigger is better, and we got to create the best model and how many parameters is GPT-5 going to be. In your view, why is bigger not always better? Specifically related to models and the customers you serve, where do you find that bigger is not always better?

Dev: My favorite customer quote is, “Generalized intelligence is great, but I don’t need my point-of-sale system to recite French poetry.” I think that customers have this intuition that bigger isn’t better, and we don’t always even have to convince them that much. They sort of hate this idea that they’re paying for this general-purpose, high-capacity model to be able to do something rote. I want it to be able to classify my calls and tell me did I request a follow-up or not. It’s a very common type of task that people might be able to do using GPT-4. Today, they have a model that can do everything from that to French poetry to write code. There’s this intuition that when you’re using a large model like that you’re paying for all that excess capacity, both in terms of the literal dollars but also in the latency, reliability, and deferred ownership.

When I talk to customers, I think they’re very enamored with this idea of smaller task-specific models that they can own and deploy that are right-sized toward their task. What they need is a good solution for something very, very narrow. Where I think the trick is that customers have is, well, can those small models do as well as the big models? It’s very fair if you’ve played around with some of these open-source models, especially some of the base model versions, you have that intuition that they don’t do as well as the big models as soon as you start to prompt them. That’s where we’ve spent a lot of our time investing in research to figure out what actually allows a small model to do and punch above its weight and be as good as a large model.

To us, what we’ve just unlocked might not be a massive secret, but it’s been around data and fine-tuning. What we found is, that if you fine-tune a much smaller model, a seven billion parameter model, a two billion parameter model, probably an order or two orders of magnitude smaller than some of the bigger models people are using, you can get a parody with or even outperform the larger models, and you can do it in a lot more cost-effective way and also be able to do it a lot faster so you don’t have to wait for that spinner that you often see with some of these larger models.

Travis: A big aspect of this that every organization should think about is the type of tasks that they’re trying to do with the model. If you’re primarily interested in very open-ended tasks where you do want to be able to say, like have a chat application where you want to be able to ask anything from generate French poetry to solve this math problem to whatever, you do need a lot of capacity. That’s why ChatGPT is as successful as it is, is because when a user comes in and uses it, they don’t know a priori what type of question the user is going to have in mind. You do need something very general purpose.

When you’re productionize something behind an API, it’s just like an endpoint, you’re calling it, “Classify this document,” and you’re going to call that over and over and over again thousands of times, you don’t need all those extra parameters just at the baseline. That point depends on how complex your task is the capacity of the model that you need. The less capacity you need, the smaller the model you can use, and the lower the latency. It goes on a task-by-task basis that people should be evaluating these sorts of things.

Vivek: I feel like we are now in the moment of time where we are seeing that the balance swing back to, I might not need this really, really massive model with a trillion parameters and all of that. I need something that works for me, that works for my use case. Dev, you mentioned customers there and what your customers have been saying to you. Take us to the first customer, how did your first customer come through the door? How did you land them?

Dev: I wish there was a very repeatable lesson for founders here, but sometimes your first customer is a little bit of luck mixed with just a little bit of internal network and elbow grease. I remember we started the company in March 2021. I had no idea how we were supposed to get our first customer. What people told us is the common advice, and I think this is correct, is your first few customers are probably in your network. Looking initially at my network, I didn’t know really where to start digging. We ended up getting in-bounded from a large healthcare company based here in the US. They were curious just to know, they had seen Ludwig out there. They weren’t even active users at the time, but they had seen Ludwig and they’d seen it solve a very specific use case that Uber had published a case study around, which was customer service automation. They wanted to know if there was something that could be applied with Ludwig inside their organization.

I’ll never forget the very first customer meeting that we had was with this organization. We started in March, this customer meeting was in April. It was an hour-long meeting where we walked them through what Ludwig had been for a few minutes, but also what our vision was and what we were building with Predibase. The meeting ended with, “If you guys have a design partner signup sheet, just put our name on that list.” That was the end of the very first customer meeting that we had. It came in because of that open-source credibility. I walked away being like, “Are they all that easy?” They’re not just as a very quick recap, but the first one for us did come in a way through network, but really from organic open source inbound that came through.

From there what we’ve found is helpful is the repeatable lesson of content that attacked a use case that somebody cared about and that allowed them to come to us. That’s something we still see as a pretty effective channel now as we’re landing our next sets of customers as well.

Vivek: See, folks, it’s just that easy, just start the company and a month later someone’s going to ping you. Having that channel, as you mentioned, this is one of the nice great benefits of open source is people can start playing with it. Someone may find intrinsically there’s a lot of value and say, “Hey, how do we go from where we are today to doing even more with this?”

Zooming out a little bit, I would say to the outside observer, at least today compared to a few years ago, the AI infrastructure space has become very, very crowded. It seems like there are a lot of infra companies building at the inference layer, at the training layer, doing things around fine-tuning. I would say often to the outside observer and even sometimes to the inside observer, it’s hard to tell what’s real, what’s working, what’s the difference, do we need all these products.

In some ways, it’s really healthy because it gives people a lot of options, and I think when we’re early in the AI era as we are right now, you need a lot of these options, and there’s a lot of space to build. How do you both think about it? One, there’s a day-to-day of just maybe there’s hand-to-hand combat against some product, some more than others maybe, and then there’s just the long term of how do we resonate and stay above the noise and build for the long term.

Dev: I think this market is extremely competitive and that there is a lot of noise that gets introduced into the system. In terms of staying above the noise, the only way that we have found to be effective to stand out as an organization, especially when you don’t have that hour with a customer, you need to go ahead and build a brand where people are just going to look at you for maybe a few seconds or a minute and make an assessment of, “Is this worth my time?” The only way that we’ve seen work is you have to do work that advances the ecosystem yourself.

What I’m saying is I think you need to do something that can be a narrow slice but somewhat novel or a differentiated take. There are two ways that we’ve thought about doing this. The first is people have always liked this idea, that small task-specific models that are fine-tuned will be able to dominate these larger models. I spoke with a customer who said something that stuck with me; he said, “I want to believe that these small task-specific models actually will be the way my organization will go, and I want to use open source, but I just don’t know if it actually works.”

The world out there today is a lot of anecdotal experiments and memes on Twitter and others. One of the first things we did was we started to benchmark data and put out results in our benchmark. We put out a launch in February called LoRA Land, a mix of La La Land as well as the play on LoRA fine-tuning, which is how we did the process, where we took 29 datasets and fine-tuned Mistral-7B against those datasets. We wanted to compare initially to see how much fine-tuning helps against the base model. What we actually found was that fine-tuned Mistral-7B will be at parity with or actually, in many cases, outperform GPT-4. When we do some prompt engineering and try and find prompts that work the best for both of them, it’ll outperform that one out of the box.

That became a moment where we went semi-viral. We were at the top of Reddit for a little while. We had a partnership with Hugging Face and Mistral to go ahead and re-share it; Yann at Meta also, I think, re-shared this. It was a way for us just to start to put some data in the industry. We also released all these models as open source. We even built a little interactive playground where people could play around with these models directly firsthand and start to see what actually the model performances would like.

From there, we’ve scaled this out 10 times. We’ve not only now trained 27 models, but we’ve trained over 270, because we stopped just benchmarking Mistral. People would say, “What about Llama 2? What about Microsoft’s Phi?” We’ve added a number of different models, Google’s Gemma, to this entire list, and we’ve started to just build out our own internal benchmarking to understand what is the fine-tuning leaderboard. We’ve put this content out there, we’ve put these models out there, and there’s going to be more on that very soon. That was one way that we thought about advancing the ecosystem.

The second way, I would say, is honestly building novel frameworks that didn’t exist before. The best example of this is LoRAX. I’ll let Travis speak towards LoRAX as the lead author and creator of the framework, but one of the things that made it very popular was we attacked a problem in a way that no one else had been thinking about, and that really helped us cut above the noise.

Travis: To Dev’s point, attacking a narrow slice of the market is the only way that I found to be able to stay above the noise. The reality is that we talked about how pre-LLMs so much of our focus was on getting people to care, even understanding what the value proposition was, and now everyone cares. Therefore, there’s tons of people in the market attention and many of them are much better capitalized than we are. We’re talking about companies that have hundreds, thousands of employees, in some cases, working on this stuff.

The challenge, I think, was definitely on a product and engineering side to think about ways that we could attack something that, while they were technically capable of doing it, we could do better than them just by sheer focus and execution. We saw an opportunity with this multi-LoRA inference in the second half of last year. It was definitely on people’s radars; there were some early blog posts about it and some research that was happening at institutions like the University of Washington and UC Berkeley, but no one had productized something in this space. We launched LoRAX in November of 2023 and really tried to make it clear that this was a paradigm shift for organizations that instead of thinking about productionizing one general purpose model, you could productionize hundreds or thousands of very narrow task-specific models but solve the essential question, which was, how do you do that at scale in a way that doesn’t break the bank? The previous conventional wisdom was every model is a GPU at a bare minimum. If you have hundreds of models and hundreds of GPUs then you’re paying tens and thousands of dollars per month.

Breaking down that conventional wisdom was the first way that we saw to attack this problem, the goal, of course, being to establish ourselves as a thought leader in that particular space of building these very task-specific models in a way that’s cost-effective and scalable. LoRA Land was a way of building on top of that, saying, “Now that we have this foundational layer with LoRAX, here’s what you can do with it.” I think that that demo of being able to swap between all these different adapters that were better than GPT-4 at the specific task that they’re doing and do it with sub-second latency, I think that started to prove to people that there’s actually something real here. Not to diminish research, but it wasn’t just research, it’s something that you can be using in your organization today.

Vivek: I love reading the Predibase benchmarking reports and seeing how these various models do. There’s almost a sense of fun every time a model comes out, “How’s it going to do? How well does it perform relative to all these benchmarks and relative to all these other open-source models out there?” And because you guys are so close to this and have a great perspective, right now Llama 3 launched. Meta, they’re crushing the open-source model game right now, and obviously, they’re spending a lot of money, resources, and time behind this. It seems to be initially resonating really well. I am curious: how do you see the open-source model world playing out over time? Is it feeling like we’re going to have a handful of providers of large open-source models, Llama, Mistral, Google, or do you think that there’s going to be a world where we’re going to see a long tail of developers and we’re going to see many different types of open-source models and providers of open-source models?

Travis: My opinion on this is that it will break down a little bit by model class. I think that these larger foundational models, particularly the ones that people are open sourcing for general applicability as opposed to building internally on proprietary data to be IP in some way, that is a little bit constrained by the resources of these larger organizations like that. It’s not something that’s generally accessible to a small group of enthusiasts or something like that today. That might change in the future, but right now, the two big barriers are you need lots and lots of compute, and you need lots and lots of data, and both of those things are difficult to come by.

I do think that, at least for the foreseeable future, there’s going to be a requirement to lean onto some of these larger organizations like the Metas of the world to provide those foundational models. Where I do have a lot of optimism on the ability of less well-capitalized, the GPU-poor, to be able to make inroads is definitely on the fine-tuning side. I think that as fine-tuning matures from the research point of view and from the data efficiency point of view, there is an opportunity to create much better-tailored models for specific tasks that do have general applicability that can potentially be something that is valuable to lots of organizations beyond just the individual that made it.

Once we start seeing that become true, there’s a whole new space of creators similar to how content creators create art or videos or music or whatever, being able to create fine-tunes that attack very specific problems and have that be something that people consume at scale. A very interesting opportunity that I believe is coming on the horizon. We’ve seen it to some extent in the computer vision space with diffusion models where diffusion LoRAs for style transfer are starting to become mainstream, and communities around finding different LoRAs that help adjust the way that the models generate images. I definitely think that that moment is coming for large language models as well, where this sort of work moves beyond being restricted to individual organizations that have lots of data to something that can be even more open and transparent, and community-driven.

Vivek: Well, this leads me to my next question, which is for both of you, what do you think is over-hyped and under-hyped in AI today?

Dev: Over-hyped today is chatbots. A lot of organizations started to see value in GenAI actually post GPT-3. The very first thought that organizations had was, “I need ChatGPT for my enterprise.” We were talking to some companies about a year ago and they were like, “I need ChatGPT for my enterprise.” I was like, “Great, what does that mean to you?” They said, “Don’t know. I need to be able to ask the same ChatGPT-style questions but over my internal corpus.”

A lot of the early AI use cases have looked at how to build a chatbot that I can start to ask questions over documents. And one of the main reasons for that is because the interface that went viral was ChatGPT. It was the ability to do this in a consumer setting. The way I think about GenAI models is you essentially have an unlimited army of high school-trained humans essentially that can do different workflows. If you had this kind of unlimited army of knowledge workers, is the most interesting thing you’d really apply them to just better Q&A and better chat? I struggle to think that’s the case. Instead, where I think a lot of the case is going to be in these automation workflows. We’ve used the back-of-kitchen analogy, but there are also the back-of-office tasks, ones that are repetitive and mundane; how do I go ahead and automate document processing? How do I need to go ahead and automate being able to reply to these items? Now, we’ve started to see this become more of a thing.

That’s where a lot of the future for AI is going to go. The over-hyped sentiment is all of these organizations that are saying, “I want ChatGPT for my enterprise,” probably want to start thinking a lot more about, “How do I consider the fact that I have access toward this large essential talent pool that has general purpose knowledge and then can be fine-tuned to do a particular task very well?” I think about it like a college specialization. I can take this high school level agent and give it my college specialization in how Predibase does customer support, for example, and put it to work. That’s the biggest delta that I see from what might be hyped in the market today and where I think a lot of the production workloads are going to be going over the next 12 to 24 months.

Travis: I liken it too saying that it’s the boring AI that’s really under-hyped right now, but that’s where most of the value’s going to come from. I think that in any hype cycle, there’s this very overly optimistic view that we’re going to get 1,000X productivity improvements because we’re going to replace every knowledge worker with an AI system of some sort. Already we’re starting to see the reality unfolding is that, oh, it’s not that easy. It’s never that easy by nature of these things; the 80/20 law — getting the little details right ends up being where the majority of effort is spent, but those things matter.

We’re still quite a way from generalized intelligence and chat interfaces being able to do everything, like replace coders, but certainly, I think that it’s very real that we can get material productivity improvements and efficiency improvements on the order of 20% here, 50% there on very specific parts of the business. It’s going to be through these very narrow applications, to Dev’s point of saying, “We have a system here that requires humans to manually read all these documents. What if we can automate that into something that just turns it into a JSON or turns it into a SQL table for them, and they can just run a few quick sanity checks on it and then send it downstream to the next system?” Those are the sorts of things I can see having a very meaningful impact on the bottom line of businesses, and those are the things that are actually attainable.

Vivek: That’s part of the fun of the hype cycle. Now the initial euphoria has worn off, what are the really interesting things to build from here, right? Let’s get in the nitty-gritty and figure out something that may not have been just the initial like, okay, let’s go build this chatbot on top of GPT. There’s so much more you can do with all of this. Guys, let’s end with this, you both came from iconic companies, as well as Piero, and you obviously have seen a lot of really interesting things and been around some great people, and you’re all first-time founders. If you had to give one tip to a first-time founder, what do you think it would be? And maybe Travis, we’ll start with you, and Dev, we’ll end with you.

Travis: I think the biggest learning for me is you’re always a bit too ambitious when you start out with what you think you can do as an individual or as an organization. Oftentimes, particularly if you spent most of your time being an individual contributor, you have this idea that “If I’m a good engineer and then we hire 10 good engineers, then we’ll be 10X more productive, and we can do all these amazing things.” The reality of doing something and then doing something well enough that people want to buy it and rely on it in production every day is quite a big gap. Definitely being very narrow in terms of the type of problem that you want to tackle early on and say, “Let’s do something very highly specific that maybe doesn’t have a very big TAM in and of itself, but get that working perfectly and then start to think about where we go from there,” that’s definitely been the biggest learning for me I’d say.

Dev: One tip I would say is to be wary anytime someone suggests that you should pick something that’s strategically important to your business, like pick your go-to-market motion or pick what you want your differentiator to be. A real risk that happens toward first-time founders is you sit on a couch, and you’re like, “Hey, what can we do that would be really interesting?” And that’s a really interesting trap, I think, to be able to fall into that doesn’t essentially take in the customer lens of what customers actually care about. I think a lot of first-time founders are very smart. They worked at iconic tech companies, and they’re like, “I saw this happen at, let’s say, a Google, or I saw this happen at an Uber, and so the right way for the future is X, Y, Z.”

That’s a really good starting point, but that needs to be baked in what you’re hearing directly from customers 100% of the time. It’s very easy to essentially pick something that you think would be interesting, cool and differentiated that customers don’t care about. The reality is the thing that you’re suggesting might be the right idea, but it’s a different framing, it’s a different form factor that you need, and you won’t know whether or not you’re right until somebody is willing to purchase an invoice and send you money for it. Make sure you hold that as your primary objective function.

The last bit of advice I liked a lot was everyone who’s done startups has emphasized the importance of velocity. It’s very easy to mix up velocity with, “I need to go ahead and pull 16-hour days and be building a lot of code.” To me, velocity is building highly iteratively. How do you get feedback as soon as possible? The easiest way to do that is what Travis’s point is, is cutting scope. One of my favorite bits of advice that I’ve gotten, which is a bit controversial, is nothing takes longer than a week. The reason that I’ve liked that bit of advice is because I think it forces you to think, “How am I going to take whatever that I’m working on building this week and make sure that I understand at the end of the week is it actually worth doing, is it something that’s putting me in the right direction, is it delivering customer value?”

Both are about baking it into feedback and listening to customers, but understanding that you want to be able to optimize for that feedback cycle. That’s the only way that you’ll probably get to where you’re going.

Vivek: Great advice from both of you. And it’s super exciting to see everything going on at Predibase, at least from the outside. I’m sure it’s even 10 times more exciting and incredible from the inside. Congrats to both of you on all the momentum and really excited to see where things go. Thanks, Dev and Travis.

Listen on Spotify, Apple, Amazon, and Podcast Addict | Watch on YouTube.

Today, Madrona Managing Director Matt McIlwain hosts David Shim, founder and CEO of Read AI. These two have known each other for over two decades, having worked together at Farecast and then Placed, which David founded in 2011. They came together again in 2021, when David started Read AI, which has raised $32 million to date, including its $21 million round led by Goodwater Capital in 2024.

Read AI is in the business of productivity AI. They deliver AI solutions that make meetings, emails, and messages more valuable to teams with AI-generated summaries, transcripts, and highlights. In this episode, David and Matt examine how leveraging emerging technologies, getting engagement from multiple members of your ecosystem in the early days, and learning your way into a business model have been three themes across David’s startup journeys. They explore the challenges and success of implementing AI in various categories, the benefits of using AI model cocktails for more accurate and intelligent applications, and where they see the opportunity for founders well beyond productivity AI. It’s a great discussion you won’t want to miss.

This transcript was automatically generated and edited for clarity.

Matt: I’m just delighted to be back again with David Shim. Welcome back, David.

David: Excited to be here. It’s been a while.

Matt: It’s been a while, and you’ve been up to some good stuff. Let’s talk about Read AI. Why don’t you take us to the founding and the genesis?

The Genesis of Read AI

David: I was CEO of Foursquare for about two years, and I left the company. During that time when I was leaving, I was in a lot of meetings where I wasn’t making a lot of decisions at the end because I wanted the team to make those decisions because they were going to inherit that for the next 12 months. I took a step back, and I just started to watch and saw, hey, there are a lot of people in this meeting. What are they doing — and half of them are camera off and on mute. I made this term up called “Ghost Mode,” where there’s no sound, there’s no video — you’re there, but you’re not actually there. I started to notice that as I left Foursquare and took a couple of months off, bummed around different beaches in Mexico. I started to have a lot of video conferencing meetings, and I started to think about this problem again to say — this is not a good use of time to have this many people in a meeting.

A lot of the time, we know within the first two or three minutes if this is going to be a valuable meeting for me or not. I started to say, can I notice people who are and aren’t paying attention when I’m bored? I looked at one person’s camera and realized they had glasses on and the colors in the lenses looked very similar to what I had on my screen — and it was ESPN. So I looked closer to the glass reflection and they were on ESPN. So I was like, there’s got to be a model or machine learning model that you’re able to go in and say, can I identify when someone is paying attention using visual cues versus just audio cues? And then, if I combine those two things together, is that something that’s differentiated in market? And what I found out there was no one was doing it.

There were people doing it for different use cases, but not for video conferencing. That was a really interesting thing to dive into. Then I reached out to someone that I know who heads up Zoom, specifically Eric Yuan. I said, “Hey Eric, real quick question.” We know each other from emails every once in a while, but I was like, “Hey, I’m really interested in this concept of engagement and video conferencing meetings. Is this something that you’re working on? And if it is, is this something that I should be thinking about as well?” And his response back was, “We thought about it before Covid, it was one of the features that we were going to invest in, but with Covid, priorities changed. I think you should absolutely go into this category and into this space.” That checked the box to say, “Okay, now the platform is bought into saying this is something that is valuable, and this is something potentially that they might support.”

Matt: There are a couple of things that, if we grounded in Placed, will apply to Read AI. Let’s start with this. What were the emerging technologies at that time that it wasn’t obvious how they were going to be useful in terms of building a next generation company? How did you approach those emerging technologies?

Leveraging Emerging Technologies: The Journey of Placed and Read AI

David: When we started Placed in 2011, smartphones were just starting to get into the mainstream. iPhone 2 had just released, Android hadn’t been released yet, or the dev version of Android had been released. People didn’t know what exactly to do other than, hey, there’s Fruit Ninja, there’s games you can play, there’s calculators, there’s flashlights, and those are all great, but that was version one where people saw some novelty with the applications that were available, but people didn’t know what to do next with that.

Where I started with Placed was on the thesis of can you actually get some unique data out of the smartphone that you couldn’t get out of a computer. Previously. It was this concept of could you measure the physical world in the same way that you do the digital world, and is that smartphone going to be that persistent cookie. For the longest time it wasn’t the case. I had engineers going to the iPhone and the Android conferences and say, “Can you get GPS-level location data in background with software?” The answer for a while was, “No, that isn’t possible.” It was only around 2011 that all those things fell into place where that was possible to do.

Matt: You had the mobile phone that was becoming increasingly common. You had location services and GPS data. I remember one of the challenges early on was being able to ping the app, the Placed app that you had often enough that you could get that data without burning up the battery life of the phone. That was a fun adventure, huh?

David: A hundred percent. I had my cousin, I think he was 19 or 20 at the time, use 12 different phones with different models that we had bought for battery life optimization, walk around downtown Seattle, go to different places inside and outside so that we could actually infer what is the best model to last throughout the day where the user isn’t impacted, and also get enough fidelity or context to actually infer did they go into the store or did they walk by the store.

Matt: Let’s talk about Read AI. There were some interesting technological and even some societal “why nows” there. This is early but in the Covid-era. Video conferencing exploded. The cloud has come a long way — applied machine learning and all the different kinds of models. Talk us through some of the technologies that enabled Read AI that were, at a minimum, super-powered versions of things that you were using a decade before to start Placed and maybe some others too.

David: I think the bottoms-up approach to video conferencing was something that we hadn’t seen until Covid came along, and that accelerated where you’ll see all the research reports now that says five years of adoption was pulled up within the first two or three quarters of Covid because everybody went remote. You started to see that people were using the best solution available to them. It wasn’t a top-down approach of you have to use this solution like BlueJeans. A lot of us used to have the equipment for BlueJeans and video conferencing. Now it was you’re remote, what is the easiest solution to use? It was Zoom. Then you saw Google Meet come in, and Microsoft Teams come out.

Now you saw this bottoms-up approach where people were adopting video conferencing as the default where you’re interacting with someone. You don’t go in and say, “Hey, I’m going to go fly out to see you.” Now the default is, “Hey, let’s set up a call.” And the call, if you say, “Let’s set up a call,” It’s not a conference call number. I can’t remember the last time I had a conference call. It always defaults to video conferencing, but that was brought up five years ago. That brought up the demand in market where people were used to going in and saying, “I’ve now chosen a platform. I am able to use this platform on my own for essentially every single meeting that I have.”

That’s very much like the iPhone and the Android devices that came out where, at first, people were like, okay, this is kind of interesting. Then the app started to catch on, and people started to implement them, and the businesses weren’t ready; the enterprises weren’t ready for the smartphone. You started seeing people install apps, they connected their email, they connected their calendar and they didn’t have policies in place. That was actually a good thing for driving adoption because it was a block. The flood of usage was actually so strong that I believe that is the way that we are seeing the same thing that we’re seeing when it comes to Read AI and AI. It’s that we’re seeing the same level of adoption as we are for smartphones. I’d even say more so from a mainstream perspective because the cost is so minimal, it’s no longer sign a one-year or two-year subscription to get a smartphone and then you have to sign up for a data plan. It’s sign up where it’s completely free or maybe $15 or $20 a month.

Matt: Let’s go all the way back to Placed. You had to learn your way into the business model. I think you had a first assumption, and then you evolved because you listened to the market and you listened to the customers. Tell us about that evolution.

The Evolution of Business Models at Placed & Read AI

David: That was a hard one. Madrona was great for this one. I believed that location analytics was going to be a multi-billion dollar industry back in 2011-2012. I think, ultimately, it did become that; it just took a little bit longer. But the use case that I had was not the right use case. The use case I stuck with for about 12 months was I could pick any business in the United States and give you a trend of foot traffic across time. You could start to see trends like Black Friday, where did people go shop and where did they go shop next. Really cool data.

We got to the point that we were on the Wall Street Journal and the New York Times; it was not a problem getting press. It was also not a problem getting big C-level or VP level meetings because they had never seen the data before. They’re like, “Oh, you could tell me where people come after they visit my competitor? Okay, this is really interesting. I want to look at that data.” Or do you get gas first or groceries first when you go on a trip? We were able to answer those questions, but the downside was there wasn’t a use case for that data. They would come in and say, “This is a great meeting, we love it. Can you give me a little bit more data?” We send the data over and they’re like, “All right, thanks.” And we’re like, “Hey, do you want to buy anything?” And the answer was like, “No. We’re out, peace out, we’re gone.” What do we use this for?

The use case ultimately was the customer coming to us, and they said the customer wasn’t the end consumer. It wasn’t the enterprise clients that we were directly talking with, surprisingly, it was the mobile ad networks and the mobile publishers. They had come to us and said, “Hey, installing games is the ecosystem when it comes to mobile apps today, but we’re trying to get more dollars from brick-and-mortar retailers because we believe that people are in the physical world and you want to be able to close that loop.” They said, no one trusts our data because we’re already selling them the data or selling them the advertising. You don’t trust the person that sells you the advertising to tell you that it works generally in market. That’s changed a little bit today.

But they said, “We know that you have the cleanest data out there. Can you intersect our ads with your panel’s store visits and actually attribute, did someone hear an ad for Target, and then did they actually go to the brick and mortar target location data three days later?” And for the longest time, I said, “No, I believe that we’re a pure play analytics company and we’re not going to do any advertising.” Then you and Scott and the Madrona team were very much like, for the first six to 12 months, “If they’re willing to pay you money for this, maybe you should try it.”

Matt: Maybe you should see what customers who are willing to pay you money would actually be willing to do. The rest is history. It’s a very, very well-built, successful company. Let’s talk about Read AI. You’ve got these technology changes, these societal changes, and then you had to get engagement. How did you think about that? Ultimately, getting alignment with different parties, not just the consumer, but even making it work reasonably well with Zoom and these other platforms.

Engagement and Distribution: Partnering with Platforms

David: On the engagement side, we took the approach of: Work with the platforms. They have the control at the end of the day. They’re the ones that can also get you distribution. And I think with a lot of startups, distribution is a problem where you can have a great product, but if people can’t find it, if people can’t install it, that becomes a problem. And so what we did was we took the approach of working with the platforms. We had great partners at Zoom that said, “Hey, we’re launching this program called Essential Apps. And what Essential Apps does is we’ll put it in front of 10 million users on a daily basis where they will see an app on their right-hand sidebar.” So that was an incredible opportunity and we’re like, “Absolutely, let’s get it done.”

And the same thing came along this past year with Google Add-Ons, where Google said, “We are going to introduce apps or add-ons into Google Meet. We would like you to be one of six apps that are featured in that app store.” We’ve been featured over the last three or four months, and that’s driven significant adoption, and Teams has been similar in terms of the promotion that they’ve given us.

Those platforms and discoveries have enabled us to get a lot of traction. The thing I would say is I made a very similar decision with Placed, but I made it a lot faster. With Placed, I was wrong the first time, the location analytics to understand where people go in the physical world and not combining it with anything else, I just said that standalone is the use case, and that was not the case. The use case was attribution. With engagement, the use case was, “Hey, in a real-time call, when I’m talking with Matt, and he’s a venture capitalist, I want to know when he’s disengaged because when he’s disengaged I can try to recover in that meeting and say, ‘I know this slide’s not very interesting. Let me go to the next one, Matt.'”

The problem was once people started to use it, they didn’t know what to do with it. They saw this chart, it would go up and down. As it went down, they started to get more nervous, “Well, what’s going on? How do I actually recover from this?” And there wasn’t this knowledge base to pull from to say, “Oh, when engagement drops, you should do this.” And so, there was a lot of education that was involved in that process. We found a lot of users; there were certain use cases that we did really well for, especially large groups and presentations. The stickiness wasn’t there. Where we found the stickiness was to go in and say, what can we combine our ability to measure engagement and sentiment in real-time based on a multimodal approach, audio and video?

How can we take that really unique data? These are proprietary models that we’ve built out tons of training data. How can we actually apply that to something else to make it even better? What it came down to was transcripts. There were companies that had been doing transcripts for the last 10 years; some charge per minute; some spot-joined the calls, and some were built into the platform. What you started to see was they were starting to do some summarization, and this was partially due to OpenAI, partially due to their own models in hand where people were asking for, “I don’t read a twelve-page transcript after a call, but I would love to see a summary, and I would love to share that summary with other folks.”

We took that approach of, this is interesting, but everyone can do this. This is a wrapper. This is a commodity at the end of the day where I could take a whole transcript, write a couple prompts and get a summary. And that wasn’t interesting. So we said, “What do we do that is different?” And we applied the scores that we had. So when Matt says this, David reacted this way. When David said this, Matt reacted that way. We created this narration layer that wasn’t available in any transcript, and we played around with it, and we started to see that, “Okay, this is incredibly valuable. This materially changes the content of a summary because you’re taking into account that reaction from the audience.”

Matt: I think moving from trying to give me the assessment of the engagement in the sentiment, you then created what some call an instant meeting recap, but not just a superficial one, quite a robust one because you were actually using a variety of types of data, video, voice, and a variety of models. How did you think about which models are you going to take off the shelf? Which models are you going to customize? How are you going to mix these models together with the data that you have to ultimately produce this output of an instant meeting recap?

The Power of Model Cocktails: Enhancing AI Applications

David: Yeah, that’s a good question. Where we really focus in on them was we are the best when it comes to measuring engagement and sentiment, and then we are the best when it comes to layering that engagement and sentiment on topics, on sentences, on content, on video. Those things were really strong. We then went in and said, “Okay, what is a commodity?” At the end of the day, if you look at OpenAI great partners, you’ve got Llama with Google and Place, you’ve got a bunch of other open-source solutions in market. That is a very hard problem to solve, but it is kind of like the S3 bucket. At the end of the day, it is the commodity that everybody will be using at some point, and you’ll choose between the licensed models that you prefer or the open-source models.

We said, “Hey, if 90% are in-house models, that last 10% where I’ve identified 14 sentences that do the best job of recapping what that meeting was about, here are the action items. We think this had 90% engagement when this content was being discussed.” If you can then load that in to just summarize it down to four sentences, that’s what we’re going to use the third-party solution for. It wasn’t about figuring out how they can analyze engagement. It was more how do we bring our secret sauce into their system? That really did result in differentiated results where on the Coke challenge, we’ve been winning more and more over the last 12 to 18 months where we’re seeing more traction in terms of market share adoption. I think the best feature that we’re seeing is people are starting to copy our features, the legacy incumbents are starting to copy our features.

Matt: Always a great source of compliments when people are copying you. Let’s take us back maybe 12-14 months. You’ve learned a bunch of things. You’ve got this product and this ability to deliver these instant meeting recaps. It’s what I like to call an intelligent application. How do you start to get momentum around customers? At the time, I think you had very little revenue, and today, you’ve got tens of thousands of paying customers — incredible momentum in the business. How did you get from essentially zero revenue at the beginning of last year to the great growth you’ve had over the last 12-15 months?

The Momentum of Read AI

David: That was a bit of a journey. I think where we had in 2022, we had very little traction like I already talked about. We had a lot of interest, we had a lot of users, but ultimately it wasn’t what we’ve seen in the last 12 to 15 months. And that was an iterative process to be upfront. We had the summaries that launched, and we got some traction there, but then people started to come in and say, “Hey, can you do video?” At first, we were like, “Ah, we don’t know if we want to do video.” We did the video and then we started to tune the video where we’ve got three different concepts. The full video recording, everybody has that, but that’s table stakes. Then we went in, and we said, “Hey, highlight reel.” Think of it like ESPN for your meeting where we can identify the most interesting moments by looking at the reaction.

If you think about it this way, if you’ve watched an episode of Seinfeld, you can go into the laugh track, and if you actually look at 30 seconds before the Laugh Track, that actually does a really good job of understanding what people are interested and what people find funny. We started to build this highlight reel, but then we also took into account content. Now you’ve got the content plus the reaction that creates a robust highlight reel. We did something very similar to create a 30-second trailer. The idea here was customers were asking us for video, we enable that. The funny thing that we didn’t do that others did do was we didn’t roll out transcriptions until the summer. We said that is table stakes. Everybody has transcription. That is a commodity service at the end of the day. Yes, you could do it better than other folks, but everyone has it. It’s available in-platforms where you hit a button.

We said, “We don’t want to deliver another copycat product. What we want to do is be the best when it comes to meeting summaries, the intelligent recaps, the key questions, the action items, the highlight reels.” And then we started to go and say, “Okay, we got all this, this is great, but how do we activate against it?” That’s a little bit of the advertising background that I had and attribution background is how do you activate against it? Interesting data is interesting data, but if you’re able to activate it becomes valuable.

So we started to test things like email distribution and Slack distribution where we pushed out the summaries to where people consume the information. We didn’t need to be that central hub for reading the reports. We’re going to send it to wherever you’re used to reading it, and that actually started to gain more traction where people said, “Oh, this is great. After the meeting is done, I get a recap.” Or, “Hey, this is a recurring meeting and you’re going to send me the notes an hour before, so now I can actually review the notes because I forgot why we were going to even do this call.”

Matt: I love that feature. Going back to, I think, what was your original inspiration here is really trying to make me personally more productive and the teams that I meet with collectively more productive. It’s this awesome combination of productivity and action, and I think something you like to call connected intelligence. Talk a little bit more about this concept of connected intelligence.

Read AI: The Future of AI Productivity & Intelligent Agents

David: It really plays in with intelligent apps. Intelligent apps, you’ve got that marketplace set up, but when you go to connected intelligence that’s going in and saying, How do I connect those individual apps so they talk with one another? If I have a meeting about a new product launch, well, that meeting will generate a report today, and it’ll send it out to everyone, and that’s great, and there’s some actionability there, but what if that email, sorry, what if that meeting then talked to an email that was sent up that had follow-ups that said, Here’s the deck associated with that. Here’s some proposed wording for the specific product launch and timelines. Now if those two can talk together, it creates a better summary, it creates a better list of action items, and now the action items are updated where it could go in and say, Hey, David was going to send this deck over to the PR team.

Did David actually send it over? Well, the email’s going to tell the meeting; yes, David did send that over; check that off of the list. So that is a completed deliverable. Now, the follow-up in the email is, Hey, the PR team is going to provide edits associated with this, and it hasn’t been delivered yet because we’re connected to email. These entities, at the end of the day, are able to talk with one another, and they act as your team. We tried the concept of team really early on where it’s like, it joins your meetings, it does this. That was the early version.

What we’re seeing now with the prevalence of AI is you can make each one of these things an entity and these entities can independently talk with one another and deliver content just in time where you’ve got a meeting coming up, you had an action item for the pre-read, but now we’ll look at your Slack messages, your Gmail messages and say, “Hey, these things haven’t been delivered yet, or the client’s going to ask you these three questions. Keep this in mind. That’s going to create a much more productive interaction at the end of the day. A shorter interaction. a more productive interaction.

Matt: No, I love this. It’s kind of going both to some of the things that you’re already doing and also some of the vision of where you’re going with the company. There are all these business processes and all these workflows, and they’re increasingly digitally based. As you point out, it is interconnected between email or Slack or Zoom calls or whatever it might be. I like to think of them as sort of these crossflows. They’re workflows and processes that cut across different applications that I live in. And effectively all those things are different forms of applications. So maybe say a little bit more about where you see this world going and Read AI’s role in it around this vision of connected intelligence.

David: Where we’ve got two similar visions, and this is in the next year we expect to get there. One is, let’s say you’re a project manager, and you have a number of different meetings that occur. You have a number of interactions that happen via email. You’ve got internal follow-ups within Slack and teams, and then you’re updating Asana, Jira, and updating tickets. All of those things today are manual. You have to go in and connect the dots. Now I’m going to look at the meeting report and look at what was delivered to the client. Now I’m going to think about did this file actually get delivered, and then I’m going to go into Asana, I’m going to check some things off, and I’m going to go to the Jira ticket. Those are a lot of steps that take place, and those are steps where it doesn’t require critical thinking. All the information is there. That connected intelligence is there.

Where Read AI is going is we’re going to update all that for you. Where if you’re a product or project manager, all of that mundane work, that busy work that moving around of papers that is taken care of where you don’t need to worry about that, that ticket is automatically updated. Then Jira is going to send an update to the people that are looking at that ticket and then it’s going to say, “Hey, this got completed. This is the next step because that was discussed in one of those entities, one of those connected apps.” You actually inform what the status is.

A bigger problem that comes up is sellers, Salesforce, and HubSpot; where I remember leading revenue at Foursquare, I could not get my sellers to update Salesforce. I would threaten them to say like, “Hey, you’re not going to get paid unless you update it.”

At the end of the day, it didn’t matter how many times I said it, and I was CEO at the time; people were busy. They don’t have time to do it. They’re going to prioritize it internally to say, I’m going to close deals. I’m not going to update the books. Where Read AI is going to come in is we’re going to go in and update that Salesforce opportunity. We’re going to go in and increase the probability of close based on the information that’s available. By doing that, you’re enabling your sellers to do what they do best and go into market. That’s what Read AI does on the back end, is to make sure that everything is up to date, and it’s talking with one another that says, “Hey, seller, you might want to go ping that client because it’s been three weeks and normally your cadence is to ping every three weeks and we see this is missing.”

Exploring Product-Led-Growth and Market Expansion

Matt: I love those prompts and nudges that allow the individual to be more personally productive. I think that’s been one of the great attributes of Read AI. It’s really been a bottoms-up product-led growth kind of motion. What’s cool, of course, is I all the time am using it and people are like, “Oh, what’s that?” And I get to tell them about it. There’s a neat embedded virality, but how have you thought about PLG, and what have you been learning about how to be successful as a product-led growth company?

David: The approach that we took is a little bit not the norm. I think when you’re a startup, you want to focus in on one segment. I think when we started Read AI, we took the approach with support from Madrona, and Matt was the one to say, “Hey, we want to go broader.” We want to go in and be mass market when it comes to engagement sentiment and wherever we apply it because we don’t know. This is a new technology, and we don’t know where it’s going to be used. It took us a while to figure out that product-market fit, but by being broad, we’re able to see use cases that we would’ve never gotten the ability to experience or get feedback on.

I’ll give one example that’s a little bit more individual and less of a big market, but really impactful. We have someone who has dementia, and they reached out to us and said, “This has changed my life because now when I meet with my family, Read AI joins the calls, it takes the notes, and before I meet with my family again online in Zoom, or in Google Meet, I can actually look at the notes and remember what we discussed.” They actually had us connect with their caregiver to say, “Hey, this is how you make sure this is paid for. This account is up to date and make sure all these things are set up.” Because he wasn’t able to do that, but he said, “This has changed my interaction with my family.” That was awesome. That wasn’t a market that we were going after, but that was great.

We see off-label use cases too. Off-label use cases would be —we have government agencies, state agencies, we have treasury departments in certain countries that are using Read AI. When they use Read AI, a lot of times it’s bottoms up. They just saw it, and they’re like, oh, this would help me out. What we found is that the bottoms-up approach finds new use cases. For this one agency that I won’t name, they have clients which we’ll call patients and they’ll go see these patients out in the field.

The old way to do that was they would go to the meeting, they would have a tape recorder, they would record that meeting, they would take notes, and they would interact with that person, and then they would go to the next client, the next client, the next client. A lot of the time, they would spend about one day a week writing the notes, putting them into their patient and client management solution. That was a lot of work. Well, they started to go, and we introduced a feature where you could upload your audio and video. They started to record on their phone. They uploaded the audio from the interaction with the client, and they generated these meeting notes and summaries and action items and key questions, and they just cut and paste and uploaded that into their patient management solution.

They loved it. One person said, “I do not know how to write a report. I did not learn this in college. This is not what I specialized in. Now that I can use Read AI, I can interact with the clients, which is what I want to do. This is my job, and Reed will take care of it and upload all that information. They said this is phenomenal. Then they started to use our tagging feature and say, Hey, we’re going to tag individual clients, and now we can see how things are progressing across time because we don’t just summarize for a single meeting, but a series of meetings. So hey, are things improving? Or did we answer this question that came up last week? Hey, they wanted to know what was going on with this. Did we actually deliver an update on that? Those are things where, a lot of times, we get lost in the noise with the amount of work that we have. Read AI is able to make sure no one’s lost in the noise — none of those action items are lost.

Matt: That’s fantastic. I remember back to some of the earlier days of cloud collaboration and how Smartsheet, one of our companies that we backed, gosh, 16-17 years ago, started out in a very horizontal set of use cases like you’re describing. I think it’s important to have a big disruption, whether it was cloud back then or now, all these capabilities, all these different kinds of models I can use in applied AI to build connected intelligence. And I think that’s part of why you can start more horizontally at this point in time and let the market teach you and tell you about different kinds of use cases

Horiztonal v. Vertical

David: The level of understanding is key, especially in this early market because if we had gone too narrow, we would’ve missed out on these opportunities. I can tell you that 30% of our traffic is international. Outside of the US, that traffic is predominantly centralized in South America and Africa. If you said when I first started Read AI, would 30% of my traffic come from South America and Africa, I’d say, “No, that’s not the market that I would expect to adopt AI very quickly and go in and use it in their day-to-day.” What we’re finding is the adoption has been phenomenal where we’re covering 30%, 40% of a university student base where they’re starting to use it and adopt it.

We’re starting to see our peak; this was a couple of weeks ago: 2% of the population in South Africa was using Read AI, not necessarily as the host of the meeting, but Read AI was on a call that had participants in that meeting. I think those things get me really jazzed up to say like, “Wow, this is something where AI is not just about the enterprise.” There is a clear enterprise opportunity, but it’s how do you help the people from a PLG perspective? How do you actually deliver ROI to the oil driller in Nigeria who has to write a report and send it back to China, which is an actual use case — and they’re using it.

Matt: Wow, amazing set of use cases there. Then sort of embedded in that is just the ability to do this across languages and there’s all kinds of exciting things that you’ve done and you’re working on. One of my wrap-up questions here is what are the challenges for what I’ll call a gen-native company like yours, and in particular relative to the incumbents, the companies that are trying to enhance their existing software applications with generative AI capabilities, how do you think about native versus gen enhanced?

Gen-native v. Gen-enhanced

David: The gen-enhanced, if I was going to say from a competitive standpoint for Read AI, a lot of people would say, is it Copilot? Is it Zoom Companion AI? Is it Google Duets? And for us, it’s not really the case. It’s going in and saying they’re educating the market. I’ve been in a market where I had to educate everyone, and that is a very expensive thing to do. These incumbents are educating the market about the value proposition. People are using it. The free users are going to go in and 80% are going to say, “This is great, this is good enough.” There there is the audience a little older like me where it’s like — if you remember Microsoft Works, that was the cheap version of Microsoft Office. Microsoft Works was $99. Office was $300. A lot of people used Works, and they started to use it, and they’re like, “Oh, this is actually pretty good. But when I need to do a pivot table. Okay. I need to upgrade to that next version.”

What we’re seeing is there’s this whole new base of users that understand AI and the value, and they’re going in and saying, “I need more. I need specialization. I need cross-platform compatibility where half of our users use Zoom and some other solution, or use Google and some other solution, or Teams and some other solution.” From that standpoint, it has been great to actually get the incumbents to adopt this technology and evangelize it.

What you’re going to see is the Smartsheets of the world come in when it comes to AI. You’re going to see the Tableaus of the world, where there’s an incredibly large market to be had there. I think it’s just the start, and I think this is where the consumer and the horizontal play is actually really big, is that we are seeing that AI provides value even without an enterprise involved. If you can take that level of education, accelerate it, and show the value of one step above for $15-$20-$30 a month, that’s a slam dunk. We’re seeing that level of adoption today.

Importance of Model Cocktails

Matt: You’ve got this whole set of cross-platform capabilities. I’ve also been really impressed with the way that you’ve used different kinds of models, some things that you’ve fine-tuned yourself, and others that you leveraged something like an OpenAI and how you’ve brought those things together to get the transcription you were talking about before, to get the very best out of, I like to call it model cocktails, where you’ve mixed a bunch of models together to create these amazing instant meeting recaps and now increasingly these kinds of connected intelligence crossflows.

David: That will be key because if you only rely on one single model, you become a prompt engineering company at the end of the day. We’ve seen some competitors, great competitors, of course, use our solution, but competitors are good, but they’re going deep into like, “Hey, do you want to pay a little bit more for ChatGPT-4 versus 3.5 versus 3?” For us, that just highlights that you’re too dependent on that solution. You’re not differentiating, you’re not adding enough value that you’re just going to show that underlying technology that you’re utilizing. It’s been really important to go and say, “Let’s use a mix of models.”

It’s valuable from a language model perspective for transcription if you use only one single model. There are some really good ones out there, open source as well as paid. If you’re able to leverage two on top of each other, it goes in and says, “How do I stop some of the hallucination that comes up where certain words are totally incorrect? If we’ve got a score between model one and model two and the variance can’t be more than X, you can start to identify points where it starts to hallucinate a little bit or goes off the rails.” Those are kind of those checks and balances that you get when you have multiple models running, and then you bring in your own proprietary model on top of that, it says, “Okay, what other secret sauce can I put into that mix?” I think that is where the market is going to go more than a standalone.

Matt: I totally agree with you. Even more generally, apart from your specific area of personal and team productivity, this is going to be a big year around these intelligent applications and applied AI. Where do you see some of those big areas of opportunity outside of your particular category? What are things that you’re excited about in this world of intelligent applications?

Opportunities for Founders

David: From an education standpoint, I think there’s a lot of opportunity. I’ve been talking with a few teachers at different grade levels, and some of them don’t even know OpenAI exists. Some of them are starting to say like, “Hey, I’ve heard about this. I think my kids are probably using this, but I don’t have a POV there.” I think there’s an ability to provide personalized, scalable education that’s customized to the student. I’m excited about that as an opportunity, especially as an uncle to go in and say, “Hey, where you’re strong, we can make adjustments, and where you’re not as strong, we can provide a little bit more focus in the hand holding that the school system might not be able to provide at any given point in time.” That’s really interesting for me.

I think when it comes to productivity AI, so it falls in our space, but I think there are some really interesting things around things that we do every single day, like emails. Emails could be so much better. There’s the concept of a context window, and these context windows are getting larger and larger. If you can have intelligent apps that have connected intelligence, those context windows aren’t related to just email, you can start to bring in other things. The ability to bring in different data sets is going to find some interesting learnings.

Matt: I love both of those points. The education domain, there are so many opportunities to be helpful to the teachers, more personalized to the students. It’s going to be a very exciting time ahead. As you point out, whether it’s when we just had Llama 3 announced, no doubt GPT-5 is quickly around the corner here. As you say, things like context windows are going to make the capabilities even more robust. It’s going to be an exciting time ahead. You’ve been just an awesome founder and CEO to work with. You’ve got an amazing team, and I’m looking forward to the journey to build Read AI into realizing its fullest potential. Thanks for joining me here today.

David: Absolutely, Matt and you and the team at Madrona have been phenomenal champions, especially for Read AI and for Placed when we were just an idea and the market was starting to get founded. This is the opportunity that if you’re listening here, it’s a great time to actually build a company. It’s never been a better time. It’s never been faster and easier to build and scale.

Matt: Well, let’s go build. Thanks very much, David. Enjoyed having you.

David: All right. Thanks, Matt. Appreciate it.

Listen on Spotify, Apple, Amazon, and Podcast Addict | Watch on YouTube.

Today, Palak Goel hosts Nikita Shamgunov, co-founder and CEO of Neon, a 2023 IA40 winner. Nikita is an engineer at heart, with early stints at both Microsoft and Facebook. He previously co-founded SingleStore, which he grew to over $100 million in revenue. Neon is powering the next generation of AI applications with its fully managed serverless Postgres. In this episode, Nikita gives his perspective on the modern database market, building a killer developer experience, and how both are speeding up the creation and adoption of intelligent applications. Congratulations to Neon for recently hitting GA. And if you’re watching on YouTube, make sure to hit that subscribe button. With that, here’s Nikita.

This transcript was automatically generated and edited for clarity.

Palak: Nikita, thank you so much for joining me. Why don’t you share a bit about your background and what your vision is for Neon?

Nikita: I’m a second-time founder, and my core background is systems. I started, after finishing my Ph.D. in computer science, my first kind of real job, even though I was doing a lot of moonlighting jobs while studying, my first real job was at Microsoft, where I worked on the SQL Server engine. From there, I started a database company called SingleStore. It used to be called MemSQL, and I started as CTO. I wrote the first line of code. I had the experience of building a startup from scratch, went through Y Combinator, and lived in the office for a year until I got a place where we would wake up and code until we went to sleep. Then, I took over as CEO in 2017. The company had about $7 million in run rates, and I scaled it to — as a CEO, that’s how you measure yourself — I scaled it to about $40 million, after which I joined Khosla Ventures as an investing partner. Walking in, I told a famous venture capitalist, Vinod Khosla, that there was another database idea that I was really itching to build, which I couldn’t act on while a SingleStore because you can only have one engine in the company. As I was thinking about it, I already had the idea because I had been thinking about it for three years. And he’s like, “Why don’t you just incubate it here at Khosla Ventures?” Which I did. It took off really quickly. Now that’s the company. Neon is three years old, raised over $100 million, and we’re running 700,000 databases on the management.

Palak: That’s awesome and super impressive. You’ve worn a number of different hats. One of the things you said that caught my attention was that you had the idea for Neon for three years. When did you feel it was the right time to build it?

When To Launch Your Startup Idea

Nikita: I think I’m late. That’s what I think. This idea should have started when AWS introduced AWS Aurora, and it became clear that this is the right approach to building and running database systems in the cloud, as well as the separation of storage and computers. It was just such a good idea for the cloud in general. It doesn’t matter what staple system you’re running. Once that became obvious — and it became apparent in 2015, so I might be seven years late — I was trying to convince some of my former colleagues at SQL Server to go and build it while I was still running SingleStore. I said, “Every successful cloud service inside a cloud could be unbundled, and every successful SaaS service deserves an open-source alternative.” Those are the clear openings for building an AWS Aurora alternative, which I already knew was just such a juggernaut as ramping revenue really, really quickly.

That allows you to first counter-position against Aurora and have very clear messaging because people already understand what that is. I was sitting on that for a while and unable to act on it. I also thought somebody else would build it, and no one did. I see all these database attempts by building shared-nothing systems such as CockroachDB, Yugabytes, and CytosDB. None of them were that gigantically successful, but Aurora was, and I was like, “This is the right approach.” Because I was thinking about this for a while, the map in my head was already planned out. The downside is timing is everything. The opening is narrower, but our execution was very good because the plan was well-thought-out.

Palak: Technology leaders at companies have a number of different databases to choose from. You mentioned CockroachDB, Yugabytes, Aurora, Postgres, and even SingleStore. Where does Neon stand out?

Neon’s Bet on Postgres in the Modern Database Market

Nikita: It’s important to map the modern database market. It’s roughly split in half for analytics and operational systems. The company that still reigns operational systems or OLTP is Oracle, but nobody is excited to start new projects in Oracle. If you take somebody graduating college and want to build a new system or app, they’ll never choose Oracle. Because of the commoditization pressure, they will choose one of the open-source technologies, which is MySQL, Postgres, or Mongo. People won’t choose Yugabytes, or SingleStore, which breaks my heart, and people won’t choose a long tail of niche databases.

Fundamentally, there are two major use cases in that very large modern database market. One is “I want to run my analytics,” a data warehouse and use case or big data analytics use case. And then there is “I want to run my app,” your operational use case. Those of nature on the operational side, it’s clear that Postgres is becoming the dominant force, and workloads are shifting into the cloud.

Now the question is who will own Postgres in the cloud and who will own the share of Postgres as a service? Right now, the king is Amazon, which is why we’re not inventing a new database. We are actually saying we’re Postgres and will be on the trend lines of Postgres becoming the most important operational database on the planet. All the trend lines are there. We want to be on the right side of history. The question is, how do you become the default Postgres offering? And that’s the question that we ask ourselves every day. The answer to that is differentiation. Like what differentiation do you bring to this world? And, what we think that differentiation is — it is the cycle speed for developers. Operational databases exist in the service of their users, and their users are developers.

Palak: That makes a lot of sense. You even touched on a consumerization of developer experience. I’d love to get your thoughts on how you build that 10X better developer experience.

The Importance of Developer Experience

Nikita: When we think about our ICP today, it’s a team of developers who don’t have enough skills to start building their infrastructure. That small team wants to deploy daily and optimize them for cycle and overall speed. Over time every team is going to be like this. What you want to do is zoom out first and see what the set of tools developers use today is. Some of them have complete permanence. They’ve become standard. Something like GitHub has a ton of permanence, and GitHub is not going anywhere. Every day, GitHub entrenches itself more and more in developers’ minds.

Standing up VMs and running node servers in those VMs will go away. Then you keep zooming in, and in there, you will see, “What do developers do every day when they build a feature?” That comes down to the developer workflow. In that developer workflow, people create a branch, and Git Sync creates their developer environments and sends a pool request. Inside that developer workflow, modern tools plug in there very, very well. If you take something like Vercel, it allows you to generate previews. Every pool request will have a Vercel preview. It’s like a copy of your web app but created for just this pull request before you pushed it into production. Guess what doesn’t fit that workflow? Well, the database.

The database is the centralized piece, and you give access to all your previews and developers for that centralized piece, but you run this thing in production. “Whoa, this is scary.” Right? What if some developer writes a query that destroys the data or does something with the database, so now you protect the thing, and databases don’t support that branching preview slash preview workflow? We’re changing that. At Neon, you can always branch the database, so that becomes an enabler. Of course, we are prescribing developers that use Neon, and we’re telling them how they go about building features and what the role of the database is as they build features.

We introduced the notion of branching. We understand migrations, but what I mean by that is schema migrations. We let developers instantly create those sandbox environments with full protection. If there’s PII data, you can mask the PII data and whatnot. Their local and staging environments are fully functional and have access to a clone of their production database. And you can create those environments easily with an API call or a push of a button. This is an example of zooming into the developer workflow and ensuring developers feel like this is built for them. It follows their day.

Palak: As you think about building, especially a modern database company, there’s pressure to have good performance and reliability. How do you build the culture of the company? Or advice for founders who are serving the developer market? how do you keep developer experience as a first-class citizen both in the culture of the company, the people that you hire, and the customers that you bring on? And the reason why I ask is I think Neon has just done an incredible job of that.

Building a Culture of Reliability and Performance

Nikita: There are a couple of things. One is developer experience, and the other is time, performance, and reliability. The latter is incredibly important for a modern database company. If you don’t have them, you will never succeed. The first one is the developer experience, which is your differentiation. You have it, you’re succeeding because the latter — performance, reliability, uptime — they are requirements, it’s necessary. But it doesn’t guarantee that you will be able to compete with Amazon because they have it. Let’s be honest about it. RDS and Aurora are good, reliable services. You can build a business on them. You need both. So how do you get both? Reliability is a function of a few things. Do you have a robust architecture and a good team to develop that architecture? Once you have that in place, it’s a function of time.

You won’t be reliable day zero. That’s why we took more than a year to be in preview, and we started with being extremely open and transparent with ourselves and the world of where our reliability is by showing our status stage and feeling a little naked when people see how reliable your system is. For a modern database company, if you’re not reliable, it attracts a good amount of criticism. And we got it. A good amount of that was in the fall, where the usage went up, and we had this hockey stick growth. We were onboarding, call it a hundred databases a day, at the beginning of 2023. And we started to push 3000 a day by the end of 2023. And so all hell broke loose. Then, if you set the priorities right when you write postmortems, you have a high-quality team, and you slow down your feature development when you need to in the name of reliability.

The architecture is key. If the architecture is off, you’ll never get there. Then, over time, you will get incrementally better and better and better, and eventually, the reliability will be solved. Now that we’re a GA, we feel good about the past history of our reliability and the systems in place, but we can’t, with a straight face, say we are a hundred percent reliable. I don’t think there is such a thing when you run a service, but you can get incrementally better, and at some point, you have enough history to take the calculated risk of calling yourselves GA. When it comes down to developer experience, there are a few things that are important to get right. One is the team like Jiro’s Dreams of Sushi; you must eat good food to make good food.

The tools that we use internally should be top-notch. The tools that we praise out in the world should be top-notch. We love GitHub, we love Vercel, we love Linear, we love super well-crafted, razor-sharp tools that get a lot of developer love. I’m going to call my competitor, which is like a big no-no, but I think Supabase is doing a great job in terms of developer experience. We’re not shy about talking about this internally to level up and potentially exceed the developer experience that they provide. I guess it’s emphasizing what’s important, which is relentlessly investing in your team, having good architecture, understanding what good looks like, and getting better every day.

Palak: You have a long history of working on SQL Server and SingleStore, focusing on reliability and winning those upmarket accounts and workloads. When do you start thinking about when it’s the right time to prioritize that at Neon versus focusing on net new workloads and really good developer experience?

Product-Led Growth v. Sales Teams

Nikita: Heroku is a $800 million run-rate business. Half of this is Postgres, half of that is PLG Postgres, product-led growth, small teams coming into Heroku using Postgres, and the other half is enterprise. You can make a lot of money by providing the best Postgres in the world for small teams, but you don’t want to get stuck in the same place as DigitalOcean. By the way, there is no shame in building a public company, and it’s still growing very healthily all these things, but the stock does not reward DigitalOcean at this point in time. DigitalOcean didn’t go upmarket. We want to go upmarket, but we gate it. We gate it as the signal that we’re getting through our own users that are coming through the side door and just signing up.

We have a good number of enterprise customers, companies like EQT and companies like Zimmer Biomet. There’s a fairly long list of small use cases inside large enterprise customers that will obviously keep expanding, but we don’t want to spread too thin and over-focus on that. So, for example, what does it mean specifically? We don’t have a sales team. We don’t have a sales team, and in all our growth in the last month — we grew 25% months and months in revenue — it is all PLG. And so that’s what focus is looking like. We do have a partnership team and the partnerships team helps us with strategic partners such as Vercel, Re-Tool, and Replit, and making sure we give them the best quality of service possible. And now we’re also in the AWS marketplace and then working with some strategic partners.

Standing up a sales team is not a strategic priority today. We have a very good and thoughtful board that is completely aligned with us in that strategy. And so, how do we gate it? We’ll look at the signal of people coming in on the platform converting and what kind of teams those are. And we are also looking at the partner pool, and these are the things that will eventually tell us like, “Okay, now is the time.” And the book that created a very good impression on me was “Amp It Up” by Slootman. And when he talked about the data domain, they were in the technology development, and then they’ve proven their go-to-market motion in the enterprise because the updated means, obviously, enterprise.

They scaled their sales team very, very quickly. I want to live in the PLG world, and my 200 million of Heroku is telling me that there are plenty of dollars to be captured in this world, but track very closely what’s happening with larger customers driven either them coming in directly to the platform or through partners. And once that’s happening, I’m going to scale the sales team very, very quickly. What I want to avoid is a prolonged process of having a sales team and then keep scratching my head about how expensive it is and how effective it is. I think having a very tiny team early on proving everything, but then scaling very, very quickly is the right way to go.

Palak: It’s 2024, and everyone’s talking about AI. You mentioned you’re in two unique roles. Not only are you CEO of a company like Neon that’s raised $100 million, but you’re also a venture partner at Khosla. You’re uniquely able to weigh in. How big do you think this wave is going to be? Is it all hype, or is it something that is here to last for a while?

Nikita’s Hot Takes on AI

Nikita: So I’m a true believer this will change the world. It’ll change the world of software development and it will change the world of living. The question, how long this is going to take and are we going to have a trove of disillusionment sometime in the future? I don’t know the answer to this question, but I’m as bullish on this thing. I even think that we’ll live in a much more beautiful place — as in our planet — because AI will eventually drop the price of beauty down so much that it will just make sense to have it. And you know how in the seventies we had all this Brutalism architecture, which is super-duper ugly.

I was born in the Soviet Union and grew up in Russia, where Moscow was very pretty. A lot of cookie-cutter houses were built, mostly in the sixties and seventies. The reason they’re so ugly is that they’re cheap to build. I think we will arrive at a place where the cost of building things, software, or even physical things with robots and all those models will be in their robotic brains, will allow us to live in a much more beautiful world. The cost of design is going down, the cost of software engineering is going down, the cost of construction will go down. We’ll kind of re-terraform the planet through this.

Palak: One of the things that every enterprise is figuring out is that every AI strategy requires a data strategy. How are you seeing this impact at Neon, especially focused on net new workloads and net new projects?

Nikita: We’re potentially moving into OLTP, which is more important than analytics. What I mean by that is we just went from, zero cloud spend on analytics to, I think Microsoft is 20 billion. It could be 10, it could be 20, you can cross-check.

Palak: It’s a lot.

Nikita: This is your big data; this is your data warehousing; this is your data lakes; this is your training models and stuff like that. Training like old school models, just like ML and AI workflows of the past. We’re going to have more apps. Apps need databases, operational databases, modern databases. That’s not going to go away, that we’re going to have more apps and therefore we’re going to have more databases. The whole inference thing does not belong to an operational database. It’s all a new thing. It’ll be triggered by both. So that’s an additional line of spend.

Some days, I’m bullish on data warehouses, and other days, this will be older technology because it feels like we’re babying those data warehouses way too much. I’m observing by looking at the data team at Neon, where we’re obsessed with the semantic model, how clean the data is, and how much that represents. The data quality is very important because garbage is in and garbage is out. If you want to have the data in your data warehouse reflect the state of your business, you have to be obsessed with it. Today, people are working on that. I think tomorrow, that whole thing might be a lot more simplified where AI can deal with data being a little dirty because it understands where it’s dirty.

So, you won’t need to make a picture-perfect schema representing your business. I think the center of gravity might shift a little bit where people calling into AI, and I’m going to call out my friend and his tweet on Ankur Goyal and Braintrust Data, where he talked about what changes we are going to have for the data warehousing. He’s obviously looking from his standpoint, and he thinks evals will start pulling data closer to the model and start replacing observability and start replacing maybe even some product analytics at the minimum and then, over time, data warehousing there. I don’t know. It feels archaic to collect data around ETL jobs and run a SQL on a SQL report, but the alternative doesn’t quite exist today, so it’s still best in class.

But that whole AI thing should, some one way or another, disrupt those things where the quality and the full history of that data will matter slightly less.

Palak: Totally. Referencing Ankur’s tweet, it is almost like a real-time learning system of how the app can update itself using AI versus doing it offline, the human way, looking at the warehouse, looking at how users are reacting, et cetera, et cetera.

Nikita: Yes, and the engine of crunching the historical data should still exist. I still want to know how my business is performing. I still want to analyze the clicks, traffic, and product analytics, but how tedious it is today to set all those things. How tedious it is, that should somehow go away. A self-learning kind of AI system that ingests the data like kind a human brain and that thing becomes the driver for running those aggregations, that may change the center of gravity. Today, it’s firmly in the data warehouse or data lake; that is where the centerpiece of the enterprise data today is. We’re super eager to partner with all of them because that’s what the data is. You want to be close to that. Is it going to stay there or not? I don’t know.

Every App Needs A Modern Database

Palak: Do you see there being tailwinds on the transactional side or how do you see Neon fitting into some of these new modern AI architectures?

Nikita: Look who’s doing AI stuff today versus yesterday. In the past, yesterday, it was a data scientist. I don’t know if we need that many data scientists anymore. Today, it’s app developers and prompt engineering and building RAGs. All of them are real-time operational systems. Every app still needs a database. So we’re not replacing operational systems; we are augmenting them. The people who are driving AI value in the company are mostly product engineers running stuff in TypeScript.

The whole Devin demo broke the internet, and we had an incredible amount of excitement about Devin and how Devin can add print statements and, debug your code and run full task. We’re actually using Devin already at Neon. We gave Devin a task to convert our CLI tool from TypeScript to go. It didn’t complete it, but it made enough progress for us to see that maybe it’s not quite there, but it’s almost as good as an intern. And then tomorrow, well, tomorrow, we’ll just do the task. There are so many tedious things that you want to do, like you see, look at the design, and it could be picture perfect. Replace this button from red to gray. We just ran something like this today, and then a human had to do this, but those things shouldn’t be humans doing this kind of thing.

Palak: What intelligent application are you most excited about and why?

Nikita: I’m mostly excited about Devin-like systems. In the whole AI world, we’re replacing humans. We’re not giving humans sharper tools. Software engineering is one of the most, it’s an elite job. We’re able to replace a software engineer, obviously starting from a very junior one, but over time, that’s an incredible amount of economic value because most companies are bottlenecked on the amount of engineering they can ship. I think that’s probably my high-end excitement there. And we’re certainly doing work internally on that front.

Palak: Beyond AI, what trends, technical or non-technical, are you excited about and why?

Nikita: Obviously, our developer experience is very on-brand for us. I think we’re entering a world where there’s just so much technology, and the way to stand out is the incredible degree of craftsmanship that goes into creating a new product or a new tool. I’m both excited about it and in fierce competition with Supabase. They get it, too; we get it, and we will see. This is a Jiro Dreams of Sushi kind of argument here again.

Palak: This isn’t a question, but on one hand, you have AI that’s potentially replacing the junior developers, and then you also have killer developer experience that’s making software engineering more accessible.

Nikita: Correct. By the way, if you make software engineering more accessible for humans, you are also making it more accessible to agents.

Palak: What unexpected challenge have you had where something didn’t go as expected, and how did you deal with it?

Nikita: The team was born as a systems project. Then, the developer experience was layered on top, and we couldn’t really move on to anything developer experience until the foundation of that storage was built. Those things were not on the core DNA of the founding team; it took us a couple of iterations to get there, and we’re not quite there yet, but we’re better and better and better that we see material progress. Something that for a systems engineer may look like, “Oh, of course, that’s a lot easier than doing the hard stuff like the storage thing.” But that turned out to be quite hard to get, and I appreciate the skill sets and the people who go and work on those things.

Palak: Any other parting advice for founders who are just starting the founder journey?

Nikita: I think the one that I would give is, A, don’t wait. I waited on this idea. I should have started this earlier. If you feel like you are ready— and maybe you feel like you’re not quite ready, but close enough—just go and do it. Then, go on the lifelong journey of learning and being passionate about your users and the technology that goes into making delightful experiences for those users. And never look back.

Palak: Awesome. Well, thank you so much, Nikita. I really appreciate you taking the time.

Nikita: Absolutely. Thank you so much.

Listen on Spotify, Apple, Amazon, and Podcast Addict | Watch on YouTube.

Today, Vivek Ramaswami hosts May Habib, co-founder and CEO of Writer, a 2023 IA40 winner. Writer is a full-stack generative AI platform helping enterprises unlock their critical business data to deliver optimized results tailored to each individual organization. In this episode, May shares her perspective on founding a GenAI company before the ChatGPT craze, building an enterprise-grade product and go-to-market motion with 200% NRR, and whether RAG and vector DBs even have a role in the enterprise. This is a must-listen for anyone out there building an AI.

This transcript was automatically generated and edited for clarity.

Vivek: So, May, thank you so much for joining us. And to kick us off, we’d love to hear a little bit more about the founding story and your founding journey behind Writer. And speaking of, you founded Writer before this current LLM hype, before ChatGPT and in this pre-ChatGPT era, we like to call it. And so much has changed since then. So what is Writer today? What is the product? What are the main value propositions for the customers that you’re serving? We’ll get into the enterprise customers, this great list that you have, but maybe just talk a little bit about what Writer is actually doing and how you’re solving these problems for the customers you’re working with today.

May: Writer is a full-stack generative AI platform. Most of the market sells folks either API access to raw large language models or productivity assistance. And there’s lots of room for insider companies to use both solutions and different use cases. For team productivity and shared workflows, there’s a real need to hook up LLMs to unstructured data to do it in ways where there are real guardrails around the accuracy risks, the hallucination risks, the brand risks, the compliance risks, and regulated industries. And the time to value when you’re building things like synthesis apps, digital assistant apps, and insight apps, there’s a real need to capture end-user feedback.

Putting all of that in a single solution, we have drastically sped up the time market on folks spinning up really accurate, ready-to-scale applications; doing that in a secure way, we can sit inside of a customer’s virtual private cloud. And so we’ve had to be able to build that. We’ve had to own every layer of the stack. We built our own large language models. They’re not fine-tuned open source, and we’ve built them from scratch. They’re GPT-4 quality, so you’ve got to be state-of-the-art to be competitive. But we’ve hooked up those LLMs to the tooling that companies need to be able to ship production-ready stuff quickly.

The importance of building your own AI models

Vivek: You talk about how Writer has built their own models, and this has been a big part of the moat that Writer has built, I think this is something that we’ll touch on, but we continue to see as so important is being able to own your stack, given how many different technologies we see competing on either end. In terms of building your own models, what did that entail for how you thought about building this company from day one? And how would you describe what Writer has done around building models and this movement from large models to small models?

May: It’s really hard to control an end-user experience in generative AI if you don’t have control of the model. And given that we had chosen the enterprise lane, uptime matters, inference matters, cost of scale matters, and accuracy really matters. We all really deemed those things early on pretty hard to do if we’re going to be dependent on other folks’ models. And so we made the strategic decision for text, so multimodal ingestion, text production, so we can read a chart and tell you what somebody’s like, I don’t know, blood sediment rate is because we read it somewhere on the chart, we can analyze and add and tell you if those compliant to brand guidelines, but we’re not producing imagery. With the multimodal ingestion to text and insight production, we made a strategic call almost a couple of years ago that we’re going to continue to invest in remaining state-of-the-art. Today, our models are from the Palmyra-X general model, to our financial services model, to our medical model, and our GPT-4 zero-shot equivalent.

When you pair that with folks’ data, it’s pretty magnificent. Unlike other powerful and state-of-the-art models, this is a 72-bill model that can actually sit inside somebody’s private cloud and not require a ton of resources. For us, a whole host of things have allowed us to be state-of-the-art and still relatively small. That’s still a massive internet-scale model, but everything from the number of tokens the models have seen to just how we have trained them has helped us be super-efficient.

Those are all decisions that stem from that first strategic one, and really important problems are going to have to be connected to data that folks don’t want to leave their cloud. And so to do that, we’d have to be in there, and it would have to be a model that could be efficient, and so we weren’t going to have a bunch of different skills in one. That’s why we’ve got 18 different models, similar types of training data, not too dissimilar, but the skills that they are built for are different.

The role of vector databases and RAG in enterprise AI

Vivek: One point you made here makes me think of a LinkedIn post you recently wrote, illuminating in many ways. You talked about unstructured data and where Writer can go using its models. You sit inside an enterprise and take advantage of the enterprise’s data, which is so important. This is something we hear a lot from companies, which is they need to be able to use their own data securely and efficiently when entering data into these models. We’re hearing a lot about RAG, Retrieval-Augmented Generation, and a lot about vector databases, and a number of them have caught fire. We’re seeing a lot get funded. And I’m sure a number of the founders who listen to this podcast have either used or have played with a vector DB. You have an interesting perspective on RAG and vector DBs, especially from the enterprise perspective. Please share a little bit about the posts you wrote and the perspectives that you have about this tech.

May: I don’t want to be like the anti-vector DB founder. What we do is an output of just the experiences that we’ve had. If embeddings plus vector DB were the right approach for dynamic, messy, really scaled unstructured data in the enterprise, we’d be doing that, but it didn’t, at scale, lead to outcomes that our customers thought were any good. A 50-document application, so a digital assistant where folks are querying across 100 or 200 pages across a couple of things, then vector DB embedding approach, fine. But that’s not what most folks’ data looks like. If you’re building a digital assistant for nurses who are accessing both a decade-long medical history against policies for a specific patient, against best practice, against government regulation on treatment, against what the pharmaceutical is saying about the list of drugs that they’re on, you just don’t get the right answers, when you are trying to chunk passages and pass them through a prompt into a model.

When you’re able to take a graph-based approach, you get so much more detail. Folks associate words like ontologies with old-school approaches to knowledge management, but especially in the industries that we focus on and regulated markets and healthcare and financial services. Those have really served those organizations well in the age of generative AI because they’ve been huge sources of data so that we can parse through their content much more efficiently and help folks get good answers. When people don’t have knowledge graphs built already, we’ve trained a separate LLM. It’s seen billions of tokens. So this is a skilled LLM that does this, that actually builds up those relationships for them.

Vivek: I was going to say you were saying that you don’t want to be the anti-vector DB, and I don’t think it’s anti-vector DB; I think it’s things that chunking and vector DBs work for specific use cases. What was interesting about your post was that, hey, from the enterprise perspective, you need a lot more context than what chunking can provide. This is helpful because many of the founders or companies working in narrow areas don’t often see the enterprise perspective, where all of this context matters versus some chunking. You probably got some interesting responses.

May: Both customers and folks were like, “Thank you so much. I sent that to my boss. Thank you, God.” I’m a non-technical person, so when I explain things to myself, I try to share them with other non-technical folks so that they can also understand them, and that actually helps technical people explain things to other technical people.

We got a lot of folks reaching out, and thanks. Now, of course, our customers know this already. Still, we’re in market maturation, where the underlying techniques, algorithms, and technologies matter to people because they seek to understand. In a couple of years, when this is a much more mature market, people will be talking solution language. Nobody buys Salesforce and is like, so what DB is under there? What are you using? Can I bring my own? But we’re not there in generative AI. And I think that’s a good thing because you go way deeper into the conversation.

People are much better at understanding natural limitations. Nobody is signing up for things that just aren’t possible. The other side to this conversation being so technical is there are people who don’t know as much as they would like to and are now afraid to ask questions. We’re seeing that a little bit, especially in the professional services market, where folks need to come across as experts because they’ve got AI in all their marketing now. Still, it’s much harder to have real, grounded conversations.

Navigating the challenges of enterprise sales in AI

Vivek: The commercial side is interesting because there’s so many startups in AI, and there’s so many technical products and technical founders and companies, but not many of them have actually broken into commercial sales. Even fewer of those have broken into enterprise sales. I know Writer has customers like L’Oreal and Deloitte and a number of others, some of which we don’t really associate with being first movers in tech, and especially first movers in AI. And so maybe you can take us through a little bit of how Writer approaches the commercial aspect of things in terms of getting all of these AI solutions in the hands of enterprise users. Take us through the first few customers that Writer was able to get and how you broke into this. What was the story behind that?

May: Our first company sold to big companies, so in the machine translation and localization era, Waseem and I sold to large companies. And we started off selling to startups, and then I can’t remember, someone introduced us to Visa, and we were like, oh, that’s an interesting set of problems. And definitely, that was probably Qordoba circa early 2016. And so for three solid years, we penetrated enterprises with a machine translation solution that hooked up to GitHub repos, and it was great. We just learned a ton about how companies work, and it really did give us this cross-functional bird’s eye view of a bunch of processes because you think about a company going into a new market, they take a whole slice of their business cross-functionally, and that now has to operate in that language. And once you’re in kind of $100 million cost takeout land, it is really hard to go back to anything else.

Our mid-market deals are six figures, and it’s because of the impact that you can have. Now, it does mean that it’s harder to hire, so yes, we’re under 200 people. I’d love to be 400 people. But we’re interviewing so many folks, dozens and dozens for every open role because you really have to have a beginner’s mindset and just this real curiosity to understand the customer’s business. No customer right now in generative AI wants to have somebody who’s learning on the job on the other side of the phone. And the thing is, in generative AI, we’re all learning on the job because this is such a dynamic space, technology is moving so fast, the capabilities are coming so fast. Even we were surprised at how quickly we got to just real high quality. We launched 32 languages in December, GA in Jan, and it was like, whoa, I really thought it would be a year before we were at that level of quality.

All to say that, we need people who can go really deep. Enterprise sales requires everybody in the company to speak customer, and not generic customer, but you’re talking to CPG, it’s a different conversation in retail, it’s a different conversation in insurance, and really understanding how our customers see success. And it’s not this use case or that use case. That’s a real underutilization of AI when you think about it that way. But what are the business outcomes they’re trying to achieve? And really, not just tying it to get the deal done, but actually making that happen faster than without you. That’s what the whole company has to be committed to.

Hiring for GenAI

Vivek: How do you find that profile? Technology is moving so fast that we’re not experts, and many of us are learning on the job and as learning as things come through. At the same time, you have to find a terrific sales leader or an AE or someone who not only understands AI and the product but understands and can speak to enterprises. So hiring is difficult, but how do you find that person, or are there certain qualities or experiences that you look for that you think are the best fit for the sales culture and group that you’re building at Writer?

May: I would start with the customer success culture because that was hard to get to the right profile. We believe in hiring incredibly smart, curious, fast-moving, high-clock-speed people. And we’re all going to learn what we need to learn together. So there’s no like, oh, that was the wrong profile, fire everybody, and let’s hire the new profile. We don’t do that. What I mean by profile is what we need folks to do. And, of course, over time, you refine the hiring profile so you don’t have to interview so many people to get to that right set of experiences and characteristics. On the post-sales side, we’re looking for end-to-end owners. In customer success, it can be easy for folks to be happy that they got their renewal, or we’re over the DAU/MAU ratio we need to be. We’re just going through a check-the-box exercise. We have a 200% NRR business, and it’s going to be a 400% NRR business soon.

And that doesn’t happen unless you are maniacally focused on business outcomes. This is a no-excuse culture, and it’s necessary in generative AI because the gates come around all over. Matrixed organizations are the enemy of generative AI because how do you scale anything? The whole point of this transformation is that intelligence will now scale, and most organizations are not set up for that. As a CSM, you have to navigate that with our set of champions and our set of technical owners inside of the customer, and that just requires real insistence, persistence, business acumen, and a relationship builder who’s also a generative AI expert. So it’s a lot. And then on the CS side, on the sales side rather, it’s definitely the generative AI expertise, and it’s a combination of the hyperscaler salespeople swagger. And we don’t hire from the hyperscalers.

We interviewed a bunch of folks, but it’s like a guaranteed budget item and a guaranteed seven-figure salary for those sales folks. Obviously, the brands are humongous, and the events budgets are humongous, so it just hasn’t been the right profile. We have loved the swagger. When you can talk digital transformation, and you’re not stuttering over those words, there’s just a confidence that comes across. Interviewing lots of different profiles has helped us come up with ours, and it is growth mindset, humility, but real curiosity that does end up in a place of true expertise and knowledge about the customer’s business, the vertical we have potted up in terms of verticalization that’s going to extend all the way into the product as well soon. My guess is most folks who are building go-to-market orgs in generative AI companies are doing more in a year than other companies do in five because our buyer is changing, and the process is changing. It’s a lot of work streams.

Vivek: It’s a lot. And I think I heard you drop the number 200% NRR or something in there. I want to make sure I heard that correctly because that’s really incredible. And so hats off to the team that’s-

May: 209.

Vivek: That’s basically the top 1% of 1%. It’s interesting to contrast that with other AI companies that we’ve seen in the last 12 or 18 months. We’ve all heard stories of others where, probably not enterprise-focused GenAI products, but the term GenAI wrapper has been thrown around, and a lot of them have focused on more of the consumer use cases. They’ve had incredible early growth, and then in the last six to 12 months, we’ve also seen a pretty rapid decrease or a lot of churn. Is that something that you all had thought about early on in terms of the focus of Writer? Did you think about that early as a founder trying to see what worked?

Creating a Generative AI Platform

May: Around ChatGPT time, I think there was a fundamental realization among our team, and we wrote a memo about it to everybody and sent it to our investors, that real high-quality consumer-grade multimodal was going to be free. It was going to go scorched earth. That was clear, and it has come to pass. The other truths that we thought would manifest that we wrote about 15 months ago, every system of record coming up with adjacencies for AI that the personal productivity market would be eaten up by Microsoft. And so for us, what that really meant was, how do we build a moat that lasts years while we deepen and expand the capabilities of the Generative AI platform? And so what was already happening in the product around multifunctional usage right after somebody had come on, we basically were able to use that to really position horizontal from the get-go.

That got us into a much more senior conversation, and we worked to buttress the ability of our Generative AI platform to be a hybrid cloud. We’ve got this concept of headless AI where the apps that you build can be in any cloud. The LLM can be anywhere. The whole thing can be in single tenant or customer-managed clouds, which has taken 18 months to come together. We will double down on enterprise, security, and state-of-the-art models. That’s what we’re going to do, and we’re going to do it in a way that makes it possible for folks to host the solution. I think even those companies have reinvented themselves. A lot of respect for them. But the difference is that in a world of hyperscalers and scorched earth, all the great things OpenAI is doing are super innovative, and every other startup is trying to get a piece. The bar for differentiation went way up 15 months ago for everybody.

Vivek: Hats off on navigating the last 15 to 18 months in the way that you and the team have because it’s incredible to see, compared to a lot of the other companies that are both on the big side and the small size incumbents, startups that are all challenging different parts of the stack. Two questions for you that are more founder-related for you as a founder; let’s start with the first one: what is a challenge that came up unexpectedly, call it, in the last six months that you had to navigate, and how did you navigate it?

May: More than half the company today wasn’t here six months ago. Six months ago we had just closed the series B. And I think in the last few months, it’s been just this transition from running the company to leading the company — if that makes sense. Then working with the entire executive team around just naming the behaviors, the inclinations, the types of decisions that we wanted everybody to be making and to be empowered to make, and then running a really transparent org where information went to everybody pretty quickly.

We’ve got a very competitive market that’s really dynamic, that is also new. Signal-to-noise has to be super high or else everybody would end up spending 80% of their day reading the news and being on Twitter. We needed folks to make the right decisions based on the latest insights, the latest things customers and prospects were telling us, the latest things we were hearing, latest things product was working on, and all those loops had to be super tight.

Vivek: Its execution and speed matters, especially in this space.

May: Yes and execution while learning. I think it’s easier if you’re like, all right, Q1, OKRs, awesome, CU and a Q1. But this is a really dynamic space, and the hyperscalers are militant. This is really competitive.

Vivek: All right, last one for you, what’s the worst advice you’ve ever gotten as a founder?

May: The worst advice that I have taken was early in Qordoba days, hiring VPs before we were ready. It felt like a constant state of rebuilding some function or other on the executive team. That’s such a drain. We have an amazing executive team, we’ve got strengths, we’ve got weaknesses. We’re going to learn together. This is the team. And it’s why we spend so long now to fill every gap. We’ve got a head of international, CFO, CMO. We’re going to take our time and find the right fit. But those were hard-won lessons. The advice that we got recently that we didn’t take, was to not build our own models. And I’m really glad we didn’t take that advice.

Vivek: I was wondering if that might be something that came up because you’re right; we see so much activity around saying, hey, build on top of GPT, build on top of this open-source model. It works for some sets of companies, but as you say, thinking about moats early on in technology and IP moats from the very beginning is only going to help you in the long run. Well, thank you so much, May. Congrats on all of the success at Writer so far. I’m sure the journey’s only beginning for you, and we’re excited to see where this goes.

May: Thank you so much, Vivek. For folks listening, we’re hiring for literally everything you might imagine. So I’m May@writer, if this is interesting.

Vivek: Perfect. Thanks so much.

Listen on Spotify, Apple, Amazon, and Podcast Addict | Watch on YouTube.

Madrona Partner Jon Turow hosts Gabriel Hubert and Stan Polu, the co-founders of a 2023 IA40 winner, Dust. Dust enables information workers to create LLM-based agents that help them do their own work more efficiently. Gabe and Stan go way back, having started a data analytics company that was bought by Stripe in 2015 where they both stayed for about five years. After stints at a couple other places like OpenAI for Stan, they decided to come back together in 2023 to work on Dust.

In this episode, Jon talks to Gabe and Stan about the broader AI/LLM ecosystem and the classic adoption curve of deploying models within an enterprise. They talk about Dust’s bet on horizontal versus vertical, the builders of tomorrow, limitations of AI assistants, and so much more.

This transcript was automatically generated and edited for clarity.

Jon: You have made a fundamental product choice that I’d like to study a little bit, which is to build something horizontal and recomposable. Depending on what choice you made, I imagine we’d be having one conversation or the other. On horizontal, on composable, you’re going to say, “There’s so many use cases. Flexibility is important, and every different workload is a special snowflake.”

If you had decided instead to build something vertical, it doesn’t have to be AGI, but you’d be saying, “To really nail one use case, you need to go all the way to the metal.” You need to describe the workflow with software in a very, very subtle and optimized way. Both of these would be valid. How do you think about the trade-off between going vertical and going horizontal? How have you landed on that choice that you made?

Stan: It comes with complexities and risks to go horizontal. At the same time, if you assume that the product experience will evolve so fast, it’s also the best place to do that kind of exploration, that playground for product exploration. It’s the intersection of all the use cases at the intersection of all the data. During the current age of GenAI where we are in a conversational agent interface, we are making the right bet. It means we are on par with the UX, but we have access to much more data and much more flexibility.

Empirically, we have checked that hypothesis. This means that some of our users have been confronted with verticalized solutions. We beat the verticalized solution in many cases because we provide more customer stability. Customer stability comes in two flavors. First, provide instructions that better match what the user wants to do in their company, depending on the culture or existing workflows. Second, being able to access multiple sources of information.

On some use cases, having two sources of information and one being the one associated with the verticalized use case, you are much better off because you have access to all the information that lives on Slack, all the information that lives on Notion that’s related to the best case use case. Many of them could be sales and customer support use cases. That enables people to create assistants that are much better by being able to tap into that information that the verticalized SaaS products will never be able to tap into. As a verticalized product, it will be either the incumbent or somebody building on an incumbent, somebody building on a customer support platform, somebody going on the sales data platform, whatever it is.

In that interim, where the UX is being defined, we have an incredible advantage of being horizontal. That may close in the future because as we realize what it is to use models in a particular use case efficiently, then it’s all going to be about not getting a text, but it’s going to be about getting an action ready to be shipped. It’s about being able to send an email to prospects automatically from the product, etc. There, we might have a steeper curve to climb with respect to the verticalized solution because they’ll be more deeply integrated and have the ready-to-go action built probably before us. It’s still an open-ended question. As it stands today, it creates a tremendous advantage.

We’re still in the process of exploring what it means to deploy that technology within companies. So far, that bet has been good, but that’s the most important product purchase we’ve made. We’re checking it every week, every month, almost every day. Was that the right choice? We can comfortably feel that it is the right choice today. We also realize we’re inside an ecosystem that is on moving ground. That’s something that has to be revisited every day.

Gabe: Some of the convictions from data access being a horizontal problem rather than a vertical problem when you distribute to a company have helped with that choice. You can convince a CISO of the validity of your security measures and practices. It’s just as hard to do whether you need one sliver of that data set or all. Your bang for the buck is better when you want to play with the more concrete use case of having access to different sources of information to deliver on a fairly simple example. Imagine you had access to a company’s production databases. You could generate very smart queries on those production databases any day of the week. That’s a product that we hope to see today with these SQL editor interfaces that everybody in the team is supposed to use.

Where in that product is the half-page document describing an active user? Or what a large account is? What do we consider to be a low performance on customer support scores? Those documents live in a different division as the company grows. It’s a division that doesn’t even know a code version exists. It’s meetings where their definitions are updated in a very business-driven setting. That constitution of what an active user is in a separate set of data.

For somebody within the company at a low cost, to be able to ask an agent a question about the number of active users in Ireland or the number of active and satisfied users, you have to cross those data sets. That’s almost systematically the case. A lot of the underperformance you could look at or a skeleton audit in companies today comes from this siloed conversation. To us, being excited to start another company, trying to build an ambitious product company with the experience we’ve had from companies that have seen traction, that have grown fast, that have burnt out, incredible individual contributors that start seeing the overhead and the slowness and how tricky it is to just get a simple decision pushed through because nobody has the lens to see through all these divisional silos, it seems more exciting too to build a solution to that problem. When we pitch it to people who’ve seen and felt that pain, that’s the spark that goes off.

It is a risk. But the people excited with that type of software existing in their team, I argue that they were excited to build for the years to come. Come back in five years, and let’s see if we were right on it.

Jon: Oh, come on. It’s going to be one year.

Gabe: Yeah. That’s true. Fair enough. Fair enough.

Jon: It’s so tempting to project some Stripe influence on the horizontal strategy that you guys have taken. Before I hypothesize about what that would be, can you tell me what you see? What is the influence of how Stripe is put together on the way you’re building Dust?

Stan: In terms of how Stripe was operating, which influenced us in defining what we were to build with Dust, there is a lot of creating a company OS where people have the power to do their job at that desk, which means having access to information and a lot of trust. Some of that has trudged to our product decisions. We’ve built a product that’s not a great fit for a company with granular access control on who has access to that very small folder inside that Google Drive. Those people added only manually on a case-by-case basis.

We are optimistic that the companies that will work the best in the future are the ones that make their internal information available. That is, at the same time, a great fit for building AI-based assistants.

Gabe: Regarding the product, there’s a ton of inspiration from what Stripe did to developers. It gave them the tools to have payments live in the app before the CFO managed to meet with an acquiring bank to discuss a payments pool. It was like if the developer came to the meeting and said, “It’s life. We’ve already generated a hundred bucks in revenue. What are the other questions?”

I think if we can build a platform that puts to bed some of the internal discussions as to which provider of a frontier model we should go and trust for the next two years, a builder internally says, “I just built this on Dust, and we can switch the model in the back if we feel like it’s going to improve over the next months.”

That’s a scenario that the aggregation position is a good one. It requires you to be incredibly trusted. It requires composability. It does mean a set of technical decisions that are more challenging locally. But it enables, optimistically, I think to Stan’s point, some of the smarter, more driven people to have the right toolkit. That’s something that we take from Stripe. Stripe was not a fit for some slower companies when we were there, which ended up okay.

Jon: When we think about the mission you folks are going after, there’s so much innovation happening at the model layer. One thing that we’ve spoken about before is there’s a lot you can accomplish today. When we start talking about what it is that you can accomplish with Dust, can you talk about the requirements that you have for the underlying tech?

Gabe: One of the core beliefs we had in starting the company that essentially dates back to Stan’s time at OpenAI and his ability to be front-row seats at the progression of the various GPT models is that the models were pretty good at some tasks and surprisingly great at some tasks. We will face a deployment problem for that type of performance to reach the confines of the online world. The opportunity was to focus on the product and application layer to accelerate that deployment. Even if models didn’t get significantly more intelligent in the short term if you’re a little hesitant about calling an AGI timeline in the years to come, there’s an opportunity to increase the level of effectiveness people can have when they have a job that involves computers with the current technology.

In terms of requirements with the technology, for us, it’s let’s make the models that are available today, whether they’re available via API as business models or they become available because they’re open source and people decide to host them and make them available via API, et cetera, et cetera, and package them such that smart, fast-moving teams can access their potential in concrete day-to-day scenarios that are going to help them see value.

Stan: The interesting tidbit here is that the models will get better as research progresses. On their own, they’re not enough for deployment and for acceptance by workers. At the same time, the kind of use cases that we can cover depends on the capability of the model. That’s where it’s different from a classical SaaS exercise because you’re operating in an environment that moves fast. The nature of your product itself changes with the nature of the models that it evolves.

It’s something that you have to accept when you walk in that space — the hypothesis that you make about a model might be changed or might evolve with time, and that will probably require changing or evolving your own products as that happens. You have to be very flexible and be able to react quite rapidly.

Jon: There are two vectors we spoke about before when we’ve discussed this in the past. One is that if you stop progress today with the underlying models, there are years of progress that we can make. The other is that if we go one or two clicks back in history to say mobile apps, we saw that there were years of flashlight apps before the really special things started to arrive.

Where would you put us on that two by two of early models versus advanced and how much it matters versus not?

Gabe: What’s interesting is to talk about people who’ve been on the early adopter side of the curve, who’ve been curious, who’ve been trying models out, and who’ve probably been following the evolution of ChatGPT as a very consumer-facing and popular interface. You get this first moment of complete awe where the model responds in something particularly coherent. I’m asking it questions, and the answers are just amazing. Then, as I repeat use cases and try to generate outputs that fit a certain scenario, I’m sometimes either surprised or disappointed.

The stochastic nature of the output of the models is something other than what people think about at the very beginning. They attribute all of the value to pure magic. Then, as they get a little more comfortable, they realize that it might still be magic, or they might be unable to explain it technically. Still, the model isn’t always behaving in a way that’s effectively predictable enough to become a tool.

We’re early in understanding what it means to work with stochastic software. We’re on the left side of the quadrant. In terms of applications, the cutting-edge applications are already pretty impressive. By impressive, I mean that they fairly systematically deliver at a fraction of the cost or at a multiple of the usual speed, an outcome that is relatable with or on par with human performance.

Those use cases exist already. You can ask GPT-4 or a similarly sized performant model to analyze a 20-page PDF. In seconds, you will get several points that no human could compete with. You can ask for a drill-down analysis or a rewrite of a specific paragraph at a fraction of the cost of what you can ask on a Fiverr marketplace or an Upwork marketplace for some tasks. We already have that like 10x faster, 10x better.

In terms of broad adoption, especially by companies, if you take ChatGPT with a few hundred million users, that still leaves 5.4 billion adults who have never touched it and don’t even know what it means by some scale for sure. If you go into the workplace, there are very few companies that are employing generative artificial intelligence at scale in production that were not created around the premise of general artificial intelligence being available.

Some companies have been created in the last years and do, but most companies that existed before that timeline are exploring. They’re exploring use cases. They’re rebuilding their risk appetite and policies around what a stochastic tool might bring regarding upside and downside opportunities and risk. We’re still very early in the deployment of those products.

One indication is that the conversational interface is still the default that most people are using and interacting with when it’s likely that it shouldn’t just be conversational interfaces that provide generative artificial intelligence-powered value in the workplace. Many things could be workflows; many things could be CronJobs. Many applications of this technology could be non-chat-like interfaces, but we’re still in a world where most of the tests and explorations are happening in chat interfaces.

We still want the humans to be in the loop. We still want to be able to check or correct the course. It’s still early.

Stan: One analogy I like to use is that today, to recall what Gabriel just said on the conversational interface, it really feels like we are in the age of Pong, the game for models. You’re facing the model. You’re going back and forth with it. We still need to start scratching the multiplayer aspect of it. We haven’t yet started scratching and interacting with the model in new ways and more efficient ways.

You have to ask yourself, what will be the civilization for LLMs? What’s going to be the Counter-Strike of LLMs? That is equally important to dig into compared to model performance. The mission we set for ourselves is to be the best for our users to be the people who dig in that direction and try to invent what’s going to be the C-5 of interacting with models in the workspace.

Jon: Can you talk about the mission for Dust in the context of the organization that’s going to use it?

Stan: We want to be the platform that companies choose as a bet to augment their teams. We want it to be a no-brainer that you must use Dust to deploy the technology within your organization. Do it at scale. Do it efficiently. This is where we’re spending cycles. It’s funny to realize that every company is an early adopter today. The board talks about AI, the founders talk about AI, and so the company itself is an early adopter. But once you get inside the company, you face a classical adoption curve. That’s where product matters because that’s where the product can help deploy the companies through that chiasm of innovation inside the teams. We want to be the engine of that.

We’re not focusing on the models; we’re trying to be agnostic of the models, getting the best models where they are for the needed use cases. Still, we want to be that product engine that makes deploying GenAI within companies faster, better, safer, and more efficient.

Gabe: One of the verbs that we use quite a lot that is important is augmenting. We see a tremendous upside scenario for teams with humans in them. We don’t need to spend a lot of our time working for companies that are trying to aggressively replace the humans on their teams with generative artificial intelligence because that’s shaving a few percentage points of operational expenditure. There’s a bigger story, a play here, which is if you gave all of the smartest members of your team an exoskeleton, an Iron Man costume now, how would they spend their day, their week, their quarter? What could happen a few quarters from now if that opportunity compounds?

When we decide at Dust about different avenues to prioritize, one that’s consistently a factor is whether we are augmenting or replacing. By replacing, there’s a high likelihood that we, one, provide faster horses to people who see the future as extrapolated from the present. It’s like, “I need to replace my customer service team with robots because robots are cheaper,” when the entire concept of support and tickets as an interface between users and a company is to discuss how a product is misbehaving and may be challenged in the future.

It’s a tension for us because there are some quick wins or deployment scenarios that many companies are considering. It helps us explore and spend time on some of the cars instead of the faster horses scenarios dawning upon us.

Jon: I think it has implications not just for the individual workers, but to your point, Stan, and to your point, Gabriel, there’s going to be a difference in how the employees interact with one another. I’ve just put it to you one way. If I’m going to decide whether to join your company or not, and you’re going to tell me, “You should because I have Dust,” what would be the rest of that explanation?

Gabe: It’s a great point. I think that that’s an example we sometimes use to describe what an ambitious outcome for the company in a few years’ time or what an ambitious state of the world would be for our company in a few years’ time. If you take a developer today — the senior developer getting offers from a number of companies — and in the interview process getting to ask questions about how that company runs its code deployment pipeline. I can ask how easy it is to push code into production, how often the team pushes code into production, and what a review process looks like.

I can read a lot into the culture of that company on how it respects and tries to provide a great platform for developers. Today, developers are at the forefront of being able to read in the stack that the company has chosen, how they prioritize their experience. If you do not have a version control software that allows for pull requests, reviews and cloud distribution that works and is fast, I don’t think you’re very serious about pushing code.

We think that the future has more of the roles within a company having associated software. You could argue that, to a degree, Slack has created that before and after aspect, where if you’re applying at a company today and you ask how do employees check in with each other in an informal way to get a quick answer on something that they’re blocked on and the employer says, “We have a vacuum tube system where you can write a memo and just pipe it in one of the vacuum tubes that’s available at the end of the floor and you’ll get a response within two days,” that should help.

You’re like, “Okay, great.” I don’t think that real-time collaboration is prioritized. We think there’s a continuum of those scenarios that can be built. For us to be able to imagine a future where employees say, “Hey, we run on Dust.” We would love that to be synonymous with, “Hey, we don’t prioritize or incentivize busy work.” Everything that computers can do for you, which really computers should have been doing for you decades ago, we’ve invested in a technology that helps that happen. We’ve built a technology that helps burn through overhead and time sinks of finding the information, where it is, understanding when two pieces of information are contradictory within a company and getting a fast resolution as to which one should be trusted. The OS of that smart fast-growing team is something that we hope to be a part of the strategy for.

Jon: That’s such an evocative image of the vacuum tube. I actually bet if there were a physical one, people would like that as long as there was also Dust.

Gabe: It could be a fun gadget that people actually send employee kudos notes to at the end of the week and just team phrase updates.

Jon: What we’re talking about, though, is there’s a metaphor of the agent. Which is really in our 2023, and 2024 brain, we think of that as another member of the team. Maybe a junior member of the team at the moment. I think it was something you said, Gabriel. That the binary question of whether it’s good enough or not is actually not useful. But rather, how good is it? How should I think about that?

Gabe: Yeah. I stole it from the CIO of Bridgewater, who their communication around GPT-4 was compared to a median associate or analyst, I can’t remember what the name of the roles was. We believe that it performs slightly higher than a junior-level analyst on these tasks. Bridgewater is a specific company that has an opportunity to segment a lot of its tasks in a way that maybe not all companies are able to do.

As soon as you’ve said that, a number of logical decisions can be made around that. We often get asked about specific interactions that people have had with Dust assistants. Like, “Hey, why did it get that answer wrong?” I was like, “Assistants are getting a tough life in those companies because a lot of your colleagues get a lot of stuff wrong a lot of the time, and they never get specifically called out for that one mistake.”

That’s part of the adoption curve that we’re going through where it’s at the level of an interaction. You’re looking at a model that might be, on average, quite good and sometimes quite off. Instead of turning your brain off, you should probably turn your brain on and double-check. At the level of the organization, you’re getting performance that is, in specific scenarios, potentially higher than the median. Then, it was higher than if it got pushed to another team member for that specific set of tasks.

As models get better and the structural definition of the tasks we give them gets clearer, and as the interfaces that help feedback mechanisms get more and more used, those scenarios will progress. The number of times you feel like the answer is good enough, better than what you would’ve come up with on your own, or better than what you could have expected if you had asked a colleague.

One of the things that we systematically underestimate here is also the latency. Ask a good colleague to help you with something. If they’re out for lunch, they’re out for lunch. That’s two hours. General Patton is the one who says, “A good plan violently executed today beats a perfect plan executed tomorrow.” If, as a company, you can compound and rely on that compounding edge in terms of execution speed, the outcomes will be great.

Jon: What we’re talking about is assessing agents not by whether they’re right or wrong but by their percentile of performance relative to other people. Yet, there’s another thing that you both have spoken about: the failure modes will be different. It’s easy for a skeptical human, especially, to say that one weird mistake means this thing must be done.I don’t think it would be the first time in history that we’ve mis-evaluated what technology would bring us by judging it on some anecdotal dimensions.

Stan: Something interesting to jump back on your 2023 brain and how we might not be foreseeing correctly; there’s a massive difference between having intern-level or junior-level assistants where this is a human, so you want to leave the task to them entirely and leave some agency to them. The shape of tasks that can be given to that person is defined by their capability and the fact that they’re junior and have assistants where the agency is kept on the side of the person who has the skills. There’s a message difference between what you can do with junior-level assistants where you keep the agency versus just junior assistants for the humans.

It will be interesting to see how that automation and the augmentation of humans play out. It might be the case that it will be very different from adding 10 interns to a human and adding 10 AI-based assistants to human. It may well be the case that 10 AI assistants augment humans much more than having 10 interns. There’s going to be an interesting landscape to discover.

Jon: Depending on how you frame this, I’m either going forward or backward in history from unlimited manual labor to spreadsheets. A Dust agent reminds me in many ways of a spreadsheet.

Gabe: In terms of capability and the level of abstraction versus the diversity of the tasks, that’s not a bad analogy. It’s unclear if the primitives that currently exist on Dust are sufficient to describe the layer and space that we really intend on being a part of. If we are successful, Dust will probably retain some properties of malleability, the ability to be composable, programmable, and interpretable by the various humans that are using it, which does remind me of spreadsheets, too.

Jon: One thing that you see in your product today is a distinction between the author and the consumer of a Dust agent. It’s reasonable to expect there’s going to be a power law of distribution of more people consuming these things than creating them. Were there some way to really measure spreadsheet adoption? I’m quite sure we’d see the same. That a handful of spreadsheets, especially the most important ones, get created by a small number of us and then consumed by many more of us.

These things are infinitely malleable, and many people can create a silly one that is used once and thrown away.

Gabe: We see that today in some of our customers, who will see the assistants. I had a product manager admitting that they had created a silly assistant mixing company OKRs and astrology to give people a one-paragraph answer on how they should expect to be doing in the quarter to come. They were admitting that it was a distribution mechanism for them. It’s like, “I just want people to know how easy it is to create a Dust assistant, how easy it is to interact with it, and how non-lethal it is to interact with it, too.” There’s always that fear of use case, all of this usage scenario.

The reason we believe that they’re not going to be developers is that the interface has become a natural language in many cases; you’re essentially just looking at the raw desire for some humans to bend the software around them to their will. I think the builders of tomorrow with this type of software have more in common with very practical people around the house who are always fixing things and who won’t let a leak go for two weeks unattended, who’ll just fix the leak with some random piece of pipe and an old tire. It just works, and it’s great. That is seeing opportunity and connecting the Lego bricks of life.

One of the big challenges for companies like us is how to identify them. How do you let them self-identify? How do you quickly empower them such that the rest of the team sees value rapidly? One of the limitations of assistants like Dust within a company is access to the data that the company has provided to Dust. The number of people controlling access to the data gates is even smaller than the number of people who can build and experiment with assistants in some cases. How can a builder reach out to somebody at the company with the keys to a specific data set and say, “Hey, I have this really good use case? This is why I feel we should try it out. How could I get access to the data that allows me to run this assistant on it?” Those are all the product loops.

They have nothing to do with model performance. They have everything to do with generating trust internally about the company, the way the product works, the technology, and where the data goes, all these things that are product problems.

Jon: If I move half a click forward in history, you start to think about data engineering and how empowering it was for analysts to get tools like dbt and other things that allowed them to spin up and manage their own data pipelines without waiting for engineers to turn the key for them. That created a whole wave of new jobs, a whole wave of innovation that wasn’t possible before. To the point that now, it’s impossible to hire enough data engineers.

There’s this empowering effect that you get from democratizing something within a company that was previously secured — even if for a really good reason. I’m connecting that to the point that you made, Gabe, the data itself that feeds the agents is often the key ingredient and has been locked down until today. Based on the use cases that you’re seeing, this is going to be a fun lightning round. My meta question is, has the killer agent arrived? Maybe you can talk about some of the successes and maybe even some of the fun things that aren’t quite working that your customers have tried.

Gabe: I think that killer agent is a product marketable concept that you can slap on your website, and 90% of people who visit upgrade, regardless of their stage of company, their developing, et cetera, et cetera, I don’t think we’re there yet. They ask questions that the Dust, let alone an LLM without a connection to the data, would have no chance of answering.

Those are some interesting cases where I think we’re failing locally and tactically because the answer is not satisfying. Where I’m seeing weak signals of success is that people are going to Dust to ask the question in the first place.

On some of the use cases that we’re incredibly excited about, it’s almost similar situations, but with satisfactory answers, where people are asking surprisingly tough questions that require crisscrossing documents from very different data sources and getting an answer that they unmistakably judge as being way better than what they would’ve had by going to the native search of one SaaS player or by asking a team member, et cetera, et cetera.In some cases, the number of assistants that midsize companies generate on Dust is high. Do you see that as a success or a failure? Does that mean that you’ve been able to give the tools for a very fragmented set of humans to build what they need, and you interpret it as success? Or we’ve essentially capped the upside that they can get from these two specific assistants? That’s still one of the questions that we’re spending a lot of time on today.

Jon: If we go back to our trusty spreadsheet metaphor, there are many spreadsheets. They’re not all created equal.

Gabe: Yeah, it’s fine. Maybe it’s fine. Maybe yeah, not all spreadsheets need to be equal.

Jon: Thank you so much for talking about Dust and your customers. I think customers are going to love it.

Gabe: Awesome. Thank you very much for having us.

Stan: Thank you so much.

Coral: Thank you for listening to this IA40 Spotlight Episode of Founded & Funded. Please rate and review us wherever you get your podcasts. If you’re interested in learning more about Dust, visit www.Dust.tt. If you’re interested in learning more about the IA40, visit www.IA40.com. Thanks again for listening, and tune in a couple of weeks for the next episode of Founded & Funded.

Listen on Spotify, Apple, Amazon, and Podcast Addict | Watch on YouTube.

Today Madrona investor Palak Goel has the pleasure of chatting with Glean founder and CEO Arvind Jain. Glean provides an AI-powered work assistant that breaks down traditional data silos, making info from across a company more accessible. It should be no surprise that Glean is touted as the search engine for the enterprise because Arvind spent over a decade at Google as a distinguished engineer where he led teams in Google search. Glean has raised about $155 million since launching in 2019 and was named a 2023 IA40 winner. Palak and Arvind talk about Glean’s journey and the transformative power of enterprise AI on workflows, the challenges of building AI products, how AI should not be thought of as a product but rather as a building block to create a great product, the need for better AI models and tooling, and advice for founders in the AI space, including the importance of customer collaboration in product development, the need for entrepreneurs to be persistent – and so much more!

This transcript was automatically generated and edited for clarity.

Palak: Arvind, thank you so much for coming on and taking the time to talk about Glean.

Arvind: Thank you for having me. Excited to be here.

Palak: To kick things off, in 2023, VCs have invested over a billion dollars into generative AI companies, but I can probably count on one hand how many AI products have actually made it into production. As someone who’s been building these kinds of products for decades, why do you think that is and what should builders be doing better?

Arvind: People are doing a great job building products. I’ve seen a lot of really good ideas out there leveraging the power of large language models. Still, it takes time to build an enterprise-grade product that can be reliably used to solve specific business problems. When AI technology allowed people to create fantastic demos that you can amazingly solve problems, the expectation in the market went up very quickly. Here’s this magical technology, and it will solve all of our problems. That was one of the reasons we went through this phase of disappointment later on, which is it turns out that AI technology, while super powerful, is extremely hard to make work in your business.

One of the big things for enterprise AI to work is that you connect your enterprise knowledge and data with the power of large language models. It is hard, clunky, and takes effort, so I think we need more time. It’s great to see this investment in this space because you’re going to see fundamentally new types of products that are going to get built and which are going to add significant value to the enterprise. I expect we will see a lot of success in 2024.

Palak: What are some of those products or needs that you feel are extremely compelling or products you expect to see? I’m curious about the Glean platform. From that perspective, how are you enabling some of those applications to be built?

Arvind: If you think about enterprise AI applications, a common theme across all of them is that you connect your enterprise data and knowledge with the power of LLMs. Given a task that you’re trying to solve, you have to assemble the right pieces of information that live within your company and provide it to the LLMs so that the LLM can then do some work on it and solve the problem in the form of either providing you with an answer or creating some artifact that you’re looking for.

From a Glean perspective, that’s the value that we are adding. We make connecting your enterprise data easy with the power of large language model technology. We want to take all the work of building data, ETL pipelines, figuring out how to keep this data fresh and up to date, and setting up a retrieval system where you can put that data in so that you can retrieve it at the time a user wants to do something.
We want to remove all of that technical complexity of building an AI application from you and instead give you a no-code platform that you can use and focus more on your business or application logic and not worry about how to build a search engine or a data ETL pipeline. We will enable developers inside the enterprise to build amazing AI applications on top of our platform.

The use cases I always hear from our enterprises are, first, for developers, you already know how the code copilots add a lot of value. Statistics show that about 30% of all the code is written by copilots. You’re starting to see good productivity wins from AI for software development. You can use these models for test-case generation, and there are a lot of opportunities. We’re still improving that 10% of the time that developers spend.
Developers are not spending most of their time writing code. Most of the time, they’re figuring out how to solve a problem or a design for the solution. Most of the focus is on the scope of AI. AI focuses on 10 or 20% of the time a developer spends to bring some efficiencies there.

Next year, we will see many more sophisticated tools to increase productivity in that entire developer lifecycle. Similarly, for support and customer care teams, AI is starting to play a significant role in speeding up customer ticket resolution and answering tickets that customers have. These are some of the big areas in which we see a lot of traction today.

Palak: As developers move from prototyping to production, do you think it’s a lack of sophistication in the tooling around some of these models or is it the models themselves that need to get better?

Arvind: It’s both. AI models are smart, but everybody has seen how they can hallucinate. They can make things up because, fundamentally, they are probabilistic models. All the model companies are starting to see, whether in an open-domain or closed-domain models, that incredibly fast progress is happening there to make these models better, more predictable, and more accurate. At the same time, a lot of work is happening on the tooling side. For example, if I’m building an AI application, how do I ensure it’s doing a good job? The eval when we started earlier this year when we built an application, all the evaluation of that system was manual. Our developers were spending the majority of their time trying to evaluate whether the system was doing well because there was no way to sort of automate that process. Now, you’re seeing a lot of development in AI evaluation systems.

Similarly, there is infrastructure on models, making sure you’re not actually putting sensitive information into the models or taking sensitive output and showing it back to the user, so privacy and security filtering, that plumbing layer is getting built in the industry right now.

There’s a lot of work on post-processing of AI responses because when the models are unpredictable, how can you take the output of the models and then apply technologies to find out that something like that happened? If it hallucinates, then you suppress those responses. The entire toolkit is undergoing a lot of development. You can’t build a product in three or six months and expect it to solve real enterprise problems, which was an expectation in the market. You have to spend time on it. Our product has worked because we were on this journey for the last five years, so we didn’t start building it in November of 2022 after ChatGPT and suddenly expected it to work for our enterprises. This technology takes time to build.

Palak: As somebody who has gone on that journey from prototype to production, what advice do you have for founders that are starting similar journeys and are looking to be part of these conversations with big enterprise customers?

Arvind: A generic piece of advice for folks doing this for the first time is that building a product is always hard. There are a lot of times when you’ll feel that, oh, maybe this is a bad idea, and I should not pursue it, or it’s too difficult, or there’s a lot of other people doing it, and they may do it better than me. I constantly remind people that a startup journey is hard, and you’ll keep having these thoughts and just have to persist. The thing that you have to remember is that it’s hard for anybody to go and build a great product. It takes a lot of effort and time, and you also have that time. If you persist, you’re going to do a great job. That’s my number one advice to any startup or founder out there.

The second piece of advice concerning AI is if you start to think of AI as your product, then you will fail. We don’t see AI as fundamentally any different from other technologies we use. For example, we use infrastructure technologies from cloud providers, and that’s a big enabler for us. I have built products in the pre-cloud era, and I know how the cloud has fundamentally changed the quality of the products we built and how easy it is to build scalable products.
AI should be no more than one of the building blocks you will use, and you still have to innovate outside of the models. You have to innovate to make workflows better for people, but something beyond that, hey, yeah, I can do something better, and therefore, I’m going to build a product.

Palak: Yeah, I think that makes a lot of sense and I love the customer centricity there, really figuring out what their needs are and building a product to best serve those needs rather than taking more of a technology-first approach and taking a hammer and sort of looking for nails.

To keep on this AI trend a little bit more, I think every Fortune 500 CEO this year was asked by their board, what is your AI strategy? And we’ve seen companies spin up AI skunkworks projects and evaluate a lot of early offerings. Naturally, a lot of founders and builders want to be a part of those conversations. I’m curious how you approach that at Glean and if you have more advice for founders looking to be a part of those conversations.

Arvind: In that sense, 2023 has been an excellent year for enterprise AI startups because you have this incredible push that you’re getting from the boardrooms where CIOs and leaders of these enterprises are motivated to experiment with AI technologies and see what they can do for their businesses. We have found it very helpful because it allows us to bring out technology. There’s more acceptance and urgency for somebody to try Glean and see what it does for them.
I’ve heard from enterprises that many of these experiments have yet to work out. A lot of POCs have failed, and so my advice to founders is to have an honest conversation with your customers, with the leaders that you’re trying to sell the product to. If you create an amazing demo, which is easy to create, and sell something you don’t have, you lose the opportunity and the credibility. It’s hard to bounce back from it. Even the enterprise leaders understand that, hey, this technology is new, it’s going to take time to mature, and they’re just looking for partners to work with who they feel have integrity and who have the ability to be on this journey with them and build products over time. That’s my advice to folks: be honest and share the vision, share the roadmap, and show some promise right now, and that’s enough. You don’t need to over-promise and under-deliver.

Palak: That will become the conversation in 2024: what’s the ROI on our AI strategy? From the enterprise leader’s perspective, who had mixed results trying out a POC, do they double down and stick with the effort? How do you think about that, and how are you seeing it from the Glean perspective?

Arvind: AI is a wave that you have to ride. For example, when cloud technology was just information, there was a lot of skepticism about it. “Okay, I’m not trusting my data is going into a data center that I don’t even have a key or to lock; I’m not going to do it.” Some companies were early adopters, and some companies adopted it late. Overall, the earlier you adopt these new technologies, the better you do as a company.

The AI technologies are now bigger in terms of the transformative impact they will have on enterprises or the industry as a whole than even the cloud. It’s an undeniable trend, and the technology is powerful. You have to invest in it, and you have to, as an enterprise leader, be willing to experiment. You don’t have to commit to spending a lot of money, but you have to see what’s out there, and put in that effort to embrace and adopt this technology.

Palak: Absolutely. I’d love to double-click on that a little bit more on AI being a bigger opportunity than cloud. I’d love to get a sense of where you think that is and what are some of these amazing experiences beyond our imaginations that you think will result out of this new wave of technology?

Arvind: A decade ago, it was about $350 billion in overall technology spend. It’s probably double that now. The cloud is worth 400 billion, which is more than all the tech spending that used to happen 10 or 15 years ago. AI impacts more than software; it impacts how services are delivered in the industry. For example, think about the legal industry and the kind of services that you get from them; what impact can AI have on those services? How can it make those services better? How can it make it more automated?

If you start to think about the overall scope of it, it feels much larger. It will fundamentally change how we work, and that’s our vision at Glean. The way we look at it today, take any enterprise: you have the executives, and they get the luxury of having a personal assistant who helps do half of their work. I have that luxury, too, where I get to tell somebody to do work for me, and they look at my calendar, they look at my email, they help me process all of it, and I have that help, and I feel like it’s unfair, I have it, but nobody else in our company has it. But AI is going to change all of that.
Five years from now, each one of us, regardless of what role we play in the company, how early we are in our career, we’re going to have a really smart personal assistant that’s going to help us do most of our work, most of the work that we do manually today. That’s our vision with Glean, that’s what we’re building with the Glean assistant.
Imagine a person in your company who’s been there from day one and has read every single document that has ever been written inside this company. They’ve been part of every conversation and meeting, and they remember everything, and then they’re ready for you 24/7. Whenever you have a question, they can answer using all of their company knowledge, and that’s the capability that computers have now. Of course, computers are always good at processing large amounts of information, but now they can also understand and synthesize it.

The impact of AI is going to be a lot more than what all of us are envisioning right now. We are trying to overestimate the impact in the next year, but we underestimate the impact that the technology will have in the next 10 or 20 years.

Palak: Just so you know, Arvind, when you were starting Glean, I remember because I was working on the enterprise search product at Microsoft, and I think you were cold messaging people on LinkedIn to try out Glean, and one of the people you happened to cold message was my dad who also went to IIT. And so it was just like a funny story.

Arvind: If I’m trying to solve a problem, I want to talk to the product’s users as much as possible. So, for example, even at Glean, I was the SDR No. 1. I spoke to hundreds and hundreds of people, whoever I could find, whoever had 10 minutes to spare for me, and asked them about, hey, I’m trying to build something like this. I’m trying to actually build a search product for companies. Does it make sense? Is it going to be useful to you?

The reason it’s so powerful when you do that exercise yourself, and you don’t stop, you don’t stop after you hire a couple of people in sales, you keep going is because it generates that immense sort of conviction for you in your mission. Talking to you earlier about how often the journey’s hard, and you start to question yourself on a bad day. But if you had done that research and talked to lots and lots of users, you can always go back to that and remember that, hey, no, I’ve talked to many people. This is the product that they want. This is a problem that needs to be solved. And so that’s what I always find very helpful for me.

Palak: Arvind, you’ve been in the technology industry for a very long time and have been a part of nearly every wave of technical disruption. What have you learned from each of these waves and how have you applied those learnings to AI?

Arvind: I think each one of these big technology advances that have happened over the last three decades, we’ve seen how that fundamentally creates opportunities for new companies to come in and bring new products that are better than the products that were built before when that technology was not available to them, to anybody. And so that’s one thing that I’ve always kept in mind. Whenever a big new technology wave comes, that’s the opportunity for any company, whether you are starting a new company or if you are a startup, you have to figure out how that is going to change things, how that is going to give you opportunities to build much better products than what was possible before and then go and work on it. My approach always has been to see these big technology advances as opportunities as opposed to thinking of them as being disruptions.

Palak: I’d love to get a sense of your personal journey as you’ve gone through each of those different waves. What are some of the products that you’ve innovated on and what are you so excited about building with Glean?

Arvind: I remember the first wave; I was in my second job, and we were starting to see the impact of the Internet on the tech industry and the business sector. I got to work on building videos on the web. It was incredible to allow people to watch video content directly on their laptops, machines, and the internet. We started with videos, and they would be so tiny because there was no bandwidth available on the internet for us to provide a full-screen experience. Regardless, it was still fundamental that there was no concept before that; hey, I can watch a video when I want.

Then, the next big thing we saw was mobile with the advent of smartphones and the iPhone, and it’s fundamentally changed again. At that time, I was working at Google, and it changed our ability, for example, to personalize results, personalize Google search, and personalize maps at a new level for our users than was possible before because now we knew where they are, we knew what they’re doing. Are they traveling, or are they steady at a place? Are they in a restaurant? And you can use that context to make products so much better.

We’re in the middle of this AI trend now, and our product is a search product for companies. The AI technology, especially the large language models, has given us this opportunity to make our product 10 times better. I think back to when somebody came and asked a question to Glean, we could point them to the most relevant documents inside the company that they could go and read and get the answer to whatever questions they had. But now we can use the LLM technology to take those documents, understand them using the power of LLMs, and quickly provide AI-generated answers to our users so they can save a lot more time.
It’s been really exciting to have that opportunity to use these big technology advances and quickly incorporate them in our products.

Palak: Yeah, I think that’s one thing that’s always really impressed me about Glean. As far back as 2019 or even before ChatGPT, Glean was probably everybody’s favorite AI product. I’m curious, how has Glean’s technical roadmap evolved alongside this rapid change of innovation over the last 12 months?

Arvind: We started in 2019, and in fact, we were using large language models in our product in 2019. The large language model technology got invented in some ways at Google, and the purpose of working on this was to make search better. So when we started, we had really good LLM models in open domain that we could use and then retrain them on enterprise corpuses of our customers and make it work for them to create a really good semantic search experience. But those models were not as good as you could put them in front of the users. And all of that started to happen in the last year where finally the models are so good that you can take the output of these models and put them right in front of the users.
So this has allowed us to first completely evolve our product. What used to feel like a really good search engine, something like Google inside your company now, it’s become a lot more conversational. It’s become an assistant that you can go and talk to and do a lot of things, so not just give it complicated instruction because we can follow complicated instruction using the power of large language model technology and then solve the problem that the user wants to solve. But we can go and really parse and comprehend knowledge at a level that we couldn’t ever before and also go beyond answering questions and actually do work for you.

One thing that we realized is that the AI technology is so powerful and there’s so many internal business processes that with the power of AI, you can make them much more efficient and better, and we’re not going to be, as a company, be able to actually fix all the things like go and solve all the AI problems, but our job has now become more of that, how can you use all of your company knowledge that we have at our disposal and then give tools to our customers so that they can bring AI into every single business workflow and generate efficiencies that have never been seen before.

Palak: I think that’s one of the things that from an outsider perspective, has made Glean such a great company, just how big the vision is and how you’re starting with the customer and working backward. I’m curious how you think about internalizing some of those philosophies within your company and how that sort of evolved and how the product’s evolving to this bigger, broader platform vision that you just alluded to.

Arvind: In an enterprise business, it’s very important for us to actually spend a lot of time with prospective customers and understanding different use cases that they have that they’re trying to actually solve as a business problem. All businesses are very different in some ways, and a big part of building products that can help a wide range of customers is actually spending the time at the forefront working with our customers and really understanding the scenarios there, understanding their data, and then trying to extract common patterns and common needs and then drive our product roadmap based on that.

Our product team spends a lot of time doing this process. Every quarter we would go and look at what are the top things we are hearing from our customers, and then of course, we do have our own vision of where we think the world is headed with all these new AI technologies. And so we combine those two things to actually come up with our quarterly roadmap and then execute on it.

The reason we started, for example, working on exposing our underlying technologies as a platform that businesses can go and build applications on was exactly that, that as we talked to so many, all these enterprises and they were all actually showing us that, hey, I want actually bring efficiency in my order workflow process, and this is how it works. And somebody else comes and tells us that, hey, I want, I’m getting a lot of requests for my HR service team, and I want help people build a self-server bot for all the employees in the company whenever they have HR questions. We start to listen to all of them and you realize that, oh, we can’t do, no idea what are all the things they want to go and solve.
Our job then became that, hey, can we give them the building blocks that can make it easy for them to then take this platform and build that value that they were looking for? Do a little bit of work on top of the product and platform that we provide to solve those specific business use cases.

We started building our AI platform for that reason, because AI is so broadly applicable across so many different businesses, so many different use cases, and it became very clear to us that we need that collaboration from our customers to really get full value.

Palak: Awesome, Arvind. So I have a few rapid fire questions as we wrap up. The first is, aside from your own, what intelligent application are you most excited about and why?

Arvind: GitHub Copilot is one of the applications. I see how our developers are able to use that, and there is a clear sign, from signal from them that this is a tool that’s truly improving the productivity.

Palak: Beyond AI, what trends, technical or non-technical are you most excited about?

Arvind: I’m really excited about the nature of work, how it is evolving rapidly and quickly, distributed work, the ability for people to work from wherever they are, so technologies that is helping us become more and more effective from working from our homes. That’s the thing that I’m really excited about and we’re going to see a lot more. Hopefully we’re going to see some things with telepresence, which makes working from anywhere the same as working from office.

Palak: Yeah, that’s a good one. Arvind, thank you so much for taking the time. It was really a pleasure to have you on.

Arvind: Thank you so much for having me.

Coral: Thank you for listening to this week’s IA40 spotlight episode of Founded & Funded. We’d love to have you rate and review us wherever you listen to your podcasts. If you’re interested in learning more about Glean, visit www.Glean.com. If you’re interested in learning more about the IA40, visit www.ia40.com. Thanks again for listening and tune in a couple of weeks for our next episode of Founded & Funded.