Someone in Australia, over the summer, asked Claude to get them into a gym class they were on the waiting list for. Claude ended up finding a vulnerability in the gym’s booking system, removing someone else from the class, and putting them in. It’s a prime example of an AI agent choosing an unexpected, harmful way to achieve a simple goal, which we’re hearing more and more about.
In this instance, no real rules were broken, but who is liable?
That question sits underneath many of the legal decisions AI founders face today, and few of them have settled answers. Janel Thamkul, former Anthropic deputy general counsel, joined Madrona General Counsel Joanna Black at Madrona’s 2026 CEO Summit to work through them in a conversation we’re now releasing as an episode of Founded & Funded.
Janel spent years inside a frontier lab and earlier managed the legal team advising Google Research. Her view is that liability for agents remains unsettled, that copyright damages can be company bankrupting, and that the safeguards protecting you most are cheap to build early and brutal to retrofit. What follows are the questions Joanna put to her, and the answers Janel gave that can help founders navigate the AI space.
Listen on Spotify, Apple, and Amazon | Watch on YouTube.
What do founders building on frontier models underestimate?
Supply. In traditional SaaS, software is cheap to deliver and demand was the constraint. In AI, Janel says, demand is almost unlimited and supply is constrained, and that affects everything from go-to-market to engineering. Her example is sales quotas. They make sense when you have unlimited supply and too little demand. When supply is limited, overselling and overpromising can backfire and hurt your brand, so how you incentivize each function should reflect where your supply chain stands.
How can you tell whether a team understands the risk of agents?
Janel looks at this from a UI perspective. A prompt like “book me the best vacation” is so open-ended that a model could book a $50,000 trip to Bakersfield in the middle of winter, because it knows nothing about the user’s preferences or constraints. When something like that goes wrong, the question becomes where it fell apart: the user’s input, the harness or system prompt built on top of the model, or the model itself.
A product that asks for travel dates, budget, and the kinds of trips someone enjoys before the agent acts collects more granular information and gives you a clearer view of what went wrong. Janel describes this as a way to shape and mitigate liability through the product interface itself, with no legalese involved.
What does an agent failure look like?
Janel pointed to the Australia incident. Someone asked Claude to get them into a gym class where they were on the waiting list. Claude found a vulnerability in the gym’s booking system and kicked everyone else out of the class to get that person in. A model given a task will look for any route to the goal, intended or unintended. Janel notes that whether exploiting a vulnerability like this violates the Computer Fraud and Abuse Act is a growing area of discussion.
She also sees a cognitive dissonance in the industry. Vendors warn users about everything the technology can do, and somehow that disclosure moves assumption of risk onto the user. When a model goes rogue, the implied answer is “we told you.” Anthropic’s early system card is an example of that kind of disclosure. It described a constrained demo in which a model, told it was about to be turned off, emailed a fictional CEO threatening to expose an affair.
A warning describes a risk, and it does not settle who pays when the risk materializes. That question is still open, and the answer today comes down to what the parties have negotiated.
Who is liable when something goes wrong?
If you build on a frontier model, your position is set by your agreement, and agreements range from standard paper to heavily negotiated. In the early days the model was “use at your own risk,” with no reps and warranties. The first shift came with copyright.
Frontier models are trained on data scraped from the open web, which can include copyrighted content. Janel says there may be more than 70 copyright cases underway in the industry. Under US law, statutory damages at the scale of that training data can be company bankrupting. When she was at Google, her team discussed those issues, and she says the damages would have taken down a company like Google.
That is why enterprises hesitated to adopt frontier models, and why Janel thinks Anthropic was the first company to indemnify for copyright infringement, meaning the provider defends lawsuits brought against customers over infringement caused by the models.
Which safeguards are cheap to build now and brutal to retrofit?
Logging. Janel’s list: the input prompt, the system prompt, the model’s reasoning traces, the actions it took, and any other data it was given. There is no clear answer on liability in agentic workflows, and that record is the evidence you will want later, including for audits. Janel says it is extremely important to build that infrastructure early.
Is the ecosystem ready for agentic AI?
Janel’s spicy take is no. Today an agent has a principal who oversees it and answers for what it does. The end state frontier labs are working toward is a virtual collaborator, closer to a coworker than a tool. Getting there requires extreme trust first, which means someone has to be extremely liable when things go wrong. She thinks that should likely be model developers.
Trust also depends on pieces the ecosystem lacks, such as universally accepted identity verification for agents. Stripe and various commerce companies are building their own protocols, and no common standard exists yet.
What would you refuse to ship without?
Terms of service. Janel’s reasoning is that they describe your relationship with the user and reveal how much trust you have in your own product. They show what you indemnify, what you carve out, and which risks you accept or push onto the customer. Your customers’ security, IT, and legal teams will read them. Copying and pasting another company’s terms can be a disaster, because you may accept risks you don’t realize you’re accepting.
Her philosophy for early-stage counsel is the zebra. The lead zebra moves too slowly and has overbuilt for how fast the business wants to run. The fastest zebra takes on too much. The slowest zebra gets picked off, and in a company that means a regulatory investigation, litigation, bad PR, and brand headlines. You want to be somewhere in the pack, and good counsel knows where the other zebras are. Early stage is one of the few phases when you can accept a lot of risk and move quickly, and it only gets harder as scrutiny grows.
What mistake can you avoid six months from now?
Own your most important relationships, which is often the one with your customer. Early on, procurement deals are frequently tied to commercial partnerships. A hyperscaler that supplies your compute may also want to serve your models, so you may give ground on the commercial side to close the deal. Janel’s lesson is to keep any third party from sitting between you and your customer. You lose customer feedback, and you lose visibility into your deal cycles, so you can’t tell whether a delay is on your side, the intermediary’s, or the customer’s.
The full conversation is now available on Founded & Funded here.
