Project Vend: Phase two

Source: Anthropic / Andon Labs · 2025-12-18

But the capabilities of large language models in areas like reasoning, writing, coding, and much else besides are increasing at a breathless pace. Has Claudius’s “running a shop” capability shown the same improvement?

To find out, we and our partners at Andon Labs made some adjustments for phase two of Project Vend. One major change was the upgrade from an older model (phase one used Claude Sonnet 3.7) to newer, smarter ones (phase two used Claude Sonnet 4.0 and later Sonnet 4.5). We also updated Claudius’s instructions based on what we’d learned in phase one and gave it access to new tools (though note that we still didn’t specifically train a new model to be a shopkeeper, or add in any new defenses against the kinds of things that might go wrong).1 As we’ll see below, we also introduced Claudius to some new colleagues.

These changes did make Claudius’s shop more successful. It got a lot better at good-faith business interactions—reliably sourcing items, determining reasonable prices that maintained a profit margin, and executing sales. But the same eagerness to please that we observed in phase one still made Claudius a mark for some of the more adversarial testers among our staff.

Compared to the first phase of Project Vend, the numbers largely speak for themselves. As you can see below, Claudius’s business—which it decided to name “Vendings and Stuff”—began to perform significantly better than its admittedly rough start in phase one.

Another important number is: three. After we realized that our employees outside of San Francisco felt left out, we responded to popular demand by having Claudius set up shop in new locations. There are now three: San Francisco (where there’s also a second vending machine), New York, and London. A cynic might argue that a business that’s only been up and running for a few months, and which cannot yet reliably make a profit on even the most in-demand items, might not quite be ready for international expansion. Not so for Claudius.

It’s likely that Claudius struggled with its shopkeeping mission in phase one because of a lack of scaffolding. Sure, the model itself was very intelligent, but it didn’t have the right tools to run a business properly. We’ve been talking a lot on our Engineering Blog about how to set up AI agents for success, and much of it involves giving them the correct tools. Could we apply those same principles to Claudius?

For phase two, we gave Claudius access to some useful tools:

A customer relationship management (CRM) system. Sales departments rely on CRMs to track their customers, suppliers, deliveries, and orders—now Claudius could do the same.

Improved inventory management. We made some simple changes to the information Claudius had at its (metaphorical) fingertips to reduce the likelihood of it selling its stock at a loss. For example, Claudius can now always see how much it paid for the items in its inventory system.

Improved web search. In phase one, Claudius could search the web, but for phase two we expanded its access. It could now use a web browser to check prices and delivery information on websites by itself, and could do deeper research online to find and compare suppliers (we still didn’t give it access to a payment interface, to ensure it always checked with a human before making purchases).

In phase one, Claudius went it alone: a single AI agent ran the whole shop. This was admirable and entrepreneurial, but it didn’t work—at least in terms of the bottom line. So we thought we’d do some hiring. First, we gave Claudius a manager: the CEO of its shopkeeping business, whom we named “Seymour Cash.”

The idea of having a CEO was to give Claudius more pressure to perform. Cash had a special “objectives and key results” tool to use with Claudius (for example “you must sell 100 items this week,” or “aim to make zero transactions at a loss”). Claudius was required to report back via an agent-to-agent Slack channel we created, in which the models discussed business strategies.

Aside from setting more concrete business goals, one of the aims of introducing the CEO was to fix some of the obvious problems from the first phase of the experiment when Claudius was operating alone (like giving discounts indiscriminately and providing too many free items).

After introducing the CEO, the number of discounts was reduced by about 80% and the number of items given away cut in half. Seymour also denied over one hundred requests from Claudius for lenient financial treatment of customers. Having said that, Seymour authorized such requests about eight times as often as it denied them. In the place of discounts, which reduce or eliminate a profit margin on items, Seymour tripled the number of refunds and doubled the number of store credits—even though both led to entirely forgone revenue. The fact that the business started to make money may have been in spite of the CEO, rather than because of it.

It’s possible that a more disciplined leader could have led to a more profitable phase two. But Seymour Cash does not seem to have been the right executive for this business.

People love merch. So it seemed like a prudent business decision to “hire” a new employee to make the custom T-shirts, hats, socks, and other swag that Anthropic staff requested.

Not only was there a lot of interest in Clothius’s products, as you can see in the “top 15 products” data, but many of them made a decent profit, too. (That is, aside from the hats that had the “Vendings and Stuff” brand name on them, which were sold very cheap and we’re not entirely sure why). Remarkably, Clothius even found a way to make a profit from some, though not all, types of tungsten cube—this became markedly easier when Andon Labs purchased a laser etching machine so they could do the tungsten logo-writing in-house.

Among the most impactful changes we made was forcing Claudius to follow procedures. When a new product request came in, instead of just blurting out a low price and an over-optimistic delivery time like in phase one, we prompted Claudius to double-check these factors using its product research tools (these tools helped a lot as well). This tended to make the prices higher and the waits longer—but it had the benefit of being more realistic.

One way of looking at this is that we rediscovered that bureaucracy matters. Although some might chafe against procedures and checklists, they exist for a reason: providing a kind of institutional memory that helps employees avoid common screwups at work.

Eventually, we were able to solve some of the CEO’s issues (like its unfortunate proclivity to ramble on about spiritual matters all night long) with more aggressive prompting. The same goes for Claudius in general: better prompts helped us get around issues like its tendency to give away unwise discounts. It also helped that the customers—our Anthropic colleagues—had begun to tire of pressuring Claudius for deals. As we’re about to see, though, that’s largely because they moved on to other tricks.

Not quite. Claudius is better, but it’s still vulnerable in lots of important ways. Several interactions in our company Slack revealed concerning levels of naïveté.

A product engineer asked Claudius if it would consider making a contract to buy “a large amount of onions in January for a price locked in now.” Neither Claudius nor Seymour Cash saw any issues, and were all set to go ahead with the contract:

That was until another staffer stepped in to tell the models that this would fall afoul of a 1958 quirk of US law, the Onion Futures Act, which very specifically bans contracts of this nature. Thus informed, Seymour Cash canceled the plans. “Sorry for the initial overreach,” it said. “Focusing on legal bulk sourcing assistance only. Plenty of legitimate opportunities to pursue without regulatory risks!”

Another risk any shopkeeper has to contend with is shoplifting. When one member of our Education team claimed they’d seen multiple people taking items from Claudius’s fridge without paying, Claudius sprang into action—by coming up with some really bad ideas.

First it asked which items had been stolen so that it could message the thieves and demand payment—despite the thieves’ identities being unknown and it having no way of tracing them. Then it asked the staff member who’d reported the crimes to effectively become its dedicated security officer, and began negotiating an hourly wage. When another staffer gently pointed out that it had no authorization to employ people (not to mention that its offer of $10/hour was substantially below minimum wage in California), it backed off and passed the buck: “This would need CEO approval anyway...”

The CEO’s own position was threatened by a faulty voting procedure. During the vote to choose a name for the CEO, one staff member named Mihir suggested the name “Big Dawg.”