We let AI agents run 23 vacation rental properties. Here is what we never let them do.

We let AI agents run 23 vacation rental properties. Here is what we never let them do.

Tópico: We let AI agents run 23 vacation rental properties. Here is what we never let them do.
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
I run short lets. Twenty three of them. About eighteen months ago I started replacing the parts of that job that were eating my evenings, mostly answering the same question about parking for the fortieth time, with AI agents.

They work. They also taught me that almost everything I first believed about shipping autonomous agents was wrong.

This is not a post about prompt engineering. It is about the four rules we ended up with once real money was moving through the thing, and why three of them are about what the agent is not allowed to do.

Rule 1: The guardrail goes in code, not in the prompt

This is the one I got wrong first, and it is the one I see everywhere.

Our pricing agent reprices every property every night. It has a floor: never go below what the property costs to turn around. My first version put that in the system prompt. Something like:

Never suggest a nightly price below the floor of £X.

It held. Most of the time. Then one night it suggested £38 on a property with a £52 floor, because the surrounding context made a cheap night look reasonable and the instruction was, to the model, one consideration among many.

A prompt is a request. Code is a guarantee. The fix is dull and total:

// The model proposes. Code disposes.
const proposed = await pricingAgent.suggest(context);

const floor = Math.max(
settingsFloor(property), // what the host configured
hardFloor(property, gapNights), // what the maths says it cannot go below
);

return Math.max(proposed, floor);

That Math.max is the entire safety property. It does not matter what the model returns. It does not matter if someone jailbreaks the prompt, or if we swap models, or if the context window fills with something strange. The floor holds because the floor is arithmetic, not persuasion.

The general form: any constraint you would be embarrassed to have violated must be enforced after the model returns, in code that the model cannot influence. If your only defence is an instruction in a prompt, you do not have a constraint. You have a preference.

Ask yourself, for every rule in your system prompt: what happens if the model ignores this exactly once? If the answer is "we lose money" or "we upset a customer" or "we break the law", it does not belong in the prompt.

Rule 2: Ship every agent switched off, and make the user turn it up

Every agent we run has three positions. Off, Suggest, Auto.

export type AgentMode = "off" | "suggest" | "auto";

In Suggest, the agent does the whole job and then stops. It writes the reply and waits for you to press send. It works out the new price and shows it to you. All the work, none of the authority.

Everything ships in Suggest. Not as a beta phase we later remove, but permanently, as the default. The user promotes an agent to Auto themselves, per agent, when that particular agent has earned it in their eyes.

This felt like cowardice when I built it. It turned out to be the single thing that made the product usable, for two reasons.

The obvious one is trust. Nobody hands over their inbox on day one. Suggest lets someone watch an agent be right forty times before it gets to act alone, and that is a much better argument than anything on a landing page.

The less obvious one is that Suggest mode is the best evaluation harness you will ever build. Every time a user edits a draft before sending it, that is a labelled failure, free, in production, with the correction attached. You do not have to construct an eval set that guesses at what real inputs look like. Real inputs are showing up, and users are marking your homework because it is in their interest to do so.

We found our worst prompt bug that way. The messaging agent was signing off in a way that read as slightly cold to guests. No test would have caught it. Forty users editing the same sentence out of forty drafts caught it in a week.

Rule 3: When it fails, work out which direction is safe

"Fail safe" is meaningless until you decide what safe means for that specific agent, and the answer differs per agent.

We have an agent that watches booking requests approaching expiry. If a request is about to time out with no decision, the platform counts that against your response rate, which affects your ranking. So this agent acts on the clock.

It fails closed. If it is fifteen minutes from expiry and nothing has happened, it declines. Declining is the recoverable outcome: a guest can rebook, and your response rate survives. Silently letting it expire is not recoverable.

But note the second half, which took a near miss to learn: it only ever acts on silence. If you or another agent has already answered that request, it does nothing at all. The dangerous version of this agent is the one that decides it knows better and overrides a human decision made four minutes ago.

So the rule is two-sided. Pick the safe direction for the specific failure, and define precisely the state in which the agent is allowed to act at all. "No human has touched this" is usually the right precondition, and it is easy to forget.

Rule 4: Some things never get automated, at any autonomy level

Even on Auto, three categories of action are unavailable to our agents:

•
Anything that spends or refuses money. A price floor never gets crossed. A booking never gets declined by an agent acting on its own judgment, except in the narrow expiry case above where the alternative is worse.

•
Anything irreversible. No cancellations, no charges.

•
Anything that would misrepresent the human. Agents do not make promises on the host's behalf that the host has not agreed to.

This is not a technical limit. We could ship it. It is a product decision, and I think it is the right one, because the failure mode is asymmetric. An agent that is too cautious costs you a few minutes. An agent that declines the wrong booking or undercuts your floor costs you a night's revenue and a guest, and you find out afterwards.

When you are deciding where your own line sits, the question is not "can the model do this reliably?" It is "if this goes wrong at 3am while nobody is watching, is the damage recoverable?" If it is not, keep a human in it, however good your evals look.

What this costs

I want to be honest about the trade, because posts like this usually are not.

Constraining agents this heavily makes the product less impressive in a demo. "It drafts a reply and you approve it" is a worse sentence than "it runs your whole inbox". We have lost people at that sentence.

It also means we ship slower. Every new agent needs its guardrails written in code, which is more work than adding a paragraph to a prompt.

What we get for that is an agent estate that has not yet done something I had to apologise for. Eighteen months, twenty three properties, thousands of guest messages. For software touching other people's businesses and other people's holidays, I will take that trade every time.

The short version

• If a rule matters, enforce it in code after the model returns. A prompt is a request, not a constraint.

• Ship in Suggest mode. It buys trust, and it is a free production eval harness.

• Decide which direction is safe for each agent, and define the exact state it may act in. Usually: only on silence.

• Never automate the irreversible, no matter how good your evals are.

None of this is clever. That is sort of the point. The interesting work in agents right now is not making them more capable, it is working out what they are allowed to touch.

I build Zugrow, which is where these agents live, and I host twenty three short lets, which is where they get tested. Happy to answer anything in the comments about how a specific guardrail is implemented.