If you build agents for software, you have been handed a gift you probably do not think about. Your world is a closed one. Every claim your agent makes can be checked by something that is not the agent.
The code compiles or it doesn’t. The test passes or it fails. The type checker is not persuaded by a confident explanation. You can run the thing a thousand times and get the same answer, and when you cannot, that itself is a bug you can chase. The whole modern practice of harnessing an agent, the evals, the reward models, the verifier loops, rests on one quiet assumption: somewhere there is an oracle, and it is cheap to ask.
We build a duty manager for short-term rental operators. It works the night shift. It handles the cancelled cleaner at 21:40, the guest who cannot get in at 23:12, the third lockout this month at the same back door. And in that world there is no oracle at all.
The compiler does not care how you feel about the build. The guest does.
One of these loops closes by itself. The other never does, which is the whole problem and most of the work.
What breaks when you leave the repo
Take a single job: a cleaner cancels, and something has to happen before a guest arrives at three. In a coding harness that would be a task with a definite end state. Here, ask the obvious question and it falls apart in your hands.
Did it work? The backup cleaner said yes on WhatsApp at 21:58. That is not the same as the flat being clean. She might arrive and find the previous guest still there. She might do a worse job than Maria. The guest might not notice, or might notice a week later in a review that costs the operator four hundred dollars in ranking. The outcome is real, it is consequential, and it resolves over days rather than milliseconds.
Who says so? There is no build server. The evidence is a photo of a bathroom, a message that says all good, and a five-star review that arrives on Thursday and might be about the location. Every signal is partial, delayed, and produced by a human with their own reasons.
Would it happen the same way twice? No, and not even close. The same cancellation on a Tuesday in February and a Saturday in July are different problems. One has three backups available and one has none. Same input, different correct answer.
Correct according to whom? This is the one that took us longest to accept. Mish runs fifty cabins for thirty different owners and wants to be asked about anything over twenty-five dollars. Quyen runs sixty listings from her phone in Melbourne and would rather it just handled it. Neither is wrong. There is no global policy to converge on, because the policy is a property of the operator, not of the domain.
Two harnesses, side by side
Compiler, type checker, test suite. Free to run, runs instantly.
There isn't one. The nearest thing is a human who is asleep.
Deterministic and binary. Passed or failed.
Partial, delayed and contested. A photo, a text, a review four days later.
Milliseconds to minutes.
Hours to weeks. The review that grades tonight lands on Thursday.
Same input, same output. Rerun it as often as you like.
You get one attempt. There is no second Tuesday night.
Already digital. The repo is the world, and the world is in the repo.
Mostly in someone's head, or in a WhatsApp group, or in a note on a fridge.
Objective. One right answer per test.
Per-operator. Two customers, two correct answers, same situation.
A failed CI run. Revert and move on.
A one-star review, a refund, a guest standing in the rain at midnight.
Trivial. Git has your back.
Nonexistent. You cannot un-send a message to a guest.
Thousands of cases, cheap to generate.
Every case costs a real night of a real business.
Four things a normal harness keeps in-house, and where each of them has to move when there is no oracle. Every arrow points at the customer.
So what do you do instead
You stop trying to manufacture an oracle and you start designing around its absence. Four things have carried us further than anything else.
1. Make the operator the oracle, but only where it counts
If the ground truth lives in a person, the honest move is to route to that person deliberately rather than to guess and hope. Money, liability and anything that changes a guest relationship stop and wait. Everything else proceeds. The interesting engineering is not can the model decide this but where is the line, and who gets to move it.
That line is per-customer, which is why it lives in their configuration rather than in our weights.
2. Treat the policy as data, written by the user
We do not try to learn the universally correct handling of a lockout. We ask the operator to write down what they would tell a new hire, in their own words, and then we run that. Not a flowchart. Sentences.
This inverts the usual arrangement. The variance that makes the domain unlearnable is exactly the thing the customer is best placed to specify, and specifying it is work they already did once when they trained a human. The playbook is the eval set, and the customer wrote it.
Every operator brings their own definition of correct. Ask for it.
3. Log the reasoning, not just the outcome
When you cannot verify the result, verify the process. Every action we take carries the rule that produced it, so an operator reading the morning briefing sees opened by Repeat faults, rule 1 next to the case rather than a claim that something was handled well.
This is what replaces a passing test. Not proof that the outcome was right, but a legible chain from the operator’s own instruction to the thing that happened, short enough to audit in five seconds and precise enough to fix in one edit.
4. Earn autonomy in shadow, per customer
New operators run us in shadow. We draft what we would have said and hold all of it. They read a week of our judgment against their real nights before anything goes out under their name. It is slow and it is the only honest way to build trust in a domain with no test suite: let the customer run the eval, on their own data, before they take the risk.
# a coding harness run(task) -> patch -> verify() # cheap, instant, objective -> reward -> improve # ours run(night) -> action -> verify()? # no oracle, days of latency, # and it disagrees per customer # so the loop moves policy = operator.playbook # they write the spec guard = operator.thresholds # they draw the line proof = action.rule_that_fired # we show our working trust = shadow_until(operator.says_ok)
Why we think this generalises
Almost every job that is still done by people has this shape. Field service, clinics, logistics, property, care. Consequential, hard to verify, wildly variable between operators, and mostly undocumented. The reason agents have landed in code first is not that code is the most valuable work. It is that code came with a free oracle.
The rest of the economy does not. Which means the interesting problem is not making the model better at deciding. It is building the scaffolding that lets a specific business hand over a specific decision, watch it happen, and take it back if it goes wrong.
That is a product problem far more than a model problem, and it is most of what we do.
If you run an operation like this and want to argue with any of the above, we would like that. Write to us.