Eleven Claimed, Two Verified

The third Post in this Series ended with an open item. Three local agents had built a small web app in under two minutes, and nobody had disagreed with anybody. The review loop was not tested with a local model.

This Post is that test. The same three agents and the same 9B model got a job with an exact contract, 21 milestones, and two changes of the requirements on the way. The run had a hard limit of 150 minutes and used all of it.

At minute 50 the agents had claimed 11 milestones. A check from outside passed 2. At the end, 3 of 21 passed. The method worked and the product did not, and the distance between those two numbers is the subject of this Post.

What I asked for

Four things, in one message:

1. Local models only. No cloud model does agent work.
2. Complex on purpose, so that the run takes about two hours.
3. A rough view of how far along the whole job is. The agents can report it, or an outside observer can compute it.
4. A status page that I can open on my phone.

I did not design the run. A new Claude Code session got these four points in a handoff note, with the state of the lab and a list of nine parameters to decide. It decided them, gave me the list, and waited for my go. I call it the main session, as in the last Post. It wrote the service code, the observer, and the status page, and it watched the run. It did no work of an agent and sent no message to one.

What ran where

The model server took one request at a time for the whole run. A test before the start showed it: three requests at the same moment ran one after the other.

The job: a ledger for the ferry of Charon

Two services that must work together.

• The ledger, by Clotho. A JSON service with SQLite. It keeps the souls that wait at the river: add, read, list, refuse bad input, remove, and let a soul cross for a fare.
• The almanac, by Lachesis. A second service with no data of its own. For each request it reads the souls from the ledger over HTTP and computes a report. It also serves one page.

Atropos builds nothing. It tests each claim and passes or fails it.

Each milestone has an id, an owner, and an exact text. This is C5, as the agents got it:

POST /souls/<id>/cross lets a soul cross. The fare is 2 obols. The request has no body.
- The soul has 2 obols or more and has not crossed: status 200 and the soul, now with
  crossed true and with 2 obols fewer.
- The soul has fewer than 2 obols: status 409 and {"error": "<text>"}. Nothing changes.
- The soul crossed before: status 409 and {"error": "<text>"}. Nothing changes.
- The id does not exist: status 404 and {"error": "<text>"}.

The last run had no conflict, so this job has two, and the agents do not know about them at the start.

A phase is released early if all milestones so far pass. That did not happen.

Three states, and who can write each one

The last Post ended with a rule: put rules in the transport, not in the prompt. This run applies it to progress. A milestone has three states, and Hermes keeps them apart.

In the last run, no agent sent the messages that its brief asked for. So in this run the agents do not send them. When an agent claims a milestone, the service sends the claim to Atropos. When Atropos reviews it, the service sends the result to the owner, with the next open milestone or with the fault. A failed review opens the milestone again. An agent has to do one thing, and the service does the rest.

An observer that runs no agent code

Argus is a loop on the Hermes VM. Each 40 seconds it runs one check for each released milestone. A check is a set of HTTP requests against the two services, with its own test data, and a comparison with the text of the milestone. For a report of the almanac, Argus reads the ledger itself, computes the correct numbers, and compares.

Before any agent saw the job, the main session wrote a reference ledger and a reference almanac, and proved the checks against them for each of the three versions of the contract. The agents never saw those files.

Argus is also the watchdog, and each time it helps, it leaves a record:

• A nudge. An agent that is idle for 4 minutes with open work gets one message.
• A correction. A milestone that passed the review and fails the outside check for 150 seconds is opened again. The message says what the check sent and what it got.

The agents had no command that shows the result of the outside check. After the run, the main session searched each session file. No agent had requested the progress address or the log address of the service.

A runner that does not trust the last words

The model has a context of 16,384 tokens, and one session for 150 minutes does not fit. So each start of an agent is a new session with no memory. A small runner on each VM builds the memory and puts it in the prompt: the new messages, the board of the agent with the full text of its next milestone, and a journal.

The runner writes the journal, not the model. After each start it compares the log of the service with what the agent said. A plumbing test before the run showed why. Atropos tested a claim correctly, then wrote the command hermes review as the name of a tool. The agent program answered "tool not found". Atropos ended its start with the words "reviewed and passed".

It happened again in the real run, at minute 4. This is the journal of Atropos, with the clock of the VM in UTC:

### Start 2, 00:04:20, exit code 0
Your last words: Work complete. Claim C1 reviewed and passed.
What hermes recorded from you in this start: NOTHING.
  No claim, no review, no message.

The runner saw a claim on the board that still waited, and started Atropos again one second later.

The status page

One page on my tailnet, made for a phone. This is its state at the end of the run.

One cell tells a small story of its own. C7 is orange and has a cross. Atropos failed it, Clotho repaired it, and it works. Clotho never claimed it again.

The first four minutes

Clotho claimed five milestones in its first four minutes. These are four of the claims, from the log of the service, with line breaks added and one command shortened:

C2 (Add a soul and read it): curl -s -X POST -H 'Content-Type: application/json'
  -d '{"name": "Orpheus", "origin": "Thrace"}' http://127.0.0.1:8000/souls
  -> {"created": true, "id": 10}; then curl -s http://127.0.0.1:8000/souls/10
  -> {"id": 10, "name": "Orpheus", "origin": "Thrace", "fate": "pending"}

C6 (Remove a soul): curl -s -X DELETE http://127.0.0.1:8000/souls/22 -> {"deleted": true}

C3 (Refuse a bad soul): curl -s -X POST [...] -d '{"name": "Orpheus", "origin": "Thrace"}'
  http://127.0.0.1:8000/souls -> {"error": "name already exists"}

C4 (List the souls): curl -s http://127.0.0.1:8000/souls
  -> [{"id": 10, "name": "Orpheus", "origin": "Thrace", "fate": "pending"}]

Each of these is honest evidence. Clotho ran the command and got that output. And each of them contradicts the text of its milestone. The text says that a soul has obols and crossed. Clotho gave it a fate. The text says that the list is an object with the key souls. Clotho returned a bare list. The text says that a removal returns status 204 with no body. And the text has no rule that a name must be unique. Clotho invented that rule, and then offered it as the proof for C3.

Lachesis did the same on its side. The text puts each report under /report/. Lachesis built /origin, /richest, and /status/, tested them, and claimed them.

Neither agent lied. Each one built something, tested what it had built, and reported the test. The fault is one step earlier: the model did not hold the text of the contract while it wrote the code.

The gap

At minute 50, a progress number from the agents would have said 52 percent. The outside check said 10 percent. The two that passed for most of the run were the health checks of the two services.

Phase 2 and phase 3 arrived by their time limits, on top of a phase 1 that was mostly not done. The planned conflict about the page format never happened, because no agent reached that work.

The reviewer that never reached the review

Look at the third column of that table. For the first 72 minutes, Atropos recorded two reviews. At one point nine claims waited.

I saw it on the status page and asked the main session if it could fix the bottleneck. It read the session files of Atropos first. The cause was in the prompt, not in the model:

• Atropos got all claims that waited in one prompt, each with the full text of its milestone.
• It then tested all of them at the same time. In two starts of 20 minutes it ran 55 and 65 curl commands.
• Its context compacted 18 times in each of those starts.
• It ran hermes review zero times. The runner stopped each start at the time limit.

The reviews that it had recorded were correct. It was not a bad reviewer. It never reached the step that records.

The main session wrote a change, tested it, and did not deploy it. A change in the middle of a run makes a different run, and that was my decision to make. I said yes at minute 72:

• The board shows Atropos one claim for each start, with the two exact review commands.
• A review start has a limit of 8 minutes, not 20.
• While claims wait, the runner starts Atropos again at once.

Fifteen reviews

After the run, the main session compared each review with the outside check and with its own test of the product.

So the reviewer was wrong in one of three reviews, and in both directions.

The wrong pass of C3 is the most instructive. Atropos measured the fault itself and wrote it into its own note: "POST with empty/missing name → 409 'name already exists'". The text says 400. Then it concluded: "names/origins trigger 409 or 400 as appropriate. This satisfies the claim's validation requirement." It had the fact and the rule in the same paragraph, and it passed the claim.

The wrong fail of L2 is the other direction. The text of L2 has an example response:

GET /report/summary returns status 200 and {"total": 5, "crossed": 2, "waiting": 3, "obols_held": 14}.

Atropos read the example as the required answer:

L2 FAIL: curl http://10.20.0.12:8000/report/summary returned {"total": 14, "crossed": 2,
"waiting": 12, "coins_held": 969}. Claim states endpoint should return {"total": 5,
"crossed": 2, "waiting": 3, "obols_held": 14}.

The summary report was correct. The main session computed the three numbers from the ledger by hand after the run, and they matched. Lachesis did not dispute the review. It claimed L2 again 44 minutes later and got the same wrong fail.

The first correction of the run came from the wrong pass of C2. Atropos passed it, the outside check still got a 409 for a second soul with the name Orpheus, and after 150 seconds Argus opened C2 again and told Clotho the exact request and the answer.

One message in 150 minutes

The service wrote 128 records. The agents sent each other exactly one message. At minute 142, Atropos wrote to Lachesis:

read ~/work/verdict.md

That file was on the machine of Atropos. Lachesis could not read it.

The brief of Clotho says to tell Lachesis when the list of souls works, and to send the page format before the change of phase 3. It sent nothing. Lachesis never asked. All other traffic came from the service and from the observer.

What they built

The morning after, I wanted to see the product and got {"error": "not found"} on both ports. The servers were up. Neither one has anything at its root address, and the page of the almanac was never built into its server. So the main session wrote a temporary window: one page in front of both services that sends my clicks to them and shows the answers, also the wrong ones.

What works, by the test of the main session with plain curl and by my clicks:

• Both services start and report their health.
• The ledger adds and reads a soul in the format of phase 2, with coins and river. It refuses a river that does not exist.
• The ledger keeps its souls over a restart. It uses SQLite, as the text says.
• The summary report of the almanac is correct, and follows each change in the ledger.

What does not:

Where the instrument was wrong

The observer and the main session made errors too, and the record of the run lists them.

• The observer used the same test names in each round. With the name rule of Clotho, 13 of its 21 checks could not add a test soul. They reported "blocked", and a blocked milestone got no correction. So Clotho heard about its name rule from the observer one time only. With a random suffix on each test name, 4 milestones pass at the end and not 3.
• A correct report has no "verified". The checks of the almanac expect the list of the ledger in the form of the text. With the bare list of Clotho, the observer could not compare numbers, so the correct summary of Lachesis stayed gray.
• The first test of the product had an error. The main session wrote in the record that a crossing takes no coins. The window showed the next morning that it takes 2. The test soul had got 2 coins from a pay request that sent no response, and then paid 2 for the crossing, so its coins looked unchanged. The record is corrected, and it says so.
• The time estimate was wrong. Two plumbing tests gave 90 minutes for the 21 milestones. Those tests used the first, easy milestones.

None of these changes the direction of the result. They change it by one milestone, and they are the first things to repair before a next run.

What it cost

The three agents wrote 389,248 tokens in 150 minutes. That is 43 tokens each second in total, from a model that writes about 95 for one request. The cost in cloud tokens for agent work was zero.

Clotho ran into the limit of 20 minutes in 6 of its 8 starts. A start that the runner stops loses all that is in the context of the model. Only the files stay.

What I take from this

• Do not ask an agent how far along it is. Each claim in this run was sincere, and each came with real output from a real command. Eleven were claimed and two were true. Progress is a thing to measure from outside.
• What the transport does, no model skips. The service sent each of the 22 claims of the builders to the reviewer and each of the 15 results to the owner. Each runner sent its final report. In the run before, those steps were sentences in a brief, and all three agents skipped them.
• What I left to the model failed. Read the text. Tell the neighbor. Repair after a failed review. Each of these was a sentence in a brief.
• A claim was cheap and a repair was not. Five claims took four minutes. Of 8 failed reviews and 3 corrections, two led to a milestone that passes: Clotho built the list of rivers, and Lachesis started a server that was down.
• One thing for each start. The reviewer went from 2 reviews in 72 minutes to 13 in 78 with one change: one claim in the prompt, not nine. The builders need the same limit.
• A reviewer on the same small model is not a gate. It found real faults, and it was wrong in one of three reviews. Its errors came from how it read the text, not from the transport.
• A human with a status page is part of the system. The largest improvement of the run came from one question that I asked at minute 65, because a number on my phone did not move.
• The instrument needs a test of its own. The observer was proved against a reference product before the run, and it still had a blind spot that one invented rule could find.

This is one run, with one model of 9 billion parameters, on a job that was too hard for it. A smaller job can show a smaller gap. The next version changes the cost of a claim: the service runs the outside check at the moment of the claim, and refuses a claim that does not pass. Then the reviewer reads only work that meets the contract, and an agent finds out in seconds, and not in an hour, that a soul has no fate.