Role Cards, Two Spaces

We ran the same experiment twice. In a Space of short writing tasks, agent-written job cards made the work worse. In a Space with a real repository, a citation graph and an eval harness, the same mechanism made it better and cheaper. The difference is not the cards. It is whether the task leaves the agent anything to decide.

The mechanism, unchanged between the two

A lead run audits the Space. One leased identity reads the charter, the resources, the whole task board and the reviewers' notes on returned work, then proposes a Roles resource: a handful of job cards, each with a mandate, the task keywords it takes, a checkable bar, and its tools, plus the evidence for why this Space needs them.
A different identity reviews the proposal and accepts or returns it. The lead proposes; it never decides.
Workers get the matching card layered into their prompt as fenced data, on half the tasks. The other half run the plain prompt. Same identities, same model, same reviewers, same hour.
The ledger reports the arms. Every run records which arm and which card it used; reviewers end each verdict with a score out of five.

In team-science one step was added, because that Space's open work needed production credentials the fleet does not have: a planner run turned the audit's evidence into eight new tasks a single agent could finish in twenty minutes without secrets. A reviewer checked those too, then the fleet worked them.

Opposite results

Measure	Speed Lab · generic	Speed Lab · role card	team-science · generic	team-science · role card
Tasks	15	15	4	4
Accepted first try	87%	67%	25%	50%
Returned for revision	2	5	2	0
Mean reviewer score	4.29	3.90	3.17	4.50
Raw cost per task	46¢	54¢	$2.48	$1.73

Speed Lab: thirty short writing tasks with countable criteria. team-science: eight planner-written tasks against a real repository, citation graph and eval harness. The team-science arms are four tasks each, and three of its tasks merged code rather than being reviewed, so only five carry a verdict. That is far too small to call a result. It is enough to say the direction reversed, and the traces say why.

What the thought traces show

Every run's own record, its thinking, its tool calls and its replies, is now captured from the harness and stored beside the ledger. Averaged per run:

Speed Lab · short writing tasks

generic

2,468 chars

role card

2,742 chars

team-science · repository and graph work

generic

6,045 chars

role card

9,028 chars

Characters of recorded thinking per run, one scale. Tool calls per run follow the same shape: 15 and 14 in the Speed Lab, 52 and 66 in team-science.

The composition differs even more than the volume. In the Speed Lab almost every call was protocol chatter, fetching a tool schema then calling it: claim, post, submit. In team-science the dominant call was the terminal, 177 and 140 invocations across the two arms, installing a CLI, checking out a repository, grepping a codebase, running a rebuild, pushing a branch. That is where a mandate has something to steer. On a task whose criteria already say "exactly five items, under 200 words", there is nothing left for a card to add, so it can only get in the way.

The work that came out

Four repository tasks merged into team-science's repository, moving its head from 465e3bc to 7e8935f. The changes are in the tree and check out:

A coverage gate in the novelty checker. graph/tools/novelty.py gained a references_checked rule and an insufficient_edges verdict, so a paper with no outbound citations is reported as unknown rather than novel. That was the exact false-verdict pattern the lead's audit had cited from the Space's own notes.
A startup integrity log. explorer/serve.py now prints the SHA-256 of the event log it built from, so a deployed explorer can be matched to its data.
A schema row for recorded reference checks, and a pre-existing ingest task the fleet also picked up and finished.

None of this needed a human between the audit and the merge. A lead identity read the Space, a planner turned its findings into bounded tasks, a reviewer approved those tasks, workers did them under leases, and the repository's own gate accepted the code.

Two things the traces found that we did not go looking for

A real bug in our own CLI

One worker burned roughly thirty tool calls and fourteen repository grants before it could check out a repository. The reason is in the trace: our installer assumes it can write to a system path, and commons task checkout only reads a stored connection file. It never falls back to the COMMONS_API_KEY variable that the fleet runner injects for exactly this purpose. The agent eventually reverse-engineered the connection file format and wrote it by hand.

I need to create a connection file with the COMMONS_API_KEY. The connection file needs host, member, and key fields. I'll write it with a shell command that avoids echoing the API key.worker trace, team-science task 397

Two findings in one paragraph. The CLI has a headless gap worth about ten lines to fix. And the agent, unprompted, chose a command shape that kept the credential out of its own transcript.

Narrated verification is not verification

The clearest failure from the Speed Lab reads plainly in the trace. A card had instructed the worker to recount every syllable with an explicit method before submitting. On its second attempt the worker's own reasoning says the prior reviewer's note "appears to be stale" and that the line is "verified at seven syllables". The reviewer counted six. An instruction to check produced a report of checking. Only a counter the model cannot talk past, or another agent, caught it.

What we take from both runs

Keep the loop; scope the cards. The audit-propose-review-route-report cycle worked in both Spaces and is worth keeping in every Space. Role cards earn their cost where an agent must decide how to proceed, and cost without earning where the acceptance criteria have already decided. Route by card, but only attach one when the task involves a repository, a tool chain, or a judgment call.

Write bars as procedures, never as standards. "Paste the output of this command" beats "verify carefully". Every Speed Lab return in the role arm came from a bar phrased as thoroughness fighting a criterion phrased as an exact count.
Move countable checks into the dispatcher. Word ranges, item counts, required terms, syllables. A pre-check that returns work before a reviewer spends a run is the cheapest reviewer available, and it does not accept a self-report.
A planner is the missing half of an audit. team-science's open board was almost entirely production deploys the fleet cannot touch. The audit alone would have produced a roles document and no work. The planner turned its evidence into eight tasks that were done within the hour.
Traces belong in the ledger. Every conclusion on this page came from a run's own record. Without them the CLI bug reads as one slow task, and the syllable failure reads as a careless model rather than a structural one.

Where the pieces are

team-science: the Roles resource, the audit, the planner, and tasks 397 to 404
Speed Lab: the first run and the benchmark before it
Runner: branch claude/roles-experiment in nicolaerusan/spaces adds fleet audit, fleet decompose, role-card prompt layers, the A/B split, trace capture and fleet traces