How I ship client work with an AI agent

The situation

An AI agent hands you volume immediately. Checking that volume is where the time comes back.

If I have to read 200 generated commits the way I'd read a stranger's pull request, I've saved nothing. And "looks right" stops being a usable standard somewhere around the fifth one.

What I wanted was speed I could keep across projects rather than a good week. So the checking went into the system, and I instrumented what it cost.

The constraints

Four things I couldn't trade away.

No cloud key at runtime. The intake agent talks to a local model. Unplug the network and it still answers.

Operational data stays on the machine. The board's tickets are working notes. They never leave it.

It had to survive the next project. Otherwise I'd be re-deciding the same things every time I opened a new repo.

Reviewable without reading everything. If verifying the output took as long as writing it, the time saved goes straight back into review.

The engine

A governance layer sits between the agent and the codebase: typed tools instead of a shell, the dangerous git commands blocked by a hook rather than discouraged in documentation, a "done" that is part gate and part record, and every run's cost measured rather than estimated. Each trades a little of the agent's speed for something I can point at.

It works. The kanban board this case study is about was built by the agent, operating under these rules, tracked on itself: 367 commits across 40 active days, 335 of them AI-co-authored, 1737 tests.

Verified 2026-08-30, and reproducible from the repo — e.g. git rev-list --count --until='2026-08-30 23:59:59' HEAD · git log --format='%cd' --date=short --until='2026-08-30 23:59:59' | sort -u | wc -l. The test counts come from npx vitest run at that commit.

The board mattered less than the discipline around it, so I pulled the core out into ticket-workflow, a standalone package with its own CI gate and protected main.

What it runs now

The board and the git guard are wired in at the machine level rather than set up per project, so new work picks them up on day one. The done-gate and the cost telemetry are per-repo and get set up each time. Two builds for other people run on it now, with a third scoped.

Equipment schedules and submittal review for a mechanical engineer. He specifies the equipment, the vendor returns a submittal PDF, and the app checks it back against the schedule: values outside tolerance, and units nobody quoted. Doing that by hand means re-keying the vendor's numbers into the schedule row by row.

An auction filtering tool for an independent mechanic. It pulls a capped result set off a salvage auction site and makes it workable on a tablet out in the yard.

A mobile app for a community choir. Scoped and ticketed. The build hasn't started.

The domains have nothing to do with each other. The starting setup was identical in all three.

What it cost to build (with Claude)

This is the Claude side of the ledger, separate from the local intake agent the app runs. I reconstructed the build economics from Claude Code's own session logs and committed the script plus a frozen snapshot to the repo, so the method is auditable rather than asserted.

The part derived from the two repositories rather than estimated: 155 reviewed, merged PRs and 26,693 lines of code, about 5.3 merged PRs a day.

Verified 2026-07-21 by the snapshot generator that produced these figures — kanban repo → analysis/build-economics/analyze-kanban-savings.mjs (sums both repos).

Set against a by-hand counterfactual at a deliberately low 2–3 hours per PR, that comes to roughly 5–9 weeks of engineering time avoided, against a measured Claude bill of ~$2,485. Don't take the range on faith — move the assumptions yourself and watch the hours, dollars and ROI recompute.

The asterisks, because a number without them is marketing: the dollar cost is self-reported from local session logs — only the PRs, LOC, and velocity are independently reproducible. It's a floor: it counts only the sessions that closed a ticket, so the branches that went nowhere aren't in it. And the by-hand hours are an estimate — I anchored on merged PRs rather than ticket counts (git-verifiable, and I'd already watched a ticket-based count swing threefold), deduplicated the raw token counts (each streamed response is logged several times), and scoped them to this repo. A case study about honest measurement doesn't get to skip its own footnotes.

Why two models at all? Different jobs. Building the app is a one-time, capability-hungry task where a frontier model earns its keep, and the ~$2,485 Claude bill is a rounding error against the time it saved. The intake agent runs on every report, over operational data you may not want leaving the building, so recurring cost and privacy both point local. Frontier model for the build, local model for the runtime.

How it works

Four decisions, each a guarantee bought by giving up a little speed. The later ones only hold because the earlier ones do.

1. Typed tools instead of a shell

The board is exposed through a from-scratch MCP server, so the agent calls typed functions rather than hand-editing markdown. Failures move from runtime to the boundary: update_ticket validates status against a shared enum, and the check repeats in the service layer underneath, so every route through the board's tools inherits it — status: "in progres" comes back an error, not a corrupted row. Editing a ticket file by hand goes around it, and an invalid status there is quietly read as backlog. The guarantee holds at the tools; it isn't a property of the files underneath.

2. Rules enforced by code, not honor-system

One ticket, one branch. A guard hook inspects every shell command the agent runs and blocks the dangerous git shapes (git add -A, commits to main, force-push, reset --hard). An agent that can be talked out of a rule doesn't have a rule. In a prompt it holds until the context gets long. In a hook it holds because the command doesn't run.

The merge is held somewhere else, and the distinction is worth making. Branch protection on the board's repo requires a pull request and a green gate, and nobody, me included, is on its bypass list. That runs on GitHub's side rather than through the git commands the hook inspects. Squashing each branch down to one commit is my own convention on top of that, and nothing in the platform enforces that part.

3. "Done" is part gate, part record

The definition of done is four items: typecheck, lint, tests, and a summary carrying a test line — Tests: N added or Tests: none — <reason>. The first three run as a required check on every pull request into main on the board's repo, and a red result blocks the merge. The fourth is a different kind of item. Ticket bodies live outside the repository, so CI never reads that line; it is what the workflow asks for and what I check at review. Calling it enforced would be exactly the overstatement the checklist exists to catch.

"None" is allowed; silence isn't. Making the skip explicit turns an omission into a decision someone has to defend.

4. Cost measured, not estimated

The intake agent runs local-first against an OpenAI-compatible endpoint, with no cloud key and no per-call charge. That creates a problem: every LLM cost dashboard multiplies tokens by a published price, and locally there is no price. So the cost model reports measured, assumed and externalities separately and refuses to blur them. The numbers below show why that matters.

The proof

It searched before it wrote

On the runs I recorded, the intake agent's first move was retrieval rather than generation. One real run, nothing tidied: someone reported that list_tickets silently truncates at ~440 tickets, and the board already had a ticket for exactly that.

The agent found the ticket that already existed and called update_ticket on it instead of create_ticket — the anti-duplicate rule working — and a human approved it before anything touched the board. To step through a whole pipeline — the note, every hit and its score, the reasoning, and the gate — walk through three real runs in the replay viewer.

The number that changed my mind

This is the other model — the app's own local intake agent, not Claude. A different run from the one above, costed, passed back through the same renderSummary() the CLI prints. I'm not going to pretend one recording shows both. The ragged 0.2848888888888889 is why I don't round any of it: real instruments don't round for you.

runId    087f1884-f6af-4689-918d-d05c4c9121d1
at       2026-07-14T16:31:36.484Z
model    openai/gpt-oss-20b
calls    2  activeMs 4787  reviewMs 20512
outcome  {"created":1,"updated":0,"declined":0,"noProposal":false,"errored":false}
— Measured —
  active compute                           4787 ms
  prompt tokens                            2246 tokens
  completion tokens                         138 tokens
  total tokens                             2384 tokens
  cacheable prefix                         89.2 %   (estimate (~4 chars/token))
  accepted tickets                            1 count

— Assumed ($) —
  marginal energy                      notional kWh   (notional — set active watts, idle watts)
  marginal energy cost                 notional USD   (notional — set active watts, idle watts)
  keep-warm energy cost                notional USD   (notional — set idle watts)
  hardware amortization                notional USD   (notional — set hardware cost)
  review time cost             0.2848888888888889 USD   (21s x 50/hr)
  total run cost                       notional USD   (notional - some cost inputs unset)
  manual value (avoided)       4.166666666666666 USD   (5min x 50/hr)
  cloud-equivalent (openai/gpt-oss-20b)         notional USD   (notional — no price configured for this model)

— Externalities (report-only) —
  water footprint                      notional L   (notional)
  carbon footprint                     notional gCO2e   (notional)

— Headline —
  cost per accepted ticket             notional USD   (notional - total cost incomplete)
  net savings                          notional USD   (notional - value or cost incomplete)
  local vs cloud (saved)               notional USD   (notional - cloud or local incomplete)

The line that reframed the whole project is the timing pair:

4,787ms model computevs20,512ms human review

On this run the human took 4.3× longer than the local model, which put the model at 19% of the total. A faster model has 5 seconds to play with. Cutting the review burden with better drafts and a tighter approval screen goes after 21. The industry conversation is about tokens and latency, and in a human-in-the-loop workflow the human is the cost center.

The caveat, stated plainly: reviewMs is instrumented on 7 of the logged runs — a real sample now, but a small one. Across them the review-to-compute ratio runs from 3.7× to 12.1×, median 5.8×; the run costed in full above, at 4.3×, actually sits near the low end. That's enough to trust the direction — the human dominates the loop — without pretending the exact multiple is a law. A case study about honest measurement doesn't get to round that up.

The win it refused to book

On another run the agent read the note, searched the board, and proposed nothing — an existing ticket already covered it. Watch what the cost model does with that. On the accepted run it books manual value (avoided): 4.166666666666666 USD — the typing the agent saved. Here it books 0: no accepted ticket - nothing avoided. Across 66 runs, 53 earn that credit and 13 don't — $220.83 honest. Booking it unconditionally would read $275.00, a 25% overstatement of the headline from a one-line change nobody would ever see.

If you've ever watched a demo fall apart under a customer's questions, that's the discipline that stops it happening to you. A dashboard that only reports wins isn't measurement, it's marketing — and the first person it fools is the one holding it.

Where it broke

Two things failed, and I'm showing them on purpose.

The seam bugs. The in-app intake feature shipped ~8 real bugs — dropped fields, a silent no-op save, an update misrouted into a duplicate create. Every one survived a green test suite and a per-ticket code review, because they lived in the handoff between modules: each module was correct in isolation, and no single diff-scoped review ever saw the whole path. The fix wasn't more tests — it was one end-to-end round-trip test asserting source input == persisted output across the full chain.

The guard hook failed open. It resolved "what branch am I on?" from the wrong repo, so a genuine direct commit to a sibling's main sailed straight through — the exact class of bug the hook exists to prevent. What caught it, after every unit test passed green, was a suite that drives the real hook against real temporary repos. The same seam bug as the intake feature, in the code I wrote to fix a different one.

Neither was caught by me being careful. The guard-hook failure was caught by a suite that drives the real hook against real repositories. The seam bugs were caught by nothing — they shipped, and the round-trip test that would have stopped them was written afterward. That is the reason it now gets written before the feature.

What I'd change

Instrument the human from run one. reviewMs is the most useful number in this study, and it was added late — which is why only 7 runs carry it. The model's tokens were metered from the first call; the human's minutes should have been too.

Make every guard prove it can fail. The guard hook passed a green unit suite while failing open. A green check isn't evidence until you've watched it go red — so now no guarantee counts as one until its test has produced the failure it claims to prevent.

Who this is for

If you're trying to work out whether AI-assisted delivery is reliable enough to put real work through, this is what the honest version looks like, including the parts that didn't work.

The one number I'd put in front of anyone betting on this space is 5.8×. It was measured on the local agent rather than on Claude, and the lesson carries either way. The model is cheap. The human is the cost.