Skip to content

Chalk harness

Chalk report card: Claude Code agents, one checkpoint at a time, with a spend cap and a test gate. It measures the cost per finished checkpoint, commits only loops that pass the rubric, and learns from detentions resolved in office hours. Chalk report card: Claude Code agents, one checkpoint at a time, with a spend cap and a test gate. It measures the cost per finished checkpoint, commits only loops that pass the rubric, and learns from detentions resolved in office hours.

Chalk runs Claude Code agents in disposable sandboxes, one small checkpoint at a time. Each loop has a spend cap, the harness runs your tests itself before anything is committed, and an agent that gets stuck stops and waits for a person instead of spending more. Finished work arrives as a GitHub pull request or a GitLab merge request.

Status: early

The whole workflow is covered by end-to-end tests with fake docker, claude, gh and glab, and the SQL is tested against a real Postgres. Chalk has had few runs against real projects yet, so expect rough edges and please report them.

Where to start

  • New to Chalk: Getting started installs it, sets up a repository and runs a first ticket.
  • Running tickets: The workflow, then Detention and office hours for when a run gets stuck.
  • Tuning: Configuration lists every setting; the report card shows what to change.
  • The words: Chalk names its parts after school. The glossary says what each one means; on every page, a term with a dotted underline shows its definition when you hover over it.
  • Contributing: Architecture says where each part of a run lives, and Testing how to check a change without Docker or spend.

How a run works

flowchart LR
  spec[Spec check] --> loop[Agent loop on the next checkpoint]
  loop --> rubric{Rubric}
  rubric -- pass --> commit[Commit]
  commit -- checkpoints left --> loop
  commit -- all done --> review{Final review}
  review -- pass --> request([Pull or merge request])
  review -- findings, fix round --> loop
  rubric -- fail, retry --> loop
  loop -- blocker --> detention([Detention])
  rubric -- retries used up --> detention
  review -- still failing --> detention
  1. Chalk starts a container with your branch cloned into a RAM disk. The container has no GitHub or GitLab credentials.
  2. A cheap model checks that every checkpoint in the spec is small, testable and unambiguous. If not, the run stops before any loop is paid for.
  3. Each loop, the agent works on the next unchecked checkpoint under a spend cap. Then the harness runs the rubric, your test command, itself.
  4. When the rubric passes and a checkpoint was ticked, the harness commits the work to your local branch. Otherwise the agent gets the failure and tries again, up to a retry limit.
  5. When every checkpoint is done, an independent agent reviews the change. Its findings get one fix round.
  6. A passing review opens the pull or merge request.
  7. A reported blocker, exhausted retries, the loop limit or a review that still fails sends the run to detention: nothing is pushed, the work is parked on a local branch, and a person takes over in office hours.

A run that needs help

This recording runs the real chalk against the test fakes, so it is free and the same every time. The agent keeps leaving a stray file that the rubric rejects; after three loops the run goes to detention, a person removes the file and explains it in office hours, and the resumed run finishes and opens a merge request.

A terminal recording, simulated with the test fakes: chalk run goes to detention after three failed loops, a person fixes the work and runs chalk office-hours, and the resumed run opens a merge request.

Select the image to play the recording.

Transcript 
shop $ # Simulated with Chalk's test fakes: no Docker, no model calls, no spend.
shop $ chalk new SHOP-7 "Add a discount code field"
created /home/sam/shop.worktrees/SHOP-7 on branch chalk/SHOP-7
next: edit specs/SHOP-7.md, commit it, then run 'chalk run' from that directory
shop $ cd ../shop.worktrees/SHOP-7
SHOP-7 $ # Spec written and committed: two checkpoints. Each loop leaves a stray file.
SHOP-7 $ chalk run
[SHOP-7] starting the database and the sandbox image
[SHOP-7] starting sandbox chalk-sandbox-shop-SHOP-7 on chalk/SHOP-7
[SHOP-7] cloning chalk/SHOP-7 into the sandbox
[SHOP-7] checking the spec with haiku
[SHOP-7] spec check passed ($0.25, 0s)
[SHOP-7] loop 1 (continue) started
[SHOP-7] loop 1 (continue): agent ok ($0.25, 0s), rubric exit 1. did the thing
[SHOP-7] loop 2 (retry) started
[SHOP-7] loop 2 (retry): agent ok ($0.25, 0s), rubric exit 1. did the thing
[SHOP-7] loop 3 (retry) started
[SHOP-7] loop 3 (retry): agent ok ($0.25, 0s), rubric exit 1. did the thing
[SHOP-7] DETENTION: rubric failed (exit 1)
[SHOP-7] next: the rubric still fails after 2 retries; fix the failure in /home/sam/.local/state/chalk/shop/runs/SHOP-7/io/rubric.log, or make the checkpoint smaller
[SHOP-7] work parked on local branch detention/SHOP-7-1791215648. To unblock:
[SHOP-7]   cd '/home/sam/shop.worktrees/SHOP-7' && git switch 'detention/SHOP-7-1791215648'
[SHOP-7]   fix the blocker, commit, then: chalk office-hours -m "what was wrong"
SHOP-7 $ # Nothing was pushed. A person looks at the parked work and fixes it.
SHOP-7 $ git switch -q detention/SHOP-7-1791215648
SHOP-7 $ git rm -q BROKEN && git commit -q -m "Remove the stray BROKEN file"
SHOP-7 $ chalk office-hours -m "Never commit scratch files such as BROKEN"
lesson: Distilled: never commit a BROKEN marker
lesson recorded; continuing on tutoring/SHOP-7-1791215650
[SHOP-7] starting the database and the sandbox image
[SHOP-7] starting sandbox chalk-sandbox-shop-SHOP-7 on tutoring/SHOP-7-1791215650
[SHOP-7] cloning tutoring/SHOP-7-1791215650 into the sandbox
[SHOP-7] spec check skipped: it passed before and the checkpoints have not changed
[SHOP-7] loop 1 (continue) started
[SHOP-7] loop 1 (continue): agent ok ($0.25, 0s), rubric exit 0. implemented a checkpoint
[SHOP-7] loop 2 (continue) started
[SHOP-7] loop 2 (continue): agent ok ($0.25, 0s), rubric exit 0. implemented a checkpoint
[SHOP-7] final review started
[SHOP-7] final review: pass. Looks complete.
[SHOP-7] all checkpoints complete (5 loops, $2.00, 1 human interventions)
https://gitlab.example.com/acme/shop/-/merge_requests/42

The report card

chalk dashboard builds one HTML page from the telemetry on your machine: the cost per finished checkpoint, the share of spend that made no progress, and notes on what to tune. Nothing is sent anywhere.

The report card, light theme, with sample data: $0.57 per finished checkpoint, the totals, two notes for next term and spend by kind of call. The report card, dark theme, with sample data: $0.57 per finished checkpoint, the totals, two notes for next term and spend by kind of call.

Sample data, not from a real run. See Verdicts and the report card.