Blog · Companion to the bake-off writeup
Published Jul 22, 2026 · Updated Aug 6, 2026
How we ran a six-week Ethereum client bake-off with Claude
How AI agents coordinated a six-week benchmark on a shared host: the test harness (measurement scripts), the orchestration model, and what broke. The results post covers the clients.
This was AI-driven, not AI-unsupervised. Humans explicitly approved every live-node destructive action and PR merge. Every result was committed using Conventional Commits and reviewed. The claim isn't that AI did it alone: the right agent–operator division of labor completed a disk-and-timing-sensitive benchmark without a person watching sync.
In 30 seconds
- Agent context runs out first. Detach node runs with systemd; wake agents only for events. Save conclusions beside the data and durable state in small files.
- Three tiers save context. Opus plans and reviews every diff. Fresh Sonnet builders implement and summarize. Independent Codex lands its own fixes on bake-off branches and adversarially reviews PRs.
- Humans control irreversible steps. One candidate at a time; 72-hour cap. A human explicitly approved wiping the synced 1.1 TiB node for pruning. A human reviewed and approved merging the measured nethermind minimal-history default; the agent never decided to merge it.
- Fast sync hides operational limits. ethrex cold-synced fastest but was not production-ready as tested: a 26-minute/132-block restart gap stalled, and longer measured gaps caused a full re-snap. Its datadir (data directory) plateaus at ~470–476 GiB but stores no history, so it is no disk win. besu synced, but its pruned re-run proved fragile after a prolonged pinned-Prysm outage.
At a glance: what, how, when & who
What
We measured final synced disk footprint and cold-sync duration for each client that finished. Seven execution clients (ELs) ran against fixed Prysm, then five consensus clients (CLs) against a fixed execution client. erigon deadlocked; reth and nimbus_eth1 hit the 72h cap.
How
Native systemd services, no Docker. One candidate at a time on a shared 12-core / ~62 GB host. Each run capped at 72 hours.
When
Jun 22 → Jul 14, 2026: 23 days. Steady-state re-measurement and restart-resume follow-ons extended the campaign through Aug 4: six weeks total.
Who
Three AI-agent tiers did the work. A human approved every destructive action and merge.
The plan
We measured each client that finished with the same protocol, across three phases over six weeks.
Six weeks, in three phases
Jun 22 → Jul 14, 2026
Seven execution clients against a fixed Prysm, then a five-way consensus sweep. Triage, installer fixes, and the first full syncs.
10 milestones · 23 days
Jul 26 → Jul 29, 2026
Re-read footprints once compaction settled, plus a third CL sweep against a nethermind anchor.
3 milestones · footprints settled
Jul 31 → Aug 4, 2026
Restart-resume experiments, the minimal-history prune default, and shipping it as a human-reviewed PR.
6 milestones · no restart cliff found
Full milestone log — all 19 dated entries
Phase 1 — Initial measurement campaign
Jun 22 → Jul 14, 2026- 2026-06-22Campaign starts, Stage A triage begins
- 2026-06-26Stage-A installer fixes shipped
- 2026-07-01besu completes un-pruned sync
- 2026-07-05besu pruned re-run abandoned, deadlocked twice
- 2026-07-06CL sweep vs ethrex anchor, 5 CLs
- 2026-07-08CL cross-check vs geth anchor
- 2026-07-10ethrex restart-cliff bisected, geth 52h resume verified
- 2026-07-13Installer / config correctness fixes shipped
- 2026-07-13nimbus_eth1 72h capped run completes
- 2026-07-14Initial campaign closes — harness and results docs shipped
Phase 2 — Steady-state re-measure
Jul 26 → Jul 29, 2026- 2026-07-26Third anchor: CL sweep re-run against nethermind
- 2026-07-28Steady-state re-measure: ethrex plateau confirmed (~470–476 GiB)
- 2026-07-29EL-disk convergence reframe published (no disk winner)
Phase 3 — Restart-resume & prune follow-ons
Jul 31 → Aug 4, 2026- 2026-07-31Nethermind fresh re-sync in 1h53m (establish run)
- 2026-08-01Steady-state re-measure: nethermind full-history datadir ~1.06 TiB
- 2026-08-01Nethermind resumes a 10,607-block gap in 35 min — restart-resume measured
- 2026-08-03Bisection finds no cliff at any gap (12 min → ~35h); prysm clean-resume measured, n=4
- 2026-08-03Nethermind prune tuning produces a minimal-history default (~250–280 GiB, no-history tier), measured and shipped as a human-reviewed PR
- 2026-08-04Minimal-history default merged to master, shipped as the installer default
Every date is a shipped fix or a completed measurement run, sourced from CLIENT_BAKEOFF_RESULTS.md and the repo's merged-PR history — not reconstructed from memory.
What broke
Four incidents changed client verdicts. The issues log records fourteen client-level problems across triage and full sync. The table covers those four.
Through 23 min / 124 blocks
Resumed
Imported the gap
26 min / 132 blocks
Stalled
Head stopped advancing
1.5–2h
Re-snapped (~2h)
Discarded state
production-viable — restart-resume measured 2026-08-01
13.3h silent stall: head frozen at block 4,651, 0 peers. Everything else looked healthy.
Root cause: P2P (peer-to-peer) bind pinned to loopback, a local-only address (Network.LocalIp=127.0.0.1)
Resolution: Advertise the real external IP
to a prolonged CL outage
Mid-sync deadlock: downloader thread died. Process stayed alive and answered RPC (remote procedure calls).
Root cause: Stale pinned CL (prysm v7.1.5) stalled the beacon ~28h. The snap-sync pivot aged beyond the ~25-min servable-state window.
Resolution: No fix available: upstream CL issue. Keep CL binaries current. Added a harness stall-watchdog.
as tested — young client, may improve
Measured restart outcomes appear above. Un-pruned datadir plateau: ~470–476 GiB, drifting 470.2 → 475.5 GiB over ~42h.
Root cause: Old head leaves the ~128-block servable-state window. The head can freeze; longer gaps can trigger full snap-sync instead of importing missing blocks.
Resolution: None — inherent to current design (v19.0.0)
on this host/CL combination
Head froze a few thousand blocks behind tip; consensus stayed is_optimistic=true indefinitely
Root cause: erigon3 OtterSync deadlocked closing the gap against checkpoint-synced Prysm. Resources were not the cause.
Resolution: No fix. Operator terminated the run; recorded as a no-sync.
The durable control loop
The shape of the problem
Sync benchmarks are slow, sequential, easy to misread, and destructive.
It's slow
ethrex snap-synced mainnet in ~2 hours. Full-sync-only clients never finished in three days. Each candidate got 72 hours.
It's sequential
One shared host has one execution slot and one consensus slot. geth and nethermind would compete for CPU, IO, and peers. Candidates run one at a time.
It's easy to measure the wrong thing
A client can install and follow the chain yet hide 0 peers or a frozen head. Archive mode invalidates a disk comparison. Sampling mid-compaction over-counts.
It's destructive
Each measurement wipes the previous client’s datadir on a host shared with other people’s work.
The challenge is sustained correctness: repeat the protocol dozens of times across all supported clients. Preserve final measurements before teardown. Separate shared-host quirks from client properties.
The orchestration model
Run nodes independently of agent sessions. Store durable state outside agent context. Extending the campaign to six weeks needed more state entries, not a different design or sustained attention.
Three clocks run at once. The shortest one decides how long the campaign can last.
Solved by detaching it. Native systemd units run from detached tmux. Sync continues after a Claude session dies.
Solved by not watching. Watchers notify once a run reaches an end condition. The orchestrator sleeps until a decision is needed.
The one that binds first. Raw journalctl, du, and RPC dumps consume context. Save conclusions as env.txt flags and durable state in small files.
not to scale — the claim is the ordering, not the ratio
1. The node runs; the agent doesn't watch it run
Native systemd services eth1.service and cl.service kept running after orchestrator sessions died. This happened more than once, including an out-of-memory event. The sampler continued; fresh sessions resumed from durable state without data loss.
2. Three tiers of agent, by cost and by role
Builders save the strongest model's context. A different model provides independent review.
outside the hierarchy
Full tier table — who each tier was, and what it did
| Role | Who | What they did |
|---|---|---|
| Orchestrator / reviewer | Claude Opus 4.8 | Planned the queue, made judgment calls, reviewed every diff, and wrote durable state. Did not hand-write most client code. |
| Builders | fresh Claude Sonnet subagents | Implemented written briefs and returned short summaries. Most tokens stayed outside orchestrator context. |
| Independent peer | Codex (OpenAI) | Outside Claude’s hierarchy, Codex used its own git identity and branches. It landed the ethrex restart-cliff bracket (2026-07-11) and bake-off review findings (2026-07-25). It also adversarially reviewed campaign PRs. On #212, it found two genuine P1 rerun-queue defects and one false positive. A different-model second opinion, not a cheaper model. |
Orchestrator / reviewer
Claude Opus 4.8
Planned the queue, made judgment calls, reviewed every diff, and wrote durable state. Did not hand-write most client code.
Builders
fresh Claude Sonnet subagents
Implemented written briefs and returned short summaries. Most tokens stayed outside orchestrator context.
Independent peer
Codex (OpenAI)
Outside Claude’s hierarchy, Codex used its own git identity and branches. It landed the ethrex restart-cliff bracket (2026-07-11) and bake-off review findings (2026-07-25). It also adversarially reviewed campaign PRs. On #212, it found two genuine P1 rerun-queue defects and one false positive. A different-model second opinion, not a cheaper model.
Builder subagents implemented the reth archive-default fix and six config-gate corrections under orchestrator review. Because it had never seen the reasoning that produced them, Codex caught defects that review missed.
3. Durable state is the backbone
Small files and disposable investigations keep orchestrator context available.
- Push conclusions down to where the data lives. env.txt flags replace logs: fully_synced=yes after two consecutive clean samples. With the watchdog armed, exhausted bounded restarts produce .stalled.
- Keep durable state small and in files. Small markdown files hold results, governance rules, the queue, and a live self-handoff. Context clears lose no campaign state.
- Keep transient investigation off the context path. Compute, summarize, then discard logs, probes, and sample dumps.
2 consecutive clean samples
window elapses, still not synced
opt-in watchdog hits its no-progress threshold. Returns to SYNCING, or reaches STALLED after the restart budget runs out.
Block number or head slot stops advancing. Opt-in watchdog exhausts bounded restarts, then flags the row before the 72-hour cap.
bakeoff_is_synced() combines sync_distance, is_optimistic, and el_offline. These are the implemented checks, not a peer-aware state machine. They do not trust eth_syncing=false alone. They never check peer count. The stall-watchdog is opt-in.
nethermind's 13.3h loopback stall predates the watchdog. The harness never falsely reported sync, but could not distinguish stuck from still syncing. That gap prompted the watchdog.
4. Governance the agent could not override
Rails the agent runs inside
- One candidate at a time. No batching. Ever.
- 72-hour cap per candidate. Use the last sample before teardown: at sync or the cap, never the peak. Compaction changes disk size. reth peaked at 1.06 TiB versus ~0.98 TiB at the 72h cap.
- New commits only, Conventional Commits. No force-push to master. Secrets stayed in protected local files, outside commits and agent context.
Levers only the human pulls
- Destructive data-cleans. Wiping the live shared node required fresh, explicit human confirmation for every run.
- Merging. Every PR merge required explicit human approval. The agent never decided to merge.
The harness
The measurement machinery lives in test/bakeoff: plain bash, ~1,550 lines across eight scripts. No extra runtime can drift beneath a systemd service. See the bake-off harness post for the function-by-function breakdown. Here are the parts that shaped the campaign.
The harness pipeline
The 6 core scripts — what each one owns
- run_bakeoff.sh — sequential orchestrator with a resume guard: a killed campaign restarts where it left off, never re-runs finished candidates.
- run_candidate.sh — single-candidate runner: reset → install → cap → sample. Owns the observation loop and captures a final footprint on synced, capped, stalled, and install-error paths before teardown. Preflight aborts such as insufficient disk exit before sampling starts.
- lib.sh — shared probe/sample library: sync probes, the sample writer, the disk snapshotter, and the config-optimality gate.
- apply_resource_caps.sh — systemd CPUQuota/MemoryMax caps so a heavy sync can't starve co-resident workloads on the shared host.
- summarize.sh — turns per-run artifacts into the results table, split by whether the config was verified optimal.
- run_anchor_rotation.sh — preserves the execution-client anchor during the consensus sweep.
The config-optimality gate, and why it needed six bug-fixes
Check the running configuration before trusting disk measurements. We corrupted early results by skipping this check. reth defaults to a ~2.8 TiB archive node; pruned mode (--full) projects to ~1.2 TiB. We nearly recorded the archive size as “reth's footprint”. A benchmark of your own misconfiguration is worse than no benchmark: it looks authoritative.
The config-optimality gate checks the running config and stamps rows config_optimal=yes|no.summarize.sh quarantines non-optimal rows as “superseded”.
We trusted the gate after six bug-fixes across three review rounds. Each assertion mismatched the generated config: the same failure the gate was built to catch.
Anchor-preserving mode: don't re-sync the world five times
Anchor-preserving mode tests five consensus clients with one execution-client sync, avoiding five full EL re-syncs. It keeps the synced EL running, cycles only the CL service per candidate, and purges the consensus datadir between runs.
Three sweeps, against ethrex, geth, and nethermind anchors, empirically confirmed EL/CL decoupling. All reproduced the same tiers; within-tier order changed.
Lightweight
lodestar · lighthouse
Mid
teku · grandine
Heavy
nimbus
- lodestar ↔ lighthouse flipped between anchors: geth: lodestar < lighthouse; ethrex: lighthouse < lodestar.
- teku swung ~25% on the same anchor: ~667 MiB, then ~835 MiB on a clean re-read of the nethermind anchor — enough to flip it from below grandine (~730 MiB) to above it.
Within-tier order tracks the measurement window more than the client. No client changed tiers.
Two harness-fidelity caveats from the nethermind-anchor sweep
- lodestar's first run was a measurement artifact. The anchor EL was still closing an unrelated block gap. That inflated sync time to ~76 minutes versus ~10 for the other four. We discarded it and re-read cleanly: ~7m36s, ~178 MiB.
- teku's run tripped a false positive. The watchdog reported anchor_synced=no. Independent checks found the anchor healthy: the check was wrong.
Two harness bugs that nearly cost us data
Unattended runs exposed two harness bugs beyond the four client incidents: a stopped shell and a lost snapshot.
The detached-shell landmine (SIGTTIN)
hung a run for 90 minutesAn install step logged the binary version with geth version | head -1. In detached tmux, its non-foreground process group read a tty it did not own. SIGTTIN stopped the whole subtree without killing it.
Fix: Redirect stdin from /dev/null on unattended invocations.
The measurement that vanished at the cap
cost reth its clean footprintOnly successful syncs took a disk snapshot before teardown. At the 72-hour cap, teardown wiped the datadir before snapshotting. Only reth hit the cap while this bug existed. We reconstructed its partial footprint from samples.jsonl.
Fix: Snapshot every end condition after installation, before teardown. Preflight aborts still exit before sampling. Slow clients take the easily forgotten cap path.
What we'd tell the next person
Measure on every exit path, before you destroy anything.
Cap and error paths can quietly lose data.
Gate your benchmark on config, not just on outcome.
Check each number against the client’s best mode, or risk publishing your own configuration mistake.
Honest limitations
Shared-host contention limits these measurements. The semi-production host had 12 cores, ~62 GB RAM, and co-resident workloads. That reflects many real node setups, but cannot fully isolate contention. Each client had one run at a pinned version: a data point, not a distribution or lab result.
An agent driving a shared host can destroy what it measures with one wrong wipe argument. Wrong numbers look right until checked. Those risks made the governance fence mandatory.
Bottom line
An agent can run this multi-week, disk-and-timing-sensitive benchmark unattended with three safeguards:
- Node time independent of agent time.
- Conclusions in small durable files, outside context.
- Human approval for irreversible steps.
Process, not greater Ethereum expertise, preserved the measurements and supports the client verdicts. It survived session deaths, context clears, config-gate bugs, SIGTTIN, and the cap-path snapshot bug.
Reproduce it
The harness is in the repo and the data is committed:
Read next
Sources: this page is adapted from docs/HOW_WE_TESTED_WITH_CLAUDE.md on GitHub, with every measured figure cross-checked against CLIENT_BAKEOFF_RESULTS.md.