How we ran a 23-day Ethereum client bake-off with Claude

Four real incidents hit during the campaign, all fixed or explicitly documented.

Benchmarking a sync client is deceptively expensive:

• It's slow. A single mainnet sync ranges from ~2 hours (ethrex, snap) to never finishes in three days (the full-sync-only clients). Each candidate got a 72-hour cap.
• It's sequential. One shared host, one execution slot, one consensus slot — geth and nethermind side by side would contend for CPU, IO, and peers, so candidates run strictly one at a time.
• It's easy to measure the wrong thing. A client that “installed and followed the chain” can be silently broken (0 peers, frozen head); a datadir means nothing if it's running in archive mode; a footprint sampled mid-compaction over-counts.
• It's destructive. Measuring the next client means wiping the last one's datadir on a shared box that also runs other people's work.

Multiply that across the whole supported field of clients and three weeks and you have a task defined less by any single hard step than by sustained correctness — the discipline to run the same careful protocol dozens of times, preserve the terminal measurement before teardown, and never let a shared-host quirk masquerade as a client property.

The core design choice: decouple node wall-clock from agent wall-clock, and decouple durable state from agent context. Get those two right and a three-week campaign stops needing a three-week attention span.

Every client runs as a native systemd service (eth1.service, cl.service, no Docker) in a detached tmux session — a sync proceeds for 72 hours whether or not any Claude session is alive. The orchestrating session did die mid-run more than once (once to an out-of-memory event); the systemd unit and its sampler kept going, and a fresh session picked the campaign back up from durable state with nothing lost. Instead of polling logs, the agent armed event-driven watchers that fire one notification on a terminal condition, so the orchestrator slept until something decision-worthy happened.

Not every sub-task deserves the strongest, most expensive model:

The point is context economy. A builder subagent can read ten thousand lines of client source, produce a three-line commit, and return “done, here's the diff” — the orchestrator never has to hold those ten thousand lines. It reviews the diff, not the investigation.

The harness had already solved node time. But every status check, every “is it stalled?” pulled raw journalctl, du, and RPC output into the context window, and that filled up in hours, not weeks. The fix was architectural:

• Push conclusions down to where the data lives. Each sample collapses to a couple of flags in env.txt — fully_synced=yes after two consecutive clean samples, or (with the stall-watchdog armed) .stalled once bounded restarts are exhausted — so the agent reads a file, not a log.
• Keep durable state small and in files. Results, governance rules, the queue, and a live self-handoff note live in a handful of markdown files — a mid-campaign context clear becomes a non-event.
• Keep transient investigation off the context path. Logs, probes, and sample dumps are ephemeral: computed, summarized, dropped — never carried.

This is what's actually implemented, not a peer-aware state machine: bakeoff_is_synced() checks sync_distance, is_optimistic, and el_offline together — already enough to avoid trusting eth_syncing=false alone — but there's no peer-count check anywhere, and the stall-watchdog is opt-in. nethermind's 13.3h loopback stall (see the table above) predates the watchdog: the harness correctly never reported it synced, but nothing flagged the run as stuck rather than still syncing — that gap is exactly what motivated building the watchdog afterward.

• One candidate at a time. No batching. Ever.
• 72-hour cap per candidate; footprint is the last near-cap sample, never the peak.
• Destructive data-cleans are gated behind explicit confirmation, and wiping the live shared node always required a fresh human go-ahead.
• Conventional Commits, new commits only, never a force-push to master; secrets stayed in protected local files and were never committed or exposed to agent context.
• An agent cannot merge its own pull request. A human does that.

The measurement machinery lives in test/bakeoff. It's plain bash — deliberately, so it has no runtime that can drift out from under a systemd service:

• run_bakeoff.sh — sequential orchestrator with a resume guard: a killed campaign restarts where it left off, never re-runs finished candidates.
• run_candidate.sh — single-candidate runner: reset → install → cap → sample. Owns the observation loop and captures a final footprint on synced, capped, stalled, and install-error paths before teardown. Preflight aborts such as insufficient disk exit before sampling starts.
• lib.sh — shared probe/sample library: sync probes, the sample writer, the disk snapshotter, and the config-optimality gate.
• apply_resource_caps.sh — systemd CPUQuota/MemoryMax caps so a heavy sync can't starve co-resident workloads on the shared host.
• summarize.sh — turns per-run artifacts into the results table, split by whether the config was verified optimal.
• run_anchor_rotation.sh — the anchor-preserving mode used for the consensus-client sweep.

Early on we corrupted our own results by recording a footprint before confirming the client was in its most disk-efficient mode: reth at its defaults runs a ~2.8 TiB archive node, and we nearly recorded that as “reth's footprint” when the pruned number (--full) is ~1.2 TiB. A benchmark that measures your own misconfiguration is worse than no benchmark — it just looks authoritative.

So the harness grew a config-optimality gate: before trusting a footprint, it inspects the actually-running config and stamps every row config_optimal=yes|no; summarize.sh quarantines non-optimal rows in a “superseded” section. The gate needed six bug-fixes across three review rounds before we trusted it — every one the same species (“the flag I asserted on doesn't match the real generated config”) — the exact failure mode the gate exists to catch, turned on itself.

The consensus-client matrix holds the execution client constant and cycles the CL. Naively that's five full EL re-syncs. Anchor-preserving mode keeps one already-synced execution client running and cycles only the CL service per candidate, purging just the consensus datadir between runs: five CL candidates, one EL sync. We ran the sweep twice — against an ethrex anchor and a geth anchor — to prove the EL/CL decoupling empirically. The heavyweight and lightweight tiers both reproduced on both anchors — with one nuance: the lodestar↔lighthouse order flipped between anchors (geth: lodestar < lighthouse; ethrex: lighthouse < lodestar), though both stayed in the same smallest tier, where the gap is small and measurement-window-sensitive.

• The detached-shell landmine (SIGTTIN). An install step shelled out to a version-check command. Run from a detached tmux session in a non-foreground process group, that read raised SIGTTIN against a tty it didn't own — which stops (not kills) the whole subtree — and hung a run for 90 minutes. Fix: redirect stdin from /dev/null on unattended invocations.
• The measurement that vanished at the cap. The disk snapshot was taken only on the synced success branch. When a slow client hit the 72-hour cap, the script fell through to teardown — which wiped the datadir — and snapshotted after. Fix: snapshot every terminal run path after installation and before teardown; preflight aborts still exit before sampling. The cap path is the one you forget, and it's the one a slow client actually takes.

• The third clock is the real limit. Node wall-clock and agent wall-clock are solvable with infrastructure; agent context only scales if you push conclusions to the data and keep durable state in small files.
• Measure on every exit path, before you destroy anything. Success is the easy path. The cap and the error paths are where your data quietly disappears.
• Gate your benchmark on config, not just on outcome. Stamp every number with “was this the client's best mode?” or you will eventually publish a measurement of your own mistake.
• Give an agent a job and a fence. The agent owns the tedious, sustained correctness; the human owns the few irreversible levers.

This is a real benchmark, not a lab result. It ran on a shared, semi-production host (12 cores, ~62 GB RAM, co-resident workloads) — representative of how many people actually run nodes, but with contention the numbers can't fully isolate. Each client was measured on one run at a pinned version, so a single result is a data point, not a distribution. An AI-in-the-loop campaign also carries its own risk surface, which is exactly why the governance fence above was non-negotiable rather than advisory.

The harness is in the repo and the data is committed: