How we ran a six-week Ethereum client bake-off with Claude

Four incidents changed a client's verdict. They are not the whole list — the issues log records thirteen client-level problems across triage and full sync — but these four are the ones that moved a status.

Benchmarking a sync client is deceptively expensive:

• It's slow. A single mainnet sync ranges from ~2 hours (ethrex, snap) to never finishes in three days (the full-sync-only clients). Each candidate got a 72-hour cap.
• It's sequential. One shared host, one execution slot, one consensus slot — geth and nethermind side by side would contend for CPU, IO, and peers, so candidates run strictly one at a time.
• It's easy to measure the wrong thing. A client that “installed and followed the chain” can be silently broken (0 peers, frozen head); a datadir number means nothing if the client was running in archive mode; a footprint sampled mid-compaction over-counts.
• It's destructive. Measuring the next client means wiping the last one's datadir on a shared box that also runs other people's work.

Multiply that across the whole supported field of clients and you have a task defined less by any single hard step than by sustained correctness — the discipline to run the same careful protocol dozens of times, preserve the terminal measurement before teardown, and never let a shared-host quirk masquerade as a client property.

The core design choice: decouple node wall-clock from agent wall-clock, and decouple durable state from agent context. Get those two right and a multi-week campaign stops needing a multi-week attention span. Extending the initial 23-day run into a six-week one required no change to the orchestration model — only more entries in the same durable-state files.

Every client runs as a native systemd service (eth1.service, cl.service, no Docker) in a detached tmux session — a sync proceeds for 72 hours whether or not any Claude session is alive. The orchestrating session did die mid-run more than once (once to an out-of-memory event); the systemd unit and its sampler kept going, and a fresh session picked the campaign back up from durable state with nothing lost. Instead of polling logs, the agent armed event-driven watchers that fire one notification on a terminal condition, so the orchestrator slept until something decision-worthy happened.

Not every sub-task deserves the strongest, most expensive model:

The point is context economy. A builder subagent reads the client source, produces the commit, and returns a summary; the orchestrator holds the diff, not the investigation. The reth archive-default fix and the six config-gate corrections were all implemented this way, under orchestrator review.

The harness had already solved node time. But every status check, every “is it stalled?” pulled raw journalctl, du, and RPC output into the context window, and that filled up in hours, not weeks. The fix was architectural:

• Push conclusions down to where the data lives. Each sample collapses to a couple of flags in env.txt — fully_synced=yes after two consecutive clean samples, or (with the stall-watchdog armed) .stalled once bounded restarts are exhausted — so the agent reads a file, not a log.
• Keep durable state small and in files. Results, governance rules, the queue, and a live self-handoff note live in a handful of markdown files — a mid-campaign context clear becomes a non-event.
• Keep transient investigation off the context path. Logs, probes, and sample dumps are ephemeral: computed, summarized, dropped — never carried.

This is what's actually implemented, not a peer-aware state machine: bakeoff_is_synced() checks sync_distance, is_optimistic, and el_offline together — already enough to avoid trusting eth_syncing=false alone — but there's no peer-count check anywhere, and the stall-watchdog is opt-in. nethermind's 13.3h loopback stall (see the table above) predates the watchdog: the harness correctly never reported it synced, but nothing flagged the run as stuck rather than still syncing — that gap is exactly what motivated building the watchdog afterward.

• One candidate at a time. No batching. Ever.
• 72-hour cap per candidate; footprint is the last sample before teardown — at sync for a synced client, at the cap for a capped one — never the peak. On-disk size oscillates during compaction: reth's max sample read 1.06 TiB against the ~0.98 TiB captured at the 72h cap.
• Destructive data-cleans are gated behind explicit confirmation, and wiping the live shared node always required a fresh human go-ahead.
• Conventional Commits, new commits only, never a force-push to master; secrets stayed in protected local files and were never committed or exposed to agent context.
• An agent cannot merge its own pull request. A human does that.

The measurement machinery lives in test/bakeoff. It's plain bash (~1,550 lines across eight scripts) — deliberately, so it has no runtime that can drift out from under a systemd service. The core scripts:

• run_bakeoff.sh — sequential orchestrator with a resume guard: a killed campaign restarts where it left off, never re-runs finished candidates.
• run_candidate.sh — single-candidate runner: reset → install → cap → sample. Owns the observation loop and captures a final footprint on synced, capped, stalled, and install-error paths before teardown. Preflight aborts such as insufficient disk exit before sampling starts.
• lib.sh — shared probe/sample library: sync probes, the sample writer, the disk snapshotter, and the config-optimality gate.
• apply_resource_caps.sh — systemd CPUQuota/MemoryMax caps so a heavy sync can't starve co-resident workloads on the shared host.
• summarize.sh — turns per-run artifacts into the results table, split by whether the config was verified optimal.
• run_anchor_rotation.sh — the anchor-preserving mode used for the consensus-client sweep.

Early on we corrupted our own results by recording a footprint before confirming the client was in its most disk-efficient mode: reth at its defaults runs a ~2.8 TiB archive node, and we nearly recorded that as “reth's footprint” when the pruned number (--full) is ~1.2 TiB. A benchmark that measures your own misconfiguration is worse than no benchmark — it just looks authoritative.

So the harness grew a config-optimality gate: before trusting a footprint, it inspects the actually-running config and stamps every row config_optimal=yes|no; summarize.sh quarantines non-optimal rows in a “superseded” section. The gate needed six bug-fixes across three review rounds before we trusted it — every one the same species (“the flag I asserted on doesn't match the real generated config”) — the exact failure mode the gate exists to catch, turned on itself.

The consensus-client matrix holds the execution client constant and cycles the CL. Naively that's five full EL re-syncs. Anchor-preserving mode keeps one already-synced execution client running and cycles only the CL service per candidate, purging just the consensus datadir between runs: five CL candidates, one EL sync.

We ran the sweep three times — against an ethrex anchor, a geth anchor, and a nethermind anchor — to prove the EL/CL decoupling empirically. The same three tiers reproduced on all three anchors: a lightweight pair (lodestar, lighthouse), a mid pair (teku, grandine), and nimbus alone at the heavy end — with the caveat that within-tier order is soft: the lodestar↔lighthouse order flipped between the ethrex and geth anchors (geth: lodestar < lighthouse; ethrex: lighthouse < lodestar), and teku moved ~27% between two runs on the same nethermind anchor (~667 MiB, then ~848 MiB on a clean re-read) — enough to cross grandine (~730 MiB) and back. Each client stayed inside its own tier; the ordering within a tier tracks the measurement window more than the client. That instability is itself the finding.

The nethermind-anchor sweep also surfaced two harness-fidelity caveats worth carrying forward:

• lodestar's first run happened to start while the anchor EL was still closing an unrelated block gap, inflating its recorded sync time to ~76 minutes versus ~10 for the other four — a measurement artifact, not a lodestar result, and one we discarded in favour of a clean re-read (~7m36s, ~178 MiB).
• teku's run was flagged anchor_synced=no by the watchdog even though the anchor was independently verified healthy — a false positive in the check, not in the anchor.

• The detached-shell landmine (SIGTTIN). An install step shelled out to geth version | head -1 to log the binary version. Run from a detached tmux session in a non-foreground process group, that read raised SIGTTIN against a tty it didn't own — which stops (not kills) the whole subtree — and hung a run for 90 minutes. Fix: redirect stdin from /dev/null on unattended invocations.
• The measurement that vanished at the cap. The disk snapshot was taken only on the synced success branch. When a slow client hit the 72-hour cap, the script fell through to teardown — which wiped the datadir — and snapshotted after. Fix: snapshot every terminal run path after installation and before teardown; preflight aborts still exit before sampling. The cap path is the one you forget, and it's the one a slow client actually takes.

• The third clock is the real limit. Node wall-clock and agent wall-clock are solvable with infrastructure; agent context only scales if you push conclusions to the data and keep durable state in small files.
• Measure on every exit path, before you destroy anything. Success is the easy path. The cap and the error paths are where your data quietly disappears.
• Gate your benchmark on config, not just on outcome. Stamp every number with “was this the client's best mode?” or you will eventually publish a measurement of your own mistake.
• Give an agent a job and a fence. The agent owns the tedious, sustained correctness; the human owns the few irreversible levers.

This is a real benchmark, not a lab result. It ran on a shared, semi-production host (12 cores, ~62 GB RAM, co-resident workloads) — representative of how many people actually run nodes, but with contention the numbers can't fully isolate. Each client was measured on one run at a pinned version, so a single result is a data point, not a distribution. An agent driving a shared host can also destroy the thing it is measuring: the wipe that precedes each candidate is one wrong argument away from the wrong datadir, and a number that is wrong is indistinguishable from a number that is right until someone checks it. That is why the fence above was non-negotiable rather than advisory.

The harness is in the repo and the data is committed: