The Bake-off Harness — Function-Level Engineering Reference

test/bakeoff/
├── run_bakeoff.sh          # sequential multi-candidate orchestrator + resume guard
├── run_candidate.sh        # single-candidate driver: reset → install → caps → sample-to-verdict
├── run_anchor_rotation.sh  # multi-EL anchor-rotation driver (establish EL once, sweep CLs against it)
├── apply_resource_caps.sh  # systemd CPUQuota/MemoryMax/IOWeight apply|clear
├── lib.sh                  # shared probe/sample/config-gate library (sourced, never executed)
├── summarize.sh            # aggregates artifacts/ into summary.csv, report.md, a results skeleton
├── candidates.tsv          # the manifest: <execution>\t<consensus> pairs to run
└── test_data_dirs_sync.sh  # CI guard: BAKEOFF_DATA_DIRS must match purge_ethereum_data.sh

The executable driver scripts (run_bakeoff.sh, run_candidate.sh, run_anchor_rotation.sh, apply_resource_caps.sh, summarize.sh) each set set -Eeuo pipefail and source lib/common_functions.sh for logging (log_info/log_warn/log_error). lib.sh is the exception noted in the layout above: it's a sourced library, never executed on its own, that inherits strict mode from whichever script sources it and does not source common_functions.sh itself — its log_* calls resolve at runtime because every caller (run_candidate.sh, run_anchor_rotation.sh) already sourced common_functions.sh first.

run_bakeoff.sh owns the ordinary sequential sweep; run_anchor_rotation.sh reuses one synced execution-layer anchor while cycling consensus clients. Both paths produce the same per-candidate artifacts, so summarize.sh has one source of truth.

Usage: run_bakeoff.sh --stage=triage|full [--only=el__cl,el__cl,...] [--force]

What it does, in order:

1. Stage-dependent windows. --stage=triage sets a 5400s (90 min) sync window with 120s sampling (ETH2QS_BAKEOFF_SYNC_WINDOW_SECONDS, ETH2QS_BAKEOFF_SAMPLE_INTERVAL_SECONDS); --stage=full sets a 259200s (72h) window with 600s sampling. Both are overridable via env (the script only supplies the default with ${VAR:-default}).
2. Config swap with restore-on-exit. It backs up the current config/user_config.env to config/user_config.env.bakeoff-backup, installs config/bakeoff.env in its place, and registers trap restore_cfg EXIT so the operator's real config is always restored — even on Ctrl-C or a crash mid-run.
3. Candidate selection from candidates.tsv. The manifest is <execution>\t<consensus> pairs; run_bakeoff.sh turns each row into an el__cl pair id with awk -F'\ ' 'NF>=2{print $1"__"$2}'.
   • --only=a__b,c__d explicitly overrides the manifest.
   • --stage=full without --only auto-selects only candidates whose triage env.txt already recorded install_exit_code=0 — this is the resume guard: a full run only re-attempts candidates that passed triage, and running --stage=full twice is safe (it re-derives the same selection from disk, it doesn't remember state in the orchestrator itself).
   • --stage=triage with no --only runs every manifest row.
4. Sequential dispatch, tolerant of failure. For each selected pair it calls run_candidate.sh "$execution" "$consensus" with set +e around the call, so one candidate's non-zero exit does not abort the sweep; the exit code is appended to artifacts/<run-id>/orchestrator-<stage>.log.
5. Always calls summarize.sh at the end (|| log_warn, non-fatal if it errors).

Artifact root: artifacts/${ETH2QS_BAKEOFF_RUN_ID:-client-bakeoff-2026-06-22}/. Every subsequent script keys off the same env var, so a fresh campaign is just a new ETH2QS_BAKEOFF_RUN_ID.

Usage: run_candidate.sh <execution> <consensus>. This is the workhorse; every candidate row in the results tables is one invocation of this script. It refuses to run at all unless ETH2QS_BAKEOFF_CONFIRMED=yes is set (log_error … exit 2) — a deliberate manual safety gate because the script purges data directories and stops/disables services.

The script has three modes, selected by env vars, all mutually aware of each other:

Anchor mode has its own precondition gate before touching anything: the $execution arg must equal $ETH2QS_BAKEOFF_ANCHOR_EL (mismatch → exit 3), eth1.service must be systemctl is-active, Engine API port 8551 must answer with an HTTP-shaped code (^[1-5][0-9]{2}$, since 401 auth-required counts as “responding”), and a bounded 5-retry/2s-sleep loop must observe eth_syncing resolve to “caught up” (result == false, or a progress object where currentBlock == highestBlock and highestBlock != "0x0") before the CL sweep is allowed to start.

If $out/.done exists and ETH2QS_BAKEOFF_FORCE isn't yes, the candidate is skipped entirely (exit 0). Setting ETH2QS_BAKEOFF_FORCE=yes also clears .anchor-poisoned and .done so a forced rerun doesn't inherit a stale poison marker from a previous attempt — without that clear, the anchor watchdog guard later in the script would bypass itself and finalize anchor_synced=no for what is actually a clean rerun.

1. Stop+disable the relevant services (eth1.service cl.service validator.service, or just cl.service validator.service in anchor mode) — 2>/dev/null || true, tolerant of units that don't exist yet.
2. Purge datadirs via ./scripts/eth2qs.sh clean-data — --dry-run first (logged, non-fatal), then --confirm (must succeed). Anchor mode passes --scope=consensus so the EL anchor is never touched. < /dev/null on every clean-data and install call prevents SIGTTIN when the harness runs detached (backgrounded/tmux) — a non-interactive process group reading from a controlling TTY gets stopped by the kernel, not an error the script can catch.
3. bakeoff_snapshot_disk "$out/disk-before.tsv" — baseline footprint.
4. Disk-floor guard, checked via df -B1 --output=avail /: anchor mode requires ETH2QS_BAKEOFF_MIN_DISK_BYTES:-429496729600 (400 GiB — CL-only, since the EL already occupies disk), full mode requires :-1717986918400 (1.6 TiB — full EL+CL). Below the floor, the script aborts with exit 3 before starting an install that would fail hours later from ENOSPC.
5. ./scripts/eth2qs.sh plan|doctor|stats --json snapshots, best-effort (|| true).
6. Opportunistic GITHUB_TOKEN export from gh auth token if neither GITHUB_TOKEN nor GH_TOKEN is already set — avoids the 60/hr unauthenticated GitHub API rate limit during client-version lookups inside the install scripts.

timeout "$install_timeout" /usr/bin/time -v -o "$out/install-time.txt" \
  ./scripts/eth2qs.sh phase2 --execution="$execution" --consensus="$consensus" --mev=none \
  </dev/null > "$out/install.log" 2>&1

(Anchor mode omits --execution= since only the CL installs.) install_timeout defaults to 90m (ETH2QS_BAKEOFF_INSTALL_TIMEOUT). /usr/bin/time -v is what summarize.sh later parses for install wall-clock (see §6). The exit code is captured with set +e/set -e bracketing and written to env.txt as install_exit_code=.

Immediately after install: apply_resource_caps.sh apply (§4), then a battery of post-install snapshots (doctor, stats, monitor export, debug --service eth1, debug --service cl, systemctl status eth1 cl validator), then the config-optimality gate, bakeoff_check_config_optimal (§5.5) — non-blocking, it stamps config_optimal=yes|no into env.txt and never aborts the run.

Skipped entirely if install_rc != 0. Otherwise a while loop runs until $(date +%s) >= end_at (end_at = now + window), sleeping interval seconds between iterations. Each iteration:

1. bakeoff_write_sample "$out" "$REPO_ROOT" — the sampling primitive (§5.4).
2. bakeoff_services_alive check — sets crashed=yes if either service unit is no longer active (this becomes service_crash_observed in env.txt, a hard fail signal downstream).
3. Anchor watchdog (anchor mode only, and only while .anchor-poisoned doesn't already exist): detects if the anchor EL silently dropped out of sync (service down, or eth_syncing no longer reporting caught-up). Two consecutive misses (anchor_miss_streak -ge 2) touches .anchor-poisoned and logs an error — detection only, it never restarts or kills anything, because the anchor EL is shared state across the whole CL sweep and killing it would invalidate every remaining candidate in the rotation.
4. Stall watchdog (opt-in via ETH2QS_BAKEOFF_STALL_RESTART=yes, inert otherwise): tracks no_progress_streak — the CL's head_slot (or, if absent, a sync_distance-derived proxy) in anchor mode, or the EL's currentBlock (hex-decoded) otherwise. stall_samples (default 10) consecutive non-advancing samples triggers sudo systemctl restart on the stalled unit, up to stall_max_restarts (default 3) times; exceeding that touches .stalled, writes stall_restarts=/stall_failed=yes to env.txt, and breaks the loop — the row is invalid past that point but the harness still finalizes cleanly rather than hanging forever on a genuinely dead node.
5. Synced-streak check: bakeoff_is_synced (§5.3) must return true on two consecutive samples (synced_streak -ge 2) before the row is accepted as synced — a single true reading can be a race between the EL and CL heads. On that streak, it snapshots disk-synced.tsv, writes synced_at_utc=/fully_synced=yes, and breaks out of the window early (no need to wait for the full 72h cap once synced).

After the loop (whether by break or timeout), one final bakeoff_write_sample call captures the end state, then env.txt gets fully_synced=no if nothing set it, service_crash_observed=, and (anchor mode only) anchor_synced=yes|no depending on whether .anchor-poisoned exists.

journalctl tails (-u eth1 -n 700, -u cl -n 700, -u validator -n 300) and ./scripts/eth2qs.sh repair preview are captured regardless of outcome. A findings.md skeleton is seeded with the install/crash verdict for a human reviewer to fill in (Result: TBD-by-reviewer). bakeoff_snapshot_disk "$out/disk-final.tsv" captures the footprint pre-cleanup (synced or capped — this is what summarize.sh falls back to when disk-synced.tsv doesn't exist). Then apply_resource_caps.sh clear, then clean-data again (scoped the same way as pre-install), then a final disk-after-cleanup.tsv snapshot, ended_at_utc, touch .done, and exit "$install_rc".

Usage: apply_resource_caps.sh <apply|clear>. Purpose: keep the bake-off from starving co-resident work on a shared host by capping the node stack to roughly 8 cores / 36G total.

• cap_unit <unit> <prop...> guards on systemctl cat "$unit" existing (skips with a warning if the unit isn't defined yet) and applies each property with sudo systemctl set-property --runtime <unit> <prop> — --runtime means these are transient, gone on reboot, never written into the unit file.
• apply: eth1.service gets CPUQuota=600% MemoryMax=24G MemoryHigh=22G IOWeight=50; cl.service gets CPUQuota=200% MemoryMax=12G MemoryHigh=11G IOWeight=50.
• clear: resets both units' CPUQuota/MemoryMax/MemoryHigh/IOWeight back to infinity/infinity/infinity/100 — deliberately not systemctl revert, because these are custom units and revert can strip unit overrides/definitions entirely, not just the runtime properties this script set. In anchor mode, clear explicitly skips reverting eth1.service (ETH2QS_BAKEOFF_ANCHOR_EL set) so the anchor's caps stay consistent for the rest of the CL sweep; only cl.service reverts.

Sourced by run_candidate.sh and run_anchor_rotation.sh. Never executed directly.

The canonical list of every client datadir the harness knows about ($HOME/.ethereum, $HOME/.local/share/nethermind, … through $HOME/ethgas). This array must stay in sync with the DATA_DIRS arrays in install/utils/purge_ethereum_data.sh — test_data_dirs_sync.sh is a CI guard that fails the build if they drift (it diffs the two lists rather than sourcing purge_ethereum_data.sh, since sourcing it would invoke main and run the actual purge).

• bakeoff_snapshot_disk <outfile> — for every path in BAKEOFF_DATA_DIRS that exists, writes path\tbytes\thuman using du -sb/du -sh, then a final filesystem:/ row from df -B1 /. The du calls are 2>/dev/null with || true after the awk — du can exit non-zero on a file that vanishes mid-walk (a live datadir churning during snapshot), and || true keeps the number that was printed instead of letting pipefail propagate a fatal exit into an unguarded caller.
• bakeoff_probe_execution_sync — curl -sS --max-time 5 to http://127.0.0.1:8545 with an eth_syncing JSON-RPC payload; on a curl connection failure, prints {"error":"execution_rpc_unavailable"} instead of failing. There's no --fail flag, so an HTTP-error response with a non-JSON body (e.g. an HTML error page) is not caught by this fallback — the actual guard against malformed output downstream is bakeoff_write_sample's fromjson? // {raw: ...} fallback (§5.4).
• bakeoff_probe_beacon_sync — tries four ports in order (3500, 5051, 5052, 9596 — the different consensus clients' REST API defaults) against /eth/v1/node/syncing, returns the first non-empty body; falls back to {"error":"beacon_rest_unavailable"}.
• bakeoff_snapshot_processes — ps -eo pid=,comm=,%cpu=,%mem=,rss=,vsz=,etime=,args= piped to an awk regex over known client names; the pattern matches against the whole formatted ps row (comm and args columns together), not just the comm field — so any process whose arguments happen to contain one of those tokens is picked up too, not only ones named after a client binary. Matching lines are converted to a JSON array via jq -R -s 'split("\n")[:-1]'.
• bakeoff_services_alive — systemctl is-active --quiet eth1.service && systemctl is-active --quiet cl.service.

Returns 0 only when both layers report caught-up:

• Beacon: .data present, sync_distance <= 4, is_optimistic == false, el_offline == false.
• Execution: eth_syncing result is boolean false, OR a progress object where currentBlock == highestBlock and highestBlock != "0x0" (the != "0x0" guard rejects the pre-sync zero state, where an EL that hasn't started downloading anything yet would otherwise read as trivially “caught up”).

The per-tick sampling routine. Writes disk/execution-sync/beacon-sync/process snapshots plus ./scripts/eth2qs.sh doctor --json and stats --json (each timeout 30, best-effort) into a tmp/ scratch dir, determines alive="up"/"down" via bakeoff_services_alive, and assembles one JSON object per call with jq -cn --rawfile (each raw file parsed with fromjson? // {raw: ...} so a malformed sub-output degrades to a raw-string field instead of aborting the whole sample — with one exception: processes degrades to an empty array (($processes | fromjson?) // []), not a {raw: ...} object, since it's already expected to be a JSON array) appended as one line to samples.jsonl. A jq failure on the whole assembly is caught and logged (log_warn) rather than propagated — one bad sample should never kill the observation loop.

The config-optimality gate — arguably the harness's most important correctness mechanism, since a disk-footprint benchmark is meaningless if you can't prove the client was actually running in its most disk-efficient mode. Non-blocking (always returns 0), but stamps a verdict that later filters the results tables.

Mechanism:

1. Pull the configured ExecStart line from each systemd unit: systemctl cat eth1.service | grep -i ExecStart (same for cl.service). systemctl cat reads the unit file text (what the service is configured to run, including drop-ins) — not the live process's actual command line — so this reflects the configuration, not a runtime introspection of the running process.
2. Extract the first config file path referenced on that line — the extraction pipes through grep -oP | head -1, so only the first match is kept even if more than one flag appears on the line. Three flag styles are matched in one grep -oP alternation: --config-file=/--config-file , --rcConfig=/--rcConfig (lodestar), and --config=/--config (erigon). The comment in the source notes why the --config alternative doesn't falsely match a --config-file=x line: PCRE lookbehind is position-anchored, so (?<=--config[= ]) only fires when the character immediately after --config is = or a space — for --config-file=x that character is -, so the lookbehind never fires there.
3. If the extracted path exists as a file, its contents are appended to the “combined” haystack alongside the ExecStart line — so tokens can live in either the command line or the config file.
4. _has_token <label> <pattern> <haystack> runs grep -qP -- "$pattern" "$haystack"; the -- before the pattern is required because several patterns start with a dash (e.g. --prune-storage) and without -- those would be parsed as a grep option and silently swallowed by the 2>/dev/null as a false “miss.”
5. Per-client token table (kept current with what the install scripts actually emit):

   Client	Token checked
   geth	history.chain present
   nethermind	AncientBodiesBarrier present and PivotNumber extracted and > 0 (a zero pivot means snap sync is inert) — the pivot regex handles Nethermind's quoted-JSON form ("PivotNumber": 15900000,) with an optional closing quote between the key and the separator
   erigon	prune.mode set to full, matching both YAML-colon (prune.mode: "full") and = separator styles, quotes optional
   reth	both --prune.bodies.pre-merge and --prune.receipts.pre-merge present
   besu	history-expiry-prune=true
   nimbus_eth1	prune = true
   ethrex	no history-prune lever exists, so “optimal” is redefined as --syncmode snap being present
   prysm	beacon-db-pruning (flag or config-key form)
   lighthouse	checkpoint-sync-url present (default pruning behavior is already fine)
   teku	data-storage-mode set to minimal or prune (leading letter either case)
   nimbus (CL)	history = prune in the persistent config — deliberately not checking trustedNodeSync/--trusted-node-url, because those only appear in the one-shot bootstrap subcommand that runs before the service starts, never in the persistent ExecStart or nimbus.toml, so they're unobservable here and out of scope for this “is the running config optimal” gate
   lodestar	pruneHistory=true (or chain.pruneHistory)
   grandine	--prune-storage present

6. Any client not in the table falls through to unchecked (counted as found, not a miss) — the gate is additive; adding a new client without a token entry doesn't silently fail existing runs.
7. Writes config_optimal=yes|no and config_optimal_detail=found=...;missed=... into $out_dir/env.txt. On a miss, also touch $out_dir/.config-not-optimal and log_error.

This is the mechanism behind the “Superseded — non-optimal config” and “optimal config only” table split in summarize.sh's generated results skeleton (§6) — a row only counts toward the ranked results if every applicable token for its EL and CL was found.

Usage: run_anchor_rotation.sh <anchors_csv> [<cls_csv>] (CLs default to lighthouse,teku,nimbus,lodestar,grandine). This is how the CL matrix gets measured against more than one EL anchor without re-syncing every EL from scratch for every CL.

Per anchor in <anchors_csv>:

1. Establish: unset ETH2QS_BAKEOFF_ANCHOR_EL, set ETH2QS_BAKEOFF_KEEP_EL=yes, call run_candidate.sh <anchor> prysm — a normal full-mode candidate run (wipes any prior EL, installs fresh, syncs to tip) except that teardown preserves the EL datadir instead of purging it. The script also overrides ETH2QS_BAKEOFF_SYNC_WINDOW_SECONDS to 259200 (72h) by default even for what looks like a “just establish it” step — the source comment explains why: run_candidate.sh's own default window is 5400s (a 90-minute smoke-test default), which would make a genuine multi-hour EL sync report fully_synced=no and silently skip the entire CL sweep for that anchor.
2. Tip check: reads artifacts/<run-id>/<anchor>__prysm/env.txt and requires fully_synced=yes. If missing or not yes, logs a warning and skips the CL sweep for this anchor entirely (continue) rather than running a CL sweep against an EL that never reached head.
3. CL sweep: exports ETH2QS_BAKEOFF_ANCHOR_EL=<anchor>, then calls run_candidate.sh <anchor> <cl> for every CL in <cls_csv> — each invocation runs in anchor mode (§3.1), purging only the CL datadir between candidates and reusing the live, already-synced EL.
4. Unsets ETH2QS_BAKEOFF_ANCHOR_EL and moves to the next anchor in the list — establishing the next anchor's run_candidate.sh call will purge the previous anchor's EL datadir as part of its own normal pre-install cleanup, which is the “rotation.”

Same ETH2QS_BAKEOFF_CONFIRMED=yes gate as run_candidate.sh. All results land in the same artifacts/<run-id>/<el>__<cl>/ layout as a direct run_candidate.sh call, so summarize.sh needs no special-casing for anchor-mode rows. Ends by calling summarize.sh itself.

Reads every artifacts/<run-id>/<el>__<cl>/ directory and produces three outputs, none of which overwrite the hand-curated docs/CLIENT_BAKEOFF_RESULTS.md (that file is the blog's source of truth and this script never touches it):

• summary.csv — one row per candidate:

  pair,execution,consensus,install_exit_code,crash,sample_count,last_doctor_status,
  last_disk_bytes,residual_bytes,config_optimal,config_optimal_detail,fully_synced,
  sync_duration,sync_only,last_el_block,el_bytes,cl_bytes,anchor_synced

  • Sync-time derivation: _epoch converts started_at_utc/synced_at_utc ISO-8601 stamps to epoch seconds (date -u -d ... +%s); their difference is total_s. _install_wall_s parses the Elapsed (wall clock) line out of /usr/bin/time -v's install-time.txt (handles both h:mm:ss and m:ss.dd forms). sync_only_s = total_s - install_s isolates sync time from install time; negative results are discarded (clamped to empty) rather than shown as a nonsense negative duration. _fmt_hm renders seconds as XhYYm.
  • Last EL block: hex-decodes execution_sync.result.currentBlock from the last samples.jsonl line — a progress proxy even for candidates that hit the 72h cap without finishing.
  • Per-layer disk split: el_bytes/cl_bytes partition the chosen disk TSV (disk-synced.tsv if synced, else disk-final.tsv) by basename match against EL/CL basenames extracted from install/utils/purge_ethereum_data.sh's EL_DATA_DIRS/CL_DATA_DIRS arrays via awk+gensub — exact basename matching so e.g. nimbus-eth1 and nimbus are never confused.
• process-summary.csv — per-pair count of process-telemetry rows captured across the run (jq -r '.processes[]?' count from samples.jsonl).
• report.md — a human-readable dump: the summary.csv table (via column -s, -t) plus every candidate's findings.md, concatenated.
• A generated results skeleton at $artifact_root/CLIENT_BAKEOFF_RESULTS.generated.md (gitignored — never the committed docs/CLIENT_BAKEOFF_RESULTS.md), row-split into three sections by the config_optimal column:
  • “Results (optimal config only)” — config_optimal=yes and anchor_synced != no.
  • “Superseded — non-optimal config” — config_optimal=no, explicitly excluded from ranking with a note that footprints aren't comparable.
  • “Pre-gate — unknown config” — no verdict recorded at all (pre-gate or triage-only rows).

  Plus placeholder ## Recommendation / ## Final synced disk footprint / ## Changes driven by this bake-off sections with HTML comments telling the human reviewer what to fill in — this file is a skeleton for the curated doc, not a replacement for it.

These are defensive patterns baked into the harness as a result of real failures during the campaign (see CLIENT_BAKEOFF_ISSUES_LOG.md for the incidents that motivated them):

• du/pipefail resilience (bakeoff_snapshot_disk): du -sb ... || true survives a datadir file vanishing mid-walk during a live sync.
• Capped-path capture (_has_token in bakeoff_check_config_optimal): the -- before dash-prefixed patterns prevents grep from misparsing a pattern like --prune-storage as an option flag, which would otherwise exit 2 and get silently absorbed by 2>/dev/null into a false “miss.”
• </dev/null on every non-interactive command (clean-data, phase2 install): prevents SIGTTIN when the harness runs detached from a controlling terminal (background process group).
• Stall watchdog (ETH2QS_BAKEOFF_STALL_RESTART): entirely opt-in and inert by default; bounded restart count so a genuinely dead client still terminates the observation loop with a .stalled marker rather than looping until the window cap.
• Anchor-poison detection is read-only: the watchdog that guards a shared anchor EL during a CL sweep only ever marks state (.anchor-poisoned) and logs — it never intervenes, because restarting or killing a shared anchor would invalidate every remaining candidate in the rotation, not just the current one.
• FORCE-rerun poison-marker clear: ETH2QS_BAKEOFF_FORCE=yes clears .anchor-poisoned and .done together — without this, a stale poison marker from a previous attempt would silently survive a forced rerun and falsely finalize anchor_synced=no for an otherwise clean run.
• Two-consecutive-sample sync confirmation (synced_streak -ge 2 in run_candidate.sh, synced_streak local to the observation loop): guards against a single racy sample where the EL and CL heads transiently appear caught up.

• HOW_WE_TESTED_WITH_CLAUDE.md — the agent-orchestration model, the 23-day timeline, and the war stories behind the hardening fixes above.
• CLIENT_BAKEOFF_RESULTS.md — the numbers, source-of-truth.
• CLIENT_BAKEOFF_BLOG.md — the narrative writeup of the findings.
• CLIENT_BAKEOFF_ISSUES_LOG.md — the incident log that motivated several of the hardening fixes above.