The Bake-off Harness — Function-Level Engineering Reference

test/bakeoff/
├── run_bakeoff.sh              # sequential multi-candidate orchestrator + resume guard
├── run_candidate.sh            # single-candidate driver: reset → install → caps → sample-to-verdict
├── run_anchor_rotation.sh      # multi-EL anchor-rotation driver (establish EL once, sweep CLs against it)
├── run_queue.sh                # async TSV rerun-queue drainer (waits for preconditions, then calls run_candidate.sh)
├── apply_resource_caps.sh      # systemd CPUQuota/MemoryMax/IOWeight apply|clear
├── lib.sh                      # shared probe/sample/alert/config-gate library (sourced, never executed)
├── summarize.sh                # aggregates artifacts/ into summary.csv, process-summary.csv, report.md, a results skeleton
├── candidates.tsv              # the manifest: <execution>\t<consensus> pairs to run
├── rerun_queue.tsv.example     # queue-file format reference for run_queue.sh
└── test_data_dirs_sync.sh      # CI guard: BAKEOFF_DATA_DIRS must match purge_ethereum_data.sh

The executable drivers (run_bakeoff.sh, run_candidate.sh, run_anchor_rotation.sh, run_queue.sh, apply_resource_caps.sh, summarize.sh) each set set -Eeuo pipefail and source lib/common_functions.sh for logging (log_info/log_warn/log_error). lib.sh is a sourced library, never executed directly. It inherits strict mode from its caller and doesn't source common_functions.sh itself. Its log_* calls resolve at runtime because its logging callers (run_candidate.sh, run_anchor_rotation.sh, run_queue.sh) source common_functions.sh first. The CI guard test_data_dirs_sync.sh only reads BAKEOFF_DATA_DIRS and never reaches a log_* call.

The script has three modes, selected by env vars:

Anchor mode checks these preconditions before changing anything:

• $execution must equal $ETH2QS_BAKEOFF_ANCHOR_EL; mismatch → exit 3.
• eth1.service must pass systemctl is-active.
• Engine API port 8551 must return an HTTP-shaped code (^[1-5][0-9]{2}$). Authentication-required 401 counts as responding.
• A bounded 5-retry/2s-sleep loop must see eth_syncing caught up before the CL sweep starts: result == false, or hex-decoded currentBlock >= hex-decoded highestBlock, with highestBlock != "0x0". This comparison is numeric; bakeoff_is_execution_synced (§5.4) requires exact hex-string equality.

If $out/.done exists and ETH2QS_BAKEOFF_FORCE isn't yes, the candidate is skipped entirely (exit 0). Setting ETH2QS_BAKEOFF_FORCE=yes also clears .anchor-poisoned, .crash-looped and .done before rerunning. A stale poison marker would bypass the anchor watchdog and falsely finalize anchor_synced=no for a clean rerun.

1. Export CI_E2E=true (unless the caller set it). The bake-off runs phase2 without a prior run_1, so run_2.sh's run_1-dependent post-install security validation — which expects active UFW and security_monitor — would exit 1 and abort every install. CI_E2E is the codebase's existing switch for a run_1-less phase2; it skips UFW setup but doesn't change the installed binary, config, datadir, or sync footprint.
2. Stop+disable the relevant services (eth1.service cl.service validator.service, or just cl.service validator.service in anchor mode) — 2>/dev/null || true, tolerant of units that don't exist yet.
3. Purge datadirs via ./scripts/eth2qs.sh clean-data — --dry-run first (logged, non-fatal), then --confirm (must succeed). Anchor mode passes --scope=consensus so the EL anchor is never touched. < /dev/null on every clean-data and install call prevents SIGTTIN when detached (backgrounded/tmux). The kernel stops a background process group that reads its controlling terminal (TTY). The script cannot catch this as an error.
4. bakeoff_snapshot_disk "$out/disk-before.tsv" — baseline footprint.
5. Disk-floor guard, checked via df -B1 --output=avail /: anchor mode requires ETH2QS_BAKEOFF_MIN_DISK_BYTES:-429496729600 (400 GiB — CL-only, since the EL already occupies disk), full mode requires :-1717986918400 (1.6 TiB — full EL+CL). Below the floor, the script aborts with exit 3 before starting an install that would fail hours later from ENOSPC.
6. ./scripts/eth2qs.sh plan|doctor|stats --json snapshots, best-effort (|| true).
7. Opportunistic GITHUB_TOKEN export from gh auth token if neither GITHUB_TOKEN nor GH_TOKEN is already set — avoids the 60/hr unauthenticated GitHub API rate limit during client-version lookups inside the install scripts.

timeout "$install_timeout" /usr/bin/time -v -o "$out/install-time.txt" \
  ./scripts/eth2qs.sh phase2 --execution="$execution" --consensus="$consensus" --mev=none \
  </dev/null > "$out/install.log" 2>&1

(Anchor mode omits --execution= since only the CL installs.) install_timeout defaults to 90m (ETH2QS_BAKEOFF_INSTALL_TIMEOUT). /usr/bin/time -v is what summarize.sh later parses for install wall-clock (see §8). The exit code is captured with set +e/set -e bracketing and written to env.txt as install_exit_code=.

After install: apply_resource_caps.sh apply (§4), then snapshots (doctor, stats, monitor export, debug --service eth1, debug --service cl, systemctl status eth1 cl validator), then the config-optimality gate, bakeoff_check_config_optimal (§5.6). This non-blocking gate stamps config_optimal=yes|no into env.txt and never aborts the run.

Skipped entirely if install_rc != 0. Otherwise a while loop runs until $(date +%s) >= end_at (end_at = now + window), sleeping interval seconds between iterations. Before the loop starts, the crash-loop watchdog (below) records a baseline systemctl show <unit> -p NRestarts --value for both eth1.service and cl.service, then checks only the units under test — both in full/establish mode, just cl.service in anchor mode (the pre-existing anchor eth1.service is out of scope for that row). Each iteration:

1. bakeoff_write_sample "$out" "$REPO_ROOT" — the sampling primitive (§5.5).
2. bakeoff_services_alive check — sets crashed=yes if either service unit is no longer active (this becomes service_crash_observed in env.txt, a hard fail signal downstream).
3. Crash-loop watchdog (always-on, not opt-in). It detects flapping: repeated exits and restarts under systemd's Restart=. A config error can cause immediate exits and respawns every few seconds. The stall watchdog detects no progress. Each iteration re-reads NRestarts for every unit under test and computes the delta from the pre-loop baseline. If any unit's delta exceeds ETH2QS_BAKEOFF_MAX_RESTARTS (default 20: well above healthy behavior, well below a real crash loop), the row is invalidated immediately: it touches $out/.crash-looped; writes crash_loop_detected=yes, crash_loop_unit=, and crash_loop_restarts= (the delta) to env.txt; logs an error; fires a bakeoff_advisor_alert (error, kind crash_loop, §5.2); and breaks the observation loop rather than waiting out the rest of the window. It never restarts services. Systemd already does that; the watchdog detects the loop and exits. Motivating incident: lodestar's cl.service crash-looped on Unknown argument: chain (exit 1, ~5s per cycle) and was respawned 20,892 times over ~2 days. Before this watchdog existed, the harness kept sampling to the full 72h window on generic “service is no longer active” warnings alone and needed human intervention to produce anything.
4. Anchor watchdog (anchor mode only, and only while .anchor-poisoned doesn't already exist): detects if the anchor EL silently dropped out of sync (service down, or eth_syncing no longer reporting caught-up). “Caught-up” allows the anchor to trail the network head by up to ETH2QS_BAKEOFF_ANCHOR_LAG_BLOCKS (default 128). Some ELs return an eth_syncing object at tip with highestBlock set to the network head. While a CL warms up, nothing drives forkchoice, so the anchor can legitimately trail a few blocks. Demanding an exact catch-up there poisoned healthy rows. A real re-snap drops currentBlock to ~0, so it still trips far inside the tolerance. Two consecutive misses (anchor_miss_streak -ge 2) touches .anchor-poisoned and logs an error. It never restarts or kills anything: killing the shared EL would invalidate every remaining CL candidate.
5. Stall watchdog (opt-in via ETH2QS_BAKEOFF_STALL_RESTART=yes, inert otherwise): tracks no_progress_streak — the CL's head_slot (or, if absent, a sync_distance-derived proxy) in anchor mode, or the EL's currentBlock (hex-decoded) otherwise. stall_samples (default 10) consecutive non-advancing samples triggers sudo systemctl restart on the stalled unit, up to stall_max_restarts (default 3) times; exceeding that touches .stalled, writes stall_restarts=/stall_failed=yes to env.txt, and breaks out of the sampling window early. The row is invalid past that point, but teardown still runs cleanly rather than hanging forever on a genuinely dead node.
6. Synced-streak check: bakeoff_is_synced (§5.4) must return true on two consecutive samples (synced_streak -ge 2) before the row is accepted as synced. One true reading can reflect a race between EL and CL heads. On that streak, it snapshots disk-synced.tsv, writes synced_at_utc=/fully_synced=yes, and breaks out of the sampling window early, before the 72h cap.

After any loop exit (crash-loop, exhausted stall watchdog, synced streak, or timeout), a final bakeoff_write_sample call captures the end state, then env.txt gets fully_synced=no if nothing set it, service_crash_observed=, and (anchor mode only) anchor_synced=yes|no depending on whether .anchor-poisoned exists. A window_capped_unsynced warn-level advisor alert (§5.2) fires only when install succeeded, the window expired without fully_synced=yes, and neither the crash-loop nor the stall watchdog already alerted this same row. This is the third and last of the three give-up paths, after crash-loop and stall, ensuring at most one give-up alert per dead candidate.

Teardown captures evidence and cleans up regardless of outcome:

1. Capture journalctl tails (-u eth1 -n 700, -u cl -n 700, -u validator -n 300) and ./scripts/eth2qs.sh repair preview. Seed findings.md with the install/crash verdict (Result: TBD-by-reviewer) for human completion.
2. bakeoff_snapshot_disk "$out/disk-final.tsv" records the pre-cleanup footprint, synced or capped. summarize.sh uses it when disk-synced.tsv is absent.
3. Run apply_resource_caps.sh clear, then clean-data with the pre-install scope. Record disk-after-cleanup.tsv and ended_at_utc; run touch .done.

Normally, teardown ends with exit "$install_rc". A crash-loop instead forces exit 4, preserving all cleanup and artifacts. Even install_exit_code=0 can precede later flapping; exit 0 would make run_bakeoff.sh/run_queue.sh record an invalid measurement as clean. Exit 4 differs from 2 (operator-confirm gate) and 3 (anchor precondition/disk-floor failure).

The canonical list of every client datadir the harness knows about ($HOME/.ethereum, $HOME/.local/share/nethermind, … through $HOME/ethgas). This array must stay in sync with the DATA_DIRS arrays in install/utils/purge_ethereum_data.sh — test_data_dirs_sync.sh is a CI guard that fails the build if they drift (it diffs the two lists rather than sourcing purge_ethereum_data.sh, since sourcing it would invoke main and run the actual purge).

Operators and supervising AI can tail -f one alert channel throughout a multi-hour run. It collects crash-loop, anchor-poison, stall, install-failure, window-capped-without-sync, and malformed/timed-out queue alerts.

• Args: severity (warn|error), candidate (the el__cl pair, or - when not candidate-scoped), kind (a short machine token — crash_loop, anchor_poisoned, stall, install_failed, window_capped_unsynced, queue_malformed_row, queue_precondition_timeout, queue_candidate_failed), detail (free text — may contain quotes, backslashes, or newlines).
• Side effects, in order: first, a single-line ADVISOR_ALERT severity=... kind=... candidate=... detail=... marker to stdout — printed before the JSONL write, so it survives even if the write below fails. Then, one JSON object (ts_utc, severity, run_id, candidate, kind, detail) appended to ${ETH2QS_BAKEOFF_ALERT_LOG:-./advisor-alerts.jsonl}. Both run_candidate.sh and run_queue.sh export ETH2QS_BAKEOFF_ALERT_LOG to the same $artifact_root/advisor-alerts.jsonl (keyed by ETH2QS_BAKEOFF_RUN_ID), so one file aggregates alerts for the whole run — including rows driven asynchronously through the queue.
• Never aborts the caller: always returns 0, even under set -Eeuo pipefail, even when the target directory is unwritable, even when jq is missing (falls back to manual shell string-escaping — backslashes escaped before quotes, since escaping quotes first would re-escape the backslash the quote-escape step just inserted), even when detail itself contains characters that would otherwise corrupt a hand-rolled JSON string.

• bakeoff_snapshot_disk <outfile> — for every path in BAKEOFF_DATA_DIRS that exists, writes path\tbytes\thuman using du -sb/du -sh, then a final filesystem:/ row from df -B1 /. The du calls are 2>/dev/null with || true after the awk — du can exit non-zero on a file that vanishes mid-walk (a live datadir churning during snapshot), and || true keeps the number that was printed instead of letting pipefail propagate a fatal exit into an unguarded caller.
• bakeoff_probe_execution_sync — curl -sS --max-time 5 to http://127.0.0.1:8545 with an eth_syncing JSON-RPC payload; on a curl connection failure, prints {"error":"execution_rpc_unavailable"} instead of failing. There's no --fail flag, so an HTTP-error response with a non-JSON body (e.g. an HTML error page) is not caught by this fallback — the actual guard against malformed output downstream is bakeoff_write_sample's fromjson? // {raw: ...} fallback (§5.5).
• bakeoff_probe_beacon_sync — tries four ports in order (3500, 5051, 5052, 9596 — the different consensus clients' REST API defaults) against /eth/v1/node/syncing, returns the first non-empty body; falls back to {"error":"beacon_rest_unavailable"}.
• bakeoff_snapshot_processes — ps -eo pid=,comm=,%cpu=,%mem=,rss=,vsz=,etime=,args= piped to an awk regex over known client names; the pattern matches against the whole formatted ps row (comm and args columns together), not just the comm field. Processes with matching arguments are included, even if their binary names differ. Matching lines are converted to a JSON array via jq -R -s 'split("\n")[:-1]'.
• bakeoff_services_alive — systemctl is-active --quiet eth1.service && systemctl is-active --quiet cl.service.

bakeoff_is_execution_synced checks the execution client alone. eth_syncing must return boolean false, or a progress object with currentBlock == highestBlock and highestBlock != "0x0". The != "0x0" guard rejects pre-sync zero state: an EL that hasn't started downloading would otherwise look trivially caught up.

Between anchor-mode candidates, the CL is stopped and its datadir purged. A beacon-inclusive check cannot pass. This separate function lets callers check the preserved EL alone; run_queue.sh's anchor precondition (§7) uses it directly.

bakeoff_is_synced returns 0 only when both layers report caught-up:

• Beacon: .data present, sync_distance <= 4, is_optimistic == false, el_offline == false.
• Execution: delegates to bakeoff_is_execution_synced above.

Each tick appends one JSON object to samples.jsonl:

1. Write disk, execution-sync, beacon-sync, and process snapshots into tmp/. Add ./scripts/eth2qs.sh doctor --json and stats --json, each best-effort with timeout 30.
2. Determine alive="up"/"down" through bakeoff_services_alive.
3. Assemble with jq -cn --rawfile. Malformed sub-outputs fall back to raw-string fields through fromjson? // {raw: ...}. Exception: processes expects an array, so (($processes | fromjson?) // []) produces [], rather than a {raw: ...} object.

An assembly-wide jq failure logs log_warn without propagating. One bad sample cannot kill the observation loop.

This gate checks disk-efficient configuration before a footprint enters the ranked results. It stamps each row with a verdict, always returns 0, and never blocks a run.

Mechanism:

1. Pull the configured ExecStart line from each systemd unit: systemctl cat eth1.service | grep -i ExecStart (same for cl.service). systemctl cat reads the unit file text (what the service is configured to run, including drop-ins) — not the live process's actual command line — so this reflects the configuration, not a runtime introspection of the running process.
2. Extract only the first referenced config path using grep -oP | head -1, even with multiple flags. One grep -oP alternation matches three styles: --config-file=/--config-file , --rcConfig=/--rcConfig (lodestar), and --config=/--config (erigon). As the source comment explains, PCRE lookbehind checks preceding text at a specific position. (?<=--config[= ]) requires = or a space immediately after --config. In --config-file=x, that character is -, so this alternative cannot falsely match.
3. If the extracted path exists as a file, its contents are appended to the “combined” haystack alongside the ExecStart line — so tokens can live in either the command line or the config file.
4. _has_token <label> <pattern> <haystack> runs echo "$haystack" | grep -qP -- "$pattern" — the haystack is the ExecStart line plus any referenced config file's contents, piped in as text, not a file path. The -- before the pattern is required because several patterns start with a dash (e.g. --prune-storage) and without -- those would be parsed as a grep option and silently swallowed by the 2>/dev/null as a false “miss.”
5. Per-client token table (kept current with what the install scripts actually emit):

   Client	Token checked
   geth	history.chain present
   nethermind	AncientBodiesBarrier present and PivotNumber extracted and > 0 (a zero pivot means snap sync is inert) — the pivot regex handles Nethermind's quoted-JSON form ("PivotNumber": 15900000,) with an optional closing quote between the key and the separator
   erigon	prune.mode set to full. Matches both YAML-colon (prune.mode: "full") and = separator styles, quotes optional
   reth	both --prune.bodies.pre-merge and --prune.receipts.pre-merge present
   besu	history-expiry-prune=true
   nimbus_eth1	prune = true
   ethrex	no history-prune lever exists, so “optimal” is redefined as --syncmode snap being present
   prysm	beacon-db-pruning (flag or config-key form)
   lighthouse	checkpoint-sync-url present (default pruning behavior is already fine)
   teku	data-storage-mode set to minimal or prune (leading letter either case)
   nimbus (CL)	history = prune in persistent config. trustedNodeSync/--trusted-node-url only appear in the one-shot bootstrap subcommand, before service startup. Neither appears in persistent ExecStart or nimbus.toml, so this running-config gate cannot observe them and excludes them.
   lodestar	pruneHistory=true (or chain.pruneHistory)
   grandine	--prune-storage present

6. Any client not in the table falls through to unchecked (counted as found, not a miss). Adding clients without token entries cannot silently fail existing runs.
7. Writes config_optimal=yes|no and config_optimal_detail=found=...;missed=... into $out_dir/env.txt. On a miss, also touch $out_dir/.config-not-optimal and log_error.

This is the mechanism behind the “Superseded — non-optimal config” and “optimal config only” table split in summarize.sh's generated results skeleton (§8) — a row only counts toward the ranked results if every applicable token for its EL and CL was found.

${ETH2QS_BAKEOFF_QUEUE_FILE:-test/bakeoff/rerun_queue.tsv}, TAB-separated, # comments and blank lines ignored:

# execution\tconsensus\treason  (TAB-separated; # comments and blank lines ignored)
geth\tprysm\trerun after extract_archive -o fix
nethermind\tlodestar\trerun: lodestar pruneHistory CLI-flag fix
erigon\tteku\trerun: previous row crash-looped before the watchdog existed

A row missing the consensus column is malformed: counted, logged, fires a queue_malformed_row warn alert (§5.2), and the drain moves on — one bad line never stops the queue. A trailing \r is stripped from the execution field (tolerates a queue file edited on/copied from a CRLF host).

Same operator gate as run_candidate.sh — ETH2QS_BAKEOFF_CONFIRMED=yes is required even for --dry-run: previewing the plan for a destructive harness still implies the operator has looked at this host. Before every row, polled every ETH2QS_BAKEOFF_QUEUE_POLL_SECONDS (default 60) up to a bounded ETH2QS_BAKEOFF_QUEUE_WAIT_SECONDS (default 7200) timeout:

1. No other candidate running: pgrep -f 'run_candidate\.sh', excluding this process's PID. This safety net is never load-bearing in practice: synchronous children already exit before the next row's check.
2. Anchor readiness: when ETH2QS_BAKEOFF_ANCHOR_EL is set, require active eth1.service and bakeoff_is_execution_synced. Like the candidate preflight (§3.1), this checks only the EL. The beacon-inclusive bakeoff_is_synced cannot pass between shared-anchor candidates (§5.4); using it would time out and skip every queued row.

A timeout skips the row (queue_precondition_timeout warn alert, logged to run_queue.log) and the drain continues to the next row.

Reruns default to FORCE=yes so existing .done markers cannot turn measurements into no-op successes. After preconditions pass, the invocation is ETH2QS_BAKEOFF_FORCE="${ETH2QS_BAKEOFF_QUEUE_FORCE:-yes}" run_candidate.sh <execution> <consensus>. ETH2QS_BAKEOFF_QUEUE_FORCE defaults to “yes”. Otherwise, run_candidate.sh exits 0 without measuring when .done exists and FORCE is unset (§3.2).

Forcing overwrites that pair's previous artifacts. Set ETH2QS_BAKEOFF_QUEUE_FORCE=no to skip completed pairs and preserve their measurements.

• Shares the same ETH2QS_BAKEOFF_ALERT_LOG channel as run_candidate.sh (§5.2), defaulting under the same artifacts/${ETH2QS_BAKEOFF_RUN_ID:-...}/ root, so one file captures alerts for the whole run whether a row came from the manifest or the queue.
• Every drained row appends one line to $artifact_root/run_queue.log. A row that actually ran logs timestamp, row number, pair, rc, and the queue row's free-text reason; a malformed row logs pair=- and result=malformed, and a timed-out row logs the pair and result=skipped_precondition_timeout — neither of those two carries the reason column.
• A failing row (run_candidate.sh exits non-zero) is recorded and fires a queue_candidate_failed error alert, but never aborts the drain — the loop always continues to the next row, mirroring run_bakeoff.sh's own tolerance of a single candidate's failure.
• --dry-run prints the plan — queue file path, which preconditions would be checked, and each row's execution/consensus/reason plus the exact run_candidate.sh invocation it would make — with no side effects.

• summary.csv — one row per candidate:

  pair,execution,consensus,install_exit_code,crash,sample_count,last_doctor_status,
  last_disk_bytes,residual_bytes,config_optimal,config_optimal_detail,fully_synced,
  sync_duration,sync_only,last_el_block,el_bytes,cl_bytes,anchor_synced,crash_loop_detected

  • Sync-time derivation: _epoch converts started_at_utc/synced_at_utc ISO-8601 stamps to epoch seconds (date -u -d ... +%s); their difference is total_s. _install_wall_s parses the Elapsed (wall clock) line out of /usr/bin/time -v's install-time.txt (handles both h:mm:ss and m:ss.dd forms). sync_only_s = total_s - install_s isolates sync time from install time; negative results are discarded (clamped to empty) rather than shown as a nonsense negative duration. _fmt_hm renders seconds as XhYYm.
  • Last EL block: hex-decodes execution_sync.result.currentBlock from the last samples.jsonl line — a progress proxy even for candidates that hit the 72h cap without finishing.
  • Per-layer disk split: el_bytes/cl_bytes partition the chosen disk TSV (disk-synced.tsv if synced, else disk-final.tsv) by basename match against EL/CL basenames extracted from install/utils/purge_ethereum_data.sh's EL_DATA_DIRS/CL_DATA_DIRS arrays via awk+gensub — exact basename matching so e.g. nimbus-eth1 and nimbus are never confused.
• process-summary.csv — per-pair count of process-telemetry rows captured across the run (jq -r '.processes[]?' count from samples.jsonl).
• report.md — a human-readable dump: the summary.csv table (via column -s, -t) plus every candidate's findings.md, concatenated.
• A generated results skeleton at $artifact_root/CLIENT_BAKEOFF_RESULTS.generated.md (gitignored — never the committed docs/CLIENT_BAKEOFF_RESULTS.md), row-split into three sections by the config_optimal column:
  • “Results (optimal config only)” — config_optimal=yes and anchor_synced != no.
  • “Superseded — non-optimal config” — config_optimal=no, explicitly excluded from ranking with a note that footprints aren't comparable.
  • “Pre-gate — unknown config” — no verdict recorded at all (pre-gate or triage-only rows).

  Plus placeholder ## Recommendation / ## Final synced disk footprint / ## Changes driven by this bake-off sections with HTML comments telling the human reviewer what to fill in.

• du/pipefail resilience (bakeoff_snapshot_disk): du -sb ... || true survives a datadir file vanishing mid-walk during a live sync.
• Capped-path capture (_has_token in bakeoff_check_config_optimal): the -- before dash-prefixed patterns prevents grep from misparsing a pattern like --prune-storage as an option flag, which would otherwise exit 2 and get silently absorbed by 2>/dev/null into a false “miss.”
• </dev/null on every non-interactive command (clean-data, phase2 install): prevents SIGTTIN when the harness runs detached from a controlling terminal (background process group).
• Stall watchdog (ETH2QS_BAKEOFF_STALL_RESTART): entirely opt-in and inert by default; bounded restart count so a genuinely dead client still terminates the observation loop with a .stalled marker rather than looping until the window cap.
• Crash-loop watchdog (§3.5): always on. A flapping unit looks active again seconds after systemd's Restart=, so bakeoff_services_alive can miss a crash if no sample lands during the brief down window. Cumulative, monotonic NRestarts catches them regardless of sample timing by comparison with the pre-loop baseline. Exceeding ETH2QS_BAKEOFF_MAX_RESTARTS (default 20) touches .crash-looped and forces exit 4, overriding a possibly-zero install_exit_code.
• Structured advisor-alert channel (bakeoff_advisor_alert, §5.2): every failure mode listed in §9's advisor-alerts.jsonl row funnels into one JSONL file plus a greppable stdout marker.
• Anchor-poison detection is read-only: the watchdog that guards a shared anchor EL during a CL sweep only ever marks state (.anchor-poisoned) and logs — it never intervenes, because restarting or killing a shared anchor would invalidate every remaining candidate in the rotation, not just the current one.
• FORCE-rerun poison-marker clear: ETH2QS_BAKEOFF_FORCE=yes clears .anchor-poisoned, .crash-looped and .done together — without this, a stale poison marker from a previous attempt would silently survive a forced rerun and falsely finalize anchor_synced=no for an otherwise clean run.
• Two-consecutive-sample sync confirmation (synced_streak -ge 2 in run_candidate.sh, synced_streak local to the observation loop): guards against a single racy sample where the EL and CL heads transiently appear caught up.
• Execution-only anchor precondition in the queue (run_queue.sh, §7.2): bakeoff_is_execution_synced, not the beacon-inclusive bakeoff_is_synced — see §5.4 for why a beacon-inclusive check on a shared anchor can never pass.