Bake-off results — the raw data

• Baseline-anchored coverage (12 candidates): every execution client vs fixed Prysm (7 ELs), plus every other consensus client vs a fixed execution anchor (5 CLs). This isolates each client against a known-good counterpart instead of testing every N×M pair.
  • ELs × prysm: geth, erigon, reth, nethermind, besu, nimbus_eth1, ethrex
  • CLs × fixed anchor EL: lighthouse, teku, nimbus, lodestar, grandine. The first sweep used ethrex, already synced at tip. The originally planned geth sweep was initially deferred, then completed on 2026-07-08 as a cross-anchor check, and a third sweep against a nethermind anchor followed on 2026-07-26. Across all three anchors the same three tiers reproduce — lightweight {lodestar, lighthouse}, mid {teku, grandine}, heavy {nimbus} — while the order within each pair is measurement-window-sensitive (lodestar↔lighthouse between ethrex and geth; on the nethermind anchor, teku itself moved ~27% across two runs — ~667 MiB → ~848 MiB — enough to cross grandine's ~730 MiB; grandine < teku holds on all three anchors, so this is teku's own re-read variance, not a genuine swap with grandine).
• Two stages:
  • Stage A — triage (this doc): does each candidate install, checkpoint-sync, and authenticate the Engine API? The June triage runs used a ~5-min (300s) observation window per candidate, 60s sampling (one candidate, geth__grandine, ran 15-min/120s); the published harness now defaults `--stage=triage` to a more generous 90-min (5400s) window, 120s sampling.
  • Stage B — full sync (complete): each candidate reached a final synced, capped, or no-sync verdict; synced disk footprints are recorded below.
• Execution: strictly sequential, ONE candidate at a time on this shared semi-prod host. Resource-capped to protect co-resident agents. MEV: none. No validator keys. Destructive data-clean gated by ETH2QS_BAKEOFF_CONFIRMED=yes (secrets/validator material preserved).
• Pass criterion (Stage A): beacon head_slot reaches the network tip (~14.6M) via checkpoint import on the first sample (is_optimistic=true), el_offline=false (Engine-API JWT handshake succeeded), and sync_distance trending to 0 — i.e. the CL is live-tracking a validating EL.

el_offline is Prysm's own verdict on whether the EL is reachable and authenticating over the Engine API. False across the window = JWT wired correctly and the EL is validating payloads. restErr = beacon REST momentarily unavailable during heavy-client startup (see Resource contention below).

All 12: install_exit_code=0, no service crash, is_optimistic=true, checkpoint-sync PASS signature.

• erigon — PASS with Stage-B re-verify flag. Head stayed frozen at the checkpoint slot (14615808) and sync_distance grew 71→97 over the 5-min window. This is benign warmup, not a defect: el_offline=false throughout (Engine API reachable and authenticating), so the freeze is beacon-P2P backfill catching up, not a broken EL handshake. Re-verify head advances under Stage B's longer window.
• reth — the only EL that needed installer fixes. reth was the sole EL not passing --authrpc.jwtsecret to the shared secret that ensure_jwt_secret already created, so Prysm fell back to a non-shared auto-JWT → 401 → frozen head (el_offline=True early samples). Two commits fixed it; el_offline flips True→False once the shared JWT is wired. Also enabled HTTP-RPC on 127.0.0.1 for monitoring/consumers, matching the other ELs.
• besu / nimbus_eth1 / ethrex — clean PASS, no fix. JWT wiring correct out of the box. ethrex reached finalization (finalizedEpoch=456756). nimbus_eth1 additionally runs its own background historical EL sync from genesis (independent of the Engine API path, which works immediately).

Heavier-client startups (non-geth ELs, and lodestar) showed Prysm's beacon REST briefly unavailable for the first 1–3 minutes (restErr counts above) before recovering — consistent with startup contending for CPU/IO against co-resident agents on this shared host. It did not block any checkpoint sync, but it is the headline risk for Stage B: a multi-day, IO-heavy full sync will compete with co-resident workloads. Stage B execution strategy (sequential vs. small parallel batches) must account for this.

Installer/harness fixes landed on the bake-off branch as a direct result of triage:

• fix(reth): wire shared JWT + explicit authrpc/datadir for Engine API
• fix(reth): enable HTTP-RPC on 127.0.0.1 (eth,net,web3) for monitoring and consumers
• fix(nethermind): drop Engine from main JsonRpc.EnabledModules
• fix(lodestar): load node options via --rcConfig not --paramsFile
• fix(nimbus): checkpoint-sync via trustedNodeSync bootstrap
• fix(teku): remove invalid config keys blocking beacon startup
• fix(bakeoff): authenticate GitHub release API via gh token to avoid rate limits
• fix(bakeoff): bound doctor/stats sampling calls with timeout 30s

Before letting the slow full-sync ELs run, we audited every execution client to confirm it uses the most disk- and time-efficient sync mode available — so the Stage B footprint numbers reflect each client's best configuration, not an accidental archive run. Trigger: geth's --history.chain postmerge flag (prunes pre-merge block history, a large disk saving). We verified it was on for the baseline, then checked the rest.

Net effect: all seven ELs now run their disk-optimal sync mode. Six were already correct out of the box; reth was archive-by-default and is the one change this audit produced. Footprint comparisons across ELs are therefore apples-to-apples on configuration (the snap-vs-full time asterisk from the method section still applies — full-sync ELs execute all ~25M blocks, so time-to-sync is not comparable to geth's snap baseline, but final footprint is).

Complete (run_id client-bakeoff-stageB-2026-06-23). Runs were sequential, one candidate at a time; every candidate now has a final synced, capped, or no-sync verdict. Footprint = final synced datadir size (EL + CL); secrets/validator material excluded.

Several ELs were captured at more than one point in their lifecycle, and reporting only one number produced the two wrong disk claims this page has had to correct so far (nethermind's pre-backfill ~251 GiB read as final; ethrex's mid-climb ~467 GiB read as still-growing). Where both a fresh (at-sync) and a steady-state (settled) figure exist, both are recorded below; where only one was ever captured, that's stated explicitly rather than left ambiguous.

nethermind and ethrex are the only two ELs where a fresh-sync number was captured meaningfully before the steady-state figure; geth and besu were only ever measured at their finished, steady-state size, and reth never finished within the 72h cap.

Rule going forward: every disk figure in this corpus states which lifecycle phase it came from. Use at snap-sync / fresh, steady-state / settled, or partial @cap, and say so even when only one phase was ever measured. A bare footprint is the specific mistake that produced both corrections above: an EL's number can move by 4x between the moment it reports synced and the moment it stops growing, so a figure without its phase is not a measurement anyone can compare or reproduce.

The CL matrix holds the execution client constant and cycles the consensus client, the mirror of the EL scorecard above. The constant anchor is ethrex (not geth as first planned): ethrex was already synced at mainnet tip from its EL run, so reusing it as the fixed anchor saved a multi-day re-sync. Because the EL and CL are decoupled across the Engine API (the CL datadir is <1% of the EL and does not depend on which EL it pairs with), the anchor choice does not bias the CL comparison. To prove that empirically rather than assert it, the full 5-CL sweep was subsequently re-run against a geth anchor (2026-07-08, run_id client-bakeoff-anchor-rotation-2026-07-07) — the cross-anchor confirmation is recorded below and reproduces the ranking. The ethrex anchor stayed active and eth_syncing=false (~468 GiB / 502 GB, never restarted) across all five runs; each run cycled only cl+validator.

All five CLs checkpoint-synced to a fully validating head in ~22–23 min, config_optimal=yes, anchor_synced=yes, service_crash_observed=no. Sync time is effectively tied (checkpoint sync dominates), so the CL datadir footprint is the differentiator (figures are du -sh binary units — MiB/GiB).

CL disk ranking (smaller = better, all config-optimal + checkpoint-synced): lighthouse (~739 MiB) < lodestar (~827 MiB) < grandine (~946 MiB) < teku (~2.1 GiB) < nimbus (~5.0 GiB).

• teku required a re-run. Its first attempt (pre-TEKU_CACHE=8192m) JVM-OOM-starved the shared host, took 64 min to sync, and briefly blipped the anchor → anchor_synced=no (recorded, discarded as env.txt.poisoned-run1). The re-run with TEKU_CACHE raised to 8192m (commit bf043aa) synced clean in 22 min with a healthy anchor. Lesson: teku's JVM heap must be sized generously on a shared host or its GC pressure spills onto co-resident services. The valid 2.1 GiB row is the re-run.
• All CL footprints are roughly <~1.1% of the ethrex anchor's ~468 GiB (502 GB) EL datadir (nimbus, the largest, is ~1.05%) → confirms EL/CL decoupling: consensus-client choice does not move the EL disk ranking, and vice-versa.

The same 5-CL sweep was re-run against a geth anchor to verify the ranking is not an artifact of the ethrex anchor. All five runs were config_optimal=yes, anchor_synced=yes, no service crash; each cycled only cl+validator against the preserved geth EL datadir (~1.13 TiB, never wiped).

geth-anchor CL disk ranking (actual disk, smaller = better): lodestar (~177 MiB) < lighthouse (~518 MiB) < grandine (~725 MiB actual) < teku (~936 MiB) < nimbus (~1.2 GiB).

A third run of the same 5-CL sweep was performed against a nethermind anchor (nethermind 1.39.2, the same instance recorded in the Stage B row above) to further stress-test the EL/CL decoupling claim beyond the ethrex↔geth check. The anchor EL was brought current to mainnet tip in ~2h04m before this sweep began — a catch-up, not a re-sync; its datadir was preserved untouched — and stayed at tip across all five runs; each run cycled only cl+validator. All five: fully_synced=yes, config_optimal=yes, service_crash_observed=no.

nethermind-anchor CL disk ranking (actual disk, smaller = better): lodestar (~178 MiB) < lighthouse (~470 MiB) < grandine (~730 MiB actual) < teku (~848 MiB) < nimbus (~1.3 GiB).

Caveats (mandatory — accuracy over a clean story):

1. lodestar's row is a clean re-read; the discarded first attempt is instructive. lodestar's first run on this anchor recorded ~76m14s — not a lodestar property at all: that run started while the anchor EL was still importing a ~2-day block gap (left by an unrelated lodestar crash-loop incident on this host), so lodestar's beacon could not report is_optimistic=false until the EL closed that gap. Once the anchor was back at head and the crash-loop's root cause was fixed, lodestar was re-measured from scratch: ~7m36s / 186,083,466 B (~178 MiB), anchor_synced=yes, config_optimal=yes — in line with the other four CLs and within ~1 MiB of its geth-anchor footprint (~177 MiB). The published row is the re-read; the superseded run is retained in the artifacts (nethermind__lodestar.gapped-run-76m) for provenance. That gapped run had itself recorded 259,578,455 B (~248 MiB) — ~40% above the clean re-read, another reason not to carry it forward.
2. teku finalized anchor_synced=no on both of its runs — a reproducible watchdog false positive, not an anchor problem. The full sample sequence of the re-read shows why: at 18:14:13Z the anchor was 56 blocks behind (currentBlock=0x18720c3 vs highestBlock=0x18720fb), and at 18:16:51Z both sides were transiently behind — the anchor by 21 blocks and teku itself at sync_distance=49, is_syncing=true. From 18:19:30Z onward every sample is clean on both sides (eth_syncing=false, sync_distance=0, is_optimistic=false), including at teku's own synced_at_utc of 18:22:08Z. So the anchor was healthy when the measurement was taken; the watchdog had already latched on the warm-up samples and never re-evaluated. The cause is teku's slow JVM warm-up: it leaves the anchor briefly undriven, the anchor's head lags, and the watchdog's two-consecutive-sample rule trips. That it reproduced on a deliberate clean re-read is what makes it a known harness limitation rather than a fluke — the watchdog should tolerate a bounded head lag, or only sample once the CL reports synced. Both runs' footprints are valid; the published row is the re-read (~848 MiB, against ~667 MiB on the first run).
3. grandine sparse DB, again: apparent du -sb (1,074,340,425 B) overstates real on-disk usage; actual on-disk usage is ~730 MiB, the fair number for ranking — the same caveat as the geth-anchor row above.

Cross-anchor verdict — the tiers reproduce, not an identical order (now three anchors: ethrex, geth, nethermind):

• nimbus is the largest CL on all three anchors — the one ranking that holds without exception.
• {lodestar, lighthouse} are the two smallest CLs on all three anchors, but which one is smallest is measurement-window-sensitive: lighthouse is smallest on the ethrex anchor; lodestar is smallest on both the geth and nethermind anchors.
• {teku, grandine} form a "mid" tier, and teku shows how soft within-tier ordering is. teku was measured twice on this same anchor and moved ~27%: ~667 MiB on the first run, ~848 MiB on a deliberate clean re-read — enough to flip it from below grandine (~730 MiB) to above it. Taking the re-read as authoritative, grandine < teku holds on all three anchors; the honest reading is that this pair's internal order is measurement-window-sensitive, not a stable client property.
• Absolute footprints scale with observation time, not just the EL anchor. The geth- and nethermind-anchor numbers are much smaller than the ethrex-anchor ones (e.g. nimbus ~1.2–1.3 GiB vs ~5.0 GiB) because those sweeps were measured minutes after checkpoint-sync (a fresh datadir), while the ethrex-anchor runs ran longer post-sync and had filled more of the blob-retention / state-history window. The tiers are anchor-independent here; exact within-tier order is measurement-window-sensitive.
• Net: three different EL anchors (ethrex, geth, nethermind) reproduce the same three tiers — lightweight {lodestar, lighthouse}, mid {teku, grandine}, heavy {nimbus} — empirically supporting EL/CL decoupling, without claiming an identical total order across anchors.
• grandine's byte-identical apparent size (1,074,340,425 B) on both the geth-anchor and nethermind-anchor runs is real, not a copy/paste — it's grandine's fixed ~1 GiB sparse pre-allocation plus deterministic metadata; the actual allocated sizes (~725 MiB vs ~730 MiB) differ as expected.

Lodestar

~827 / 177 / 178 MiB (ethrex / geth / nethermind anchor)

Lighthouse

~739 / 518 / 470 MiB (ethrex / geth / nethermind anchor)

Grandine

~946 / 725 / 730 MiB (ethrex / geth / nethermind anchor)

Teku

~2150 / 936 / 848 MiB (ethrex / geth / nethermind anchor)

Nimbus

~5120 / 1229 / 1331 MiB (ethrex / geth / nethermind anchor)

Measurement notes:

• grandine uses sparse DB files → its apparent du -sb byte count (1,074,340,425 B) overstates real on-disk usage. du -sh reports ~725 MiB actual on the geth anchor and ~730 MiB on the nethermind anchor; the actual figure is the fair number for ranking. The other four CLs had apparent ≈ actual.
• Harness fix 98a52d7 (belongs in PR #190): bakeoff_snapshot_disk guarded its du | awk pipeline with || true. Without it, when the live anchor EL churned its datadir during a snapshot, du hit a vanishing file → exit 1 → pipefail killed the run (this spuriously failed grandine's first attempt; the clean re-run above is authoritative).

At full post-merge history the ELs converge on disk (~1.0–1.2 TiB: geth 1.13, nethermind ~1.06, besu 1.08, reth ~1.1–1.2 projected), so there is no meaningful on-disk ranking to draw — size is config-determined. The clients are separated below on the axes that actually differ: sync speed and restart-resume behavior. This table does not mean “failed to sync” — besu in particular synced cleanly (see below); it's here because it hit an operational limitation (fragile to a CL outage) or, for reth/nimbus_eth1/erigon, never reached a finished, tip-synced datadir.

Everything below was measured live on 2026-07-29/30 against our own synced ethrex node (ethrex/v22.0.0-HEAD-aa6c5f04750595…), across several probes taken at different moments as the head advanced.

Q: Does ethrex serve a usable RPC?

A: Yes — for anything at or after the block you synced at. No — for anything before it. The dividing line is your sync point, not “wallet vs DeFi”.

A full dapp-frontend method sweep at latest all passed — eth_call, eth_getCode, eth_getStorageAt, eth_estimateGas, eth_getBalance, eth_getTransactionCount, eth_gasPrice, eth_feeHistory, eth_maxPriorityFeePerGas, web3_clientVersion, net_version. eth_sendRawTransaction is present and validates input (a deliberately malformed payload was rejected with Invalid params: InvalidLength). Current-state DeFi reads — swap quotes, balances, allowances, pool state — work normally.

We took a real contract-creation transaction from 17 blocks back — head ~25,646,566 at the time, block 25,646,549, tx 0xc909b51c…, contract 0x227efd38ef38a798ae5ec9af062c437ee4bbef35 — and every dapp read worked: eth_getCode returned 8,043 bytes of bytecode, plus eth_getStorageAt, eth_getTransactionByHash, eth_getTransactionReceipt, and eth_getLogs from its deploy block to latest.

So: deploy a contract today, read its state, and serve a dapp frontend from ethrex — all fine.

Every pre-merge probe returns null:

The cutoff is exactly the snap-sync pivot, probed to single-block precision: pivot−1 (25,634,444) → null, pivot+0 (25,634,445) → served. At this earlier probe (head ~25,639,228), the node held only 4,783 blocks (~16h of chain). That window grows forward as it imports but never extends backward — ethrex does not backfill.

eth_call succeeds at head−100 but fails at head−500 with Vm execution error: DB error: state root missing for block N. So historical state is roughly the last ~128 blocks (~25 minutes). “What was this balance at block X” does not work.

Consequently broken:

• Indexer/subgraph backfill
• Portfolio history
• Tax and accounting exports
• Historical charts
• "Show me my transaction from last month"
• Any analytics that replays past logs

One spec deviation worth calling out: ethrex's eth_getLogs requires a topics parameter — omitting it returns Expected parameter: topics is missing, while geth treats topics as optional. Conformant tooling can therefore fail even inside the window ethrex does serve.

What this means for the RPC setup we ship: we ship an nginx/Caddy RPC setup. On geth (--history.chain postmerge) that endpoint serves post-merge history properly. The same setup on ethrex answers current-state and wallet traffic fine but returns null/errors for anything historical — so it is not a drop-in public RPC if your users expect history. If exposing an endpoint is the goal, that's an independent reason to prefer geth or nethermind.

Why this connects to the disk numbers: this is precisely why ethrex's ~470–476 GiB plateau is not a disk win — the missing ~600 GiB is the history the other clients are storing. Footprint tracks what you retain.

Stage A established viability: all 12 client pairs installed, checkpoint-synced, and authenticated the Engine API on this host. Disk size converges once ELs carry full post-merge history, so it doesn't separate the field — production instead asks “will it survive restarts, upgrades, and weeks of uptime?” Under that operational lens the field narrows sharply — and the two layers tell opposite stories: the EL layer is where the operational risk lives; the CL layer looks solved on the axes we measured (sync and footprint; only prysm was restart-tested).

• There is no disk winner — the field converges. Every EL that carries full post-merge history lands at ~1.0–1.2 TiB (geth 1.13, nethermind ~1.06, besu 1.08, reth ~1.1–1.2 projected); on-disk size is set by history-retention config, a client-agnostic knob, not by client efficiency. That knob turns both ways: an operator who does not need historical RPC can turn retention down — Update 2026-08-03: nethermind ships minimal-history by default, a fresh sync landing at ~250–280 GiB (post-merge bodies and receipts dropped, state only) and staying there with no backfill, in exchange for serving no history (pre-sync blocks return null, like ethrex; set NETHERMIND_FULL_HISTORY=true on a fresh/rebuilt datadir to keep full post-merge history for a public RPC). Among the measured no-history configurations in this campaign, nethermind is the smallest staking node — ~250–280 GiB vs ethrex’s ~470 GiB (its flat-storage state is the more compact engine), and a floor geth cannot reach (it has no clean lever to drop post-merge history below ~1.1 TiB). It is not, however, “4× leaner than geth” — that would score a no-history node against a with-history one, the same apples-to-oranges the field-converges point warns against; the win is within the no-history tier, at the cost of serving no history. If you do need history, still pick on the axes that actually differ: snap-sync speed and restart-resume stability.
• geth and nethermind both cleared the full operational bar (snap-sync to a validating tip, clean restart-resume — geth measured 2026-07-10, nethermind measured 2026-08-01 — the two most battle-tested codebases). On disk they're on par (~1.06 vs ~1.13 TiB — the field converges there, as above). Choose geth — the conservative default, largest ecosystem, cleanest ~8h28m snap, and it resumes gracefully after downtime (imports missed blocks, keeps its datadir) — or nethermind to improve client diversity (minority client, compact flat-storage state; restart-resume is now directly measured and bisected: 2026-08-01→03, every gap from 12 min to ~35h resumed by plain block import — no re-snap, no cliff). If you run one EL for the long haul, run one of these two.
• besu is a viable enterprise third — it did snap-sync to a fully validated head, and its ~1.08 TiB is the same magnitude as geth/nethermind, not an outlier. The real asterisk is operational: its snap sync is fragile to a prolonged CL outage (a stalled CL ages the pivot out of the servable-state window → SnapSyncChainDownloader thread death, observed twice, un-recoverable). Runnable in a shop that keeps its CL current and watches the pivot; not a set-and-forget solo-staker pick.
• The rest each missed the bar for a specific, documented reason — not a blanket “bad client”:
  • ethrex — fastest cold sync in the whole field (~2h16m on v19.0.0; a later re-sync took 4h09m56s on v22.0.0 — different ethrex versions and different day/host state, not a regression trend), but the restart-cliff is a real operational weakness: a 26-minute/132-block downtime gap stalled instead of resuming, and measured 1.5–2-hour gaps triggered a full ~2-hour re-snap. Separately, its datadir plateaus at ~470–476 GiB (confirmed 2026-07-28→31: a +43 GiB/hr post-sync settling climb collapsed ~300× to +0.13 GiB/hr and drifted 470.2 → 475.5 GiB over ~42 hours) — the earlier ~467 GiB reading was this same plateau caught mid-climb, not unbounded growth. That doesn't make it a disk winner: it plateaus low only because it serves no history at all — a no-history node, not a pruned-comparable one — and on a state-only basis nethermind's ~226–230 GiB state is smaller still. Snap speed is a trap if the restart cliff isn't fixed — fast to stand up, painful to operate. Fast-moving young client — v19.0.0 at first sync, v22.0.0 by the 2026-07-28 steady-state run; may improve further.
  • reth, nimbus_eth1 — full-sync-only (no snap) in the mode we tested (multi-day, capped partial here); can't reach tip inside a practical window on this host. This is a time-to-sync limit under our snap-to-tip bar, not a verdict on the clients in every context (reth in particular is widely run elsewhere).
  • erigon — deadlocked against checkpoint-synced prysm on this host (structural, reproducible), so no synced datadir.
• Stage-A note: geth, besu, nimbus_eth1, ethrex passed Stage A with zero installer changes and zero REST contention — the cleanest out-of-the-box ELs against Prysm.

Every CL (lighthouse, lodestar, grandine, teku, nimbus) checkpoint-synced to a validating head in ~22–23 min, config_optimal=yes, zero crashes, against a live anchor. Unlike the EL layer, none of them failed — so the choice is footprint + preference, not survivability:

• Recommended: lighthouse — smallest on the ethrex-anchor sweep (~739 MiB; lodestar is actually smaller on the geth and nethermind anchors, down to ~177 MiB), checkpoint-syncs in ~22 min, blob pruning on by default. lodestar (~827 MiB) and grandine (~946 MiB, with --prune-storage) are close seconds; teku (~2.1 GiB) and nimbus (~5.0 GiB) are heavier.
  Disk order (the only differentiator): lighthouse (~739 MiB) < lodestar (~827 MiB) < grandine (~946 MiB) < teku (~2.1 GiB) < nimbus (~5.0 GiB).
• Two small operational caveats: teku needs a generously sized JVM heap on a shared host (undersized, its GC pressure spilled onto co-resident services and poisoned a first run); grandine needs --prune-storage or it stores every state. nimbus is simply the heaviest (~6.9× lighthouse) but otherwise clean.
• Cross-cutting CL lesson (learned from prysm, the constant anchor): keep the CL binary current. A stale prysm v7.1.5 pin stalled ~28h on a PeerDAS/data-column bug and is precisely what aged out besu's pivot. Binary freshness is an operational requirement, not a nicety.

Bottom line: on the EL side a real long-running node comes down to geth or nethermind (besu if you're an enterprise shop that keeps its CL healthy); on the CL side any of the five works, with lighthouse the lean default. Fast initial sync (ethrex) and small archive-context footprints do not by themselves make a client operationally viable — durability across restarts and uptime is the deciding axis, and that is an EL-layer problem.

• Stage-A triage is blind to a stalled EL. Triage only checks that the CL reaches tip and the Engine-API JWT handshake works. A node whose CL checkpoint-syncs optimistically PASSES triage even with 0 EL peers and a frozen execution head (nethermind hid a 13.3h zero-progress stall this way). A sync-health verdict must combine peer-count>0 + EL-head advancing + beacon sync_distance — never sync_distance alone.
• eth_syncing=false is a trap, not a done-signal. It returns false BOTH before snap-sync starts (no pivot yet) and after it finishes. The authoritative "synced" gate is prysm is_optimistic=false (EL validated the head payload). besu's eth_syncing also returns false mid-sync — same trap.
• A synced nethermind's eth_syncing returns an OBJECT, not boolean false (currentBlock==highestBlock). The bakeoff harness now treats the EL as synced on currentBlock==highestBlock, not only boolean false (commit 5e7a93d).
• besu snap sync is two tracks: block-import reaches head first (a premature “done” signal), but world-state download/heal (Bonsai) is the real bottleneck and where the footprint balloons.
• besu snap-sync deadlocks if the CL stalls long enough (stability finding, 2026-07-05). The besu pruned re-run deadlocked twice and was abandoned. Chain: a prysm v7.1.5 data-column-sidecar/PeerDAS bug stalled the CL ~28h (besu logged Execution engine not called in 120 seconds continuously) → with no forkchoiceUpdated driving it, besu's snap-sync pivot block aged out of the network's servable-state window (full nodes serve state for only ~128 recent blocks ≈ 25 min) → world-state heal became un-completable → besu threw java.lang.IllegalStateException: The pivot block number has not increased in SnapSyncChainDownloader.consumePivotUpdate, cancelled the download, and the downloader thread died without restarting. The process stayed alive and answered RPC while the sync engine was dead (datadir frozen, zero DB writes). A restart resumed on the SAME persisted stale pivot and re-deadlocked identically. Takeaways: keep the CL binary current before a long besu snap-sync (the stall came from a stale prysm pin); besu answering eth_blockNumber ≠ besu syncing (watch DB writes); and this is the prime motivation for the harness stall-watchdog — now implemented (#31, PR #190) as an opt-in watchdog: with ETH2QS_BAKEOFF_STALL_RESTART=yes, if the unit under test makes no forward progress (EL block number / CL head_slot flat) for ETH2QS_BAKEOFF_STALL_SAMPLES polls (default 10) it performs up to ETH2QS_BAKEOFF_STALL_MAX_RESTARTS bounded restarts (default 3) of only that unit, then marks the row .stalled and fails it instead of spinning to the 72h cap.
• ethrex restart cliff begins beyond ~25 min; longer measured gaps re-snap from scratch (operational cliff, v19.0.0). A routine restart with a ~1.5–2h gap made ethrex discard its fully-synced 286 GiB state and start a fresh snap sync from near-genesis (datadir collapsed 286 GiB → ~9 GiB → climbing; journal SNAP SYNC STARTED → PHASE 1/8: BLOCK HEADERS from ~198k/25.47M; eth_blockNumber=0x0 throughout). Root cause: after the gap ethrex's old head aged out of the network's ~128-block (~25 min) servable-state window, so when prysm drove forkchoiceUpdated to the current head, ethrex re-pivoted to a full snap rather than importing the missed gap — contrast geth, which resumes by importing the missed blocks and keeps its state. Two measured re-sync costs: ~2h16m (cold) + ~2h11m (post-downtime re-snap); the re-snapped datadir then rebuilt past the old 286 GiB. Blog through-line: a client that stops resuming beyond ~25 minutes and can full-re-sync on longer gaps is operationally painful — a strong candidate explanation for ethrex's ~0% adoption despite the field's fastest cold sync. Threshold precisely bracketed (2026-07-10 restart bisection). Controlled systemctl stop eth1 → wait → start runs with a live prysm driving forkchoice: gaps of 12 min / 68 blk, 20 min / 108 blk, and 23 min / 124 blk all resumed cleanly (ethrex imported the missed blocks, datadir intact, canonical head climbed back to tip), while a 26 min / 132 blk gap stuck — the canonical head froze (eth_blockNumber flat at the pre-stop block for 12+ min, eth_syncing.currentBlock=0x0), ethrex logged FCU head state not reachable from DB state … Starting sync toward head and Failed to fetch headers for sync head — peer(s) queried but did not serve headers, and the gap widened as the tip advanced (no datadir collapse within the 12-min watch — the stuck disconnected-head state is the onset that escalates to the full snap re-sync at larger gaps). So the cliff edge is ~128 blocks ≈ 24–25 min, matching the ~128-block servable window exactly: inside it peers still serve the gap headers and ethrex bridges; beyond it they don't, the head freezes, and the ~1.5–2h gap above drove the full datadir-collapse re-snap. Caveat: young client (v19.0.0, may improve); this does not change the recorded sync-time result (2h16m, captured at synced time).
• geth resumes gracefully after a ~52h (multi-day) downtime — measured 2026-07-10 (the positive contrast to ethrex). Restarted after a stop that had left it ~15,400 blocks / ~52h behind (eth_syncing.startingBlock=25,487,154 — not genesis, no snap-pivot reset), geth kept its full multi-hundred-GB datadir and caught up purely by sequential block-import with trie-diff application — journal Imported new chain segment … triediffs=… triedirty=… on every segment, not a re-snap. Throughout: no datadir collapse (contrast ethrex's 286 GiB → ~9 GiB), eth_syncing returned an import object (never 0x0), state healing ran to completion (healingTrienodes=0x0), and it converged back to the validating tip (eth_syncing=false at block 25,502,592). This is the resume profile you want for an EL you upgrade/restart regularly, and it is why geth clears the operational bar above where ethrex's re-snap cliff does not. (Wall-clock resume time not cleanly bounded on this shared host, so only the mechanism + datadir preservation are claimed.)
• nethermind's restart-resume is now measured AND bisected — 2026-08-01→03, EXP-A. First an opportunistic CL-outage catch-up (2026-08-01): a CL restart at 13:24:55Z left nethermind 10,607 blocks (~35h of chain) behind the external tip, and it closed the entire gap by ordinary block import in 35m09s (~302 blocks/min); the beacon dropped is_optimistic at 14:00:04Z, and the datadir grew 1.165 → 1.178 TB (+1.1% — exactly the imported bodies/receipts) with no state wipe, no re-snap. Then a controlled stop→wait→start bisection (2026-08-02→03) at 12 min / 30 min / 1 h / 4 h gaps (69 / 151 / 301 / 1,196 blocks): every rung resumed geth-style — ordinary Engine-API block import, no re-pivot, no snap/state-sync in the journal, NRestarts 0. The re-pivot tell stayed silent: the state dir moved ~1.0–1.3 MiB per imported block, constant across rungs (a re-snap would rewrite the whole ~238 GiB state dir). Resume time scales gently with gap (121s @ 12 min → 186s @ 1 h → 483s @ 4 h → 35m09s @ ~35 h), dominated by the CL re-syncing its missed slots, not by EL import. nethermind has no servable-window cliff — the direct contrast to ethrex's ~128-block cliff and besu's pivot-aging deadlock. A separate establish run (2026-07-31) snap-synced nethermind fresh in 1h52m51s (00:04:45Z→01:57:36Z, ~280 GiB at snap, pivot 25,649,064, zero restarts) — far faster than the ~14.5h Stage-B figure because the pivot was minutes-old and near-tip and network conditions differ; a second data point under different conditions, not a replacement for the Stage-B number. Artifacts: exp-lab runs exp-a-nethermind-restart-resume-2026-07-31 (env.txt, disk-synced.tsv, exp-a1-resume.tsv) and exp-a-bisection-2026-08-02 (RESULTS.md, per-rung rung-*/).
• prysm restarts cleanly from its own DB — measured, n=4 (2026-08-02→03, EXP-D). A deliberate 30-min CL stop (past ethrex's ~25-min cliff, the EL left undriven with its head frozen) resumed from the existing beacon DB and was back driving the EL at sync_distance=0 within ~2m44s of start — journal-proven no re-checkpoint-sync (checkpoint-sync: Origin checkpoint found in the database, ignoring checkpoint sync flags + Blockchain data already exists in DB). The three EXP-A bisection rungs (12 min / 1 h / 4 h) each repeated the same re-checkpoint-free resume, making four clean resume events. Scope honestly stated: prysm only — this earns "prysm resumes cleanly," not "the CL layer is solved" (the other CLs are untested for resume, and prysm's intermittent discovery-listener wedge simply didn't occur in these runs — absent, not disproven). The node was beacon-only (no validator keys), so the deliberate stops were zero-risk. Artifacts: exp-lab run exp-d-cl-restart-resume-2026-08-01 (EXP-D-FINDINGS.md, exp-d-prysm-resume.tsv).
• ethrex serves no history beyond its snap-sync pivot — measured live 2026-07-29, to single-block precision. Probing the live node (head ~25,639,228) against its snap pivot (block 25,634,445): eth_getBlockByNumber returns null at pivot−1 (25,634,444) but resolves cleanly at pivot+0 and pivot+200 — the servable window's back edge is exactly the pivot, not an approximation. That window held only 4,783 blocks (~16h of chain) at measurement time; it grows forward as new blocks arrive but never backfills — deep probes at blocks 1, 1,000,000, 21,600,000, and the merge block (15,537,394) all returned null. State is an even tighter window: eth_call succeeds at head−100 but fails at head−500 with Vm execution error: DB error: state root missing for block N — historical state is available for only the last ~128 blocks (~25 min), the same servable-state window that drives the restart cliff above. What this costs the RPC-endpoint feature this repo ships (nginx/Caddy in front of the client): current-state reads work fine on ethrex — eth_chainId, eth_gasPrice, eth_getBalance/eth_getTransactionCount @ latest, eth_call @ latest, and blocks/receipts/logs at or after the pivot all resolve (615 logs returned over a 2-block USDC range at head−1000), so wallet-style traffic (balances, current quotes, allowances) is fine. But any block/log/receipt before the pivot fails (Internal Error: Could not get body for block N) — effectively all of Ethereum history — breaking indexer/subgraph backfill, portfolio history, and tax/accounting exports. A geth endpoint with --history.chain postmerge serves that same history; ethrex does not, so it is not a drop-in replacement for a public DeFi-facing RPC. One more spec deviation inside the served window: ethrex's eth_getLogs requires topics (Expected parameter: topics is missing), where geth treats it as optional — conformant tooling that omits topics can break even on blocks ethrex does serve.
• Client distribution is a WEAK/NUANCED predictor of syncability. The tempting story — low/zero-share clients all struggle — is only half true. erigon (deadlock), reth & nimbus_eth1 (full-sync-only, can't finish in 72h) did struggle, but ethrex (~0% share, Lambda Class) synced FASTEST in the whole field (~2h16m). The real driver is snap-sync availability + client robustness (ethrex has both: snap + clock-based stale-pivot self-healing), not market share per se. Don't overclaim the correlation in the blog — ethrex's steady-state footprint (now measured: a ~470 GiB plateau) isn't a disk win, since it's a no-history node; speed remains its clean, settled claim.
• Loopback-P2P class of bug. besu AND nethermind both defaulted P2P advertising to 127.0.0.1 → degraded/zero peering. Fixed (remove loopback p2p-host / inject routable ExternalIp). geth/erigon/reth/ethrex/nimbus_eth1 bind externally by default.
• erigon3 OtterSync + checkpoint-synced prysm deadlock — the one structural no-sync (see the erigon row): EL head freezes behind tip while the beacon stays optimistic; neither issues the forkchoiceUpdated that would close the gap. Raising CPU caps advanced it ~5k blocks then re-froze.
• reth is --full-only here (archive was the disk-hostile default; switched to --full); sequential full block execution can't finish a mainnet sync inside the 72h cap.
• Sampler timestamp skew (~2h): samples label local CEST times as Z. Trust file mtime for wall-clock, not the sample's timestamp_utc string.