Raw results
Published Jul 21, 2026 · Updated Aug 6, 2026
Bake-off results — the raw data
Stage A triage: synthesized from raw campaign logs on 2026-06-23. This page is the committed summary.
Every triage row, Stage B disk measurement, three-anchor consensus matrix, client limitation, and campaign lesson from docs/CLIENT_BAKEOFF_RESULTS.md. Disk figures round to whole GB/MB. Charts and collapsible detail help scanning. For the narrative, read the bake-off blog post.
In 30 seconds
- 12 / 12 passed triage. Stage A, June 2026: all 7 execution clients against Prysm and 5 consensus clients against a fixed anchor passed. Each installed, loaded a recent beacon checkpoint, and authenticated the Engine API, the execution/consensus connection.
- Disk converges at ~1.0–1.2 TiB. At steady state, with full post-merge history: geth 1.13 TiB, nethermind ~1.06, besu 1.08. reth’s ~1.1–1.2 is projected from its partial 72h-capped run. Footprint is a config knob, not a winner axis.
- Pick on speed + restart-resume. geth and nethermind clear the operational bar; besu is a qualified enterprise third. ethrex syncs fastest but has a restart cliff.
- Six weeks, every figure phased. Six weeks: June triage → Stage B disk + consensus sweeps → steady-state and restart-resume follow-ups through August 2026. Every measurement states its lifecycle phase.
Method
Sequential testing isolates each candidate on a shared semi-prod host. No MEV; no validator keys. Execution clients (ELs) process transactions. Consensus clients (CLs) track the beacon chain. A shared JWT secret authenticates their connection.
Stage A — triage
12 / 12 PASS- When
- June 2026
- Window
- ~5-min window per candidate (90-min default in the shipped harness)
- Asks
- Does each candidate install, checkpoint-sync, and authenticate the Engine API?
Stage B — full sync
footprints + verdicts below- When
- Jun–Aug 2026
- Window
- up to 72 h per candidate, sequential
- Asks
- Does each candidate reach a synced, capped, or no-sync verdict, and at what disk footprint?
Full method detail — coverage, stages, execution, pass criterion
- Baseline-anchored coverage (12 candidates): 7 ELs against Prysm; 5 other CLs against a fixed EL. Each known-good counterpart isolates the candidate without testing every N×M pair.
- ELs × prysm: geth, erigon, reth, nethermind, besu, nimbus_eth1, ethrex
- CLs × fixed anchor EL: lighthouse, teku, nimbus, lodestar, grandine. First anchor: ethrex, already at tip. The planned geth sweep was deferred, then completed on 2026-07-08; nethermind followed on 2026-07-26. All reproduced lightweight {lodestar, lighthouse}, mid {teku, grandine}, heavy {nimbus} tiers. Within-pair order depends on measurement time: lodestar↔lighthouse switched between ethrex/geth. On nethermind, teku varied ~25% (~700 → ~875 MB), crossing grandine’s ~765 MB. Published grandine < teku holds on every anchor; this is teku re-read variance, not an observed ranking swap.
- Two stages:
- Stage A — triage (this doc): does each candidate install, checkpoint-sync, and authenticate the Engine API? June runs observed each candidate for ~5 min (300s), sampling every 60s. geth__grandine used 15-min/120s. The shipped harness now defaults `--stage=triage` to 90 min (5400s), sampling every 120s.
- Stage B — full sync (complete): each candidate reached a final synced, capped, or no-sync verdict; synced disk footprints are recorded below.
- Execution: one candidate at a time, resource-capped to protect co-resident agents. No MEV or validator keys. Destructive data-clean requires
ETH2QS_BAKEOFF_CONFIRMED=yes(secrets/validator material preserved). - Pass criterion (Stage A): beacon
head_slotreaches network tip (~14.6M) on the first checkpoint-import sample (is_optimistic=true).el_offline=falseconfirms Engine-API JWT authentication;sync_distancetrends to 0. This is the signature used for CL live-tracking of a validating EL; see the stalled-EL caveat below.
Stage A results — 12/12 PASS
All 12 pairs passed triage. Prysm’s el_offline checks EL reachability and Engine API authentication. False throughout means correct JWT wiring and EL payload validation. restErr counts brief beacon REST outages during heavy-client startup; see Resource contention.
All 12 Stage A rows — slots, samples, errors, and installer fixes
geth__prysmPASS (baseline)- Install
- 0
- Crash
- no
- head (first→last)
- 14615771→14615856
- el_offline
- F×5
- restErr
- 0
- n
- 5
- Installer fix
- none
erigon__prysmPASS — flagged- Install
- 0
- Crash
- no
- head (first→last)
- 14615808→14615808
- el_offline
- F×5
- restErr
- 0
- n
- 5
- Installer fix
- none
reth__prysmPASS (after fix)- Install
- 0
- Crash
- no
- head (first→last)
- 14615808→14616097
- el_offline
- F×5 / T×4
- restErr
- 6
- n
- 9
- Installer fix
- JWT + HTTP-RPC
nethermind__prysmPASS- Install
- 0
- Crash
- no
- head (first→last)
- 14614859→14615786
- el_offline
- F×9
- restErr
- 4
- n
- 9
- Installer fix
- Engine module
besu__prysmPASS (clean)- Install
- 0
- Crash
- no
- head (first→last)
- 14616094→14616183
- el_offline
- F×5
- restErr
- 0
- n
- 5
- Installer fix
- none
nimbus_eth1__prysmPASS (clean)- Install
- 0
- Crash
- no
- head (first→last)
- 14616151→14616235
- el_offline
- F×5
- restErr
- 0
- n
- 5
- Installer fix
- none
ethrex__prysmPASS (clean)- Install
- 0
- Crash
- no
- head (first→last)
- 14616192→14616287
- el_offline
- F×5
- restErr
- 0
- n
- 5
- Installer fix
- none
geth__lighthousePASS- Install
- 0
- Crash
- no
- head (first→last)
- 14615232→14615349
- el_offline
- F×4
- restErr
- 1
- n
- 4
- Installer fix
- none
geth__tekuPASS- Install
- 0
- Crash
- no
- head (first→last)
- 14615296→14615326
- el_offline
- F×4
- restErr
- 1
- n
- 4
- Installer fix
- config keys
geth__nimbusPASS- Install
- 0
- Crash
- no
- head (first→last)
- 14615455→14615506
- el_offline
- F×5
- restErr
- 0
- n
- 5
- Installer fix
- trustedNodeSync
geth__lodestarPASS- Install
- 0
- Crash
- no
- head (first→last)
- 14615638→14615708
- el_offline
- F×10
- restErr
- 10
- n
- 10
- Installer fix
- rcConfig
geth__grandinePASS- Install
- 0
- Crash
- no
- head (first→last)
- 14615040→14615203
- el_offline
- F×7
- restErr
- 1
- n
- 7
- Installer fix
- none
All 12: install_exit_code=0, no service crash, is_optimistic=true, checkpoint-sync PASS signature.
Per-candidate notes
- erigon — PASS; re-check in Stage B. Head froze at checkpoint slot 14615808 while
sync_distancegrew 71→97 over 5 min.el_offline=falsethroughout confirms Engine API reachability and authentication. Beacon-P2P backfill explains this benign warmup, rather than an EL handshake defect. Verify advancing head over Stage B’s longer window. - reth alone needed EL installer fixes. It omitted
--authrpc.jwtsecretfor the shared secret fromensure_jwt_secret. Prysm used a non-shared auto-JWT: 401 errors froze head, with earlyel_offline=True. Two commits wired the shared JWT;el_offlineflipped True→False. HTTP-RPC was also enabled on 127.0.0.1 for monitoring/consumers, matching other ELs. - besu / nimbus_eth1 / ethrex passed cleanly without fixes. JWT wiring worked out of the box. ethrex finalized (
finalizedEpoch=456756). nimbus_eth1 also syncs historical EL blocks from genesis in the background, independently of its immediately working Engine API.
Resource contention
Startup contention caused brief REST outages, without blocking checkpoint sync. Non-geth ELs and lodestar left Prysm’s beacon REST unavailable for 1–3 minutes (restErr above), then it recovered. CPU/IO competition with co-resident agents fits this shared-host behavior. This is the headline risk for Stage B: multi-day, IO-heavy syncs compete with co-resident workloads. Execution strategy must account for it when choosing sequential runs or small parallel batches.
Client limitations
Successful sync can still leave operational limits. besu risks stalled snap sync; ethrex has history and restart limits. reth/nimbus_eth1 exceeded the time cap; erigon deadlocked.
Per-client limitations — footprints, sync status, and causes
| EL | Footprint recorded | Synced? | Why it's listed here |
|---|---|---|---|
| besu | ~1.08 TiB | yes (~19h18m, fully validated) | besu synced fine. Its caveat is prolonged CL outages during snap sync. A prune-lever re-run deadlocked twice after SnapSyncChainDownloader thread death, caused by a stale CL stall, not besu. See gotchas and Stage B. |
| reth | ~0.98 TiB partial (72h-capped) | partial | --full-only (no snap); sequential full block execution can't finish mainnet inside the 72h cap. Speed-bound, not config-bound. |
| nimbus_eth1 | ~40 GB partial @72h cap (2026-07-13; supersedes an earlier ~21 GB aborted run) | partial (~21.6%) | Full-sync-only, no snap: a stable 72h continuous run with zero restarts (2026-07-11 to 07-13, 20–25 peers) reached only ~21.6% of tip. Continuous pruning logs confirmed prune = true works online, resolving that earlier question. The limit is practical sync time on this host, not pruning. |
| erigon | ~1.21 TiB frozen partial | no | Structural no-sync: erigon3 OtterSync + checkpoint-synced-prysm optimistic gap-close deadlock. Not a synced datadir. |
| ethrex | ~286–300 GiB at sync (fresh) → ~470–476 GiB steady-state plateau (drifting 470.2 → 475.5 GiB over ~42h) | yes (~2h16m on v19.0.0, fully validated; a later re-sync on v22.0.0 took 4h09m56s — different day/host load too, not a regression) | ethrex synced cleanly, fastest in the field (~2h16m snap). Two operational caveats: no history (eth_getBlockByNumber returns null below head) and a restart cliff past ~25 minutes. It is smaller only because it retains no history, not because it’s leaner. See Stage B and gotchas. |
Sync-mode & disk-flag audit
Stage B uses each EL’s most disk- and time-efficient available sync mode. We audited before starting slow full-sync runs to avoid accidental archive comparisons. geth’s --history.chain postmerge triggered the audit: it saves disk by pruning pre-merge block history. We verified it on the baseline, then checked every other EL.
Per-EL disk/sync flag audit — full table (7 ELs)
| EL | Disk/sync flags | Status | Notes |
|---|---|---|---|
| geth | --syncmode snap + --history.chain postmerge | optimal | Verified ON in the actual 1.13 TiB baseline run (service-status.txt). Snap-sync + post-merge history prune is the disk floor for geth. |
| besu | sync-mode="SNAP" + data-storage-format="BONSAI" | optimal | Bonsai is Besu's space-efficient flat-DB layout; SNAP avoids full historical execution. |
| nethermind | SnapSync: true + FastBlocks: true | optimal | Snap on; Halite/Paprika flat storage is the modern default. |
| ethrex | --syncmode snap | optimal | Snap is the only efficient mode it exposes. |
| erigon | OtterSync (default) + prune.mode: "full" | disk-optimal | prune.mode: full is the smallest erigon3 footprint. (Separately deadlocks → no-sync; see erigon row.) |
| nimbus_eth1 | fast-sync (default) + prune = true | online-prune confirmed | prune = true prunes online: a 72h run logged continuous Pruning history … pruned=N during import. This resolves the earlier question about whether the flag works. At-tip completeness against a full era1 export remains untested: this full-sync-only node never reached tip within 72h. |
| reth | was archive (no flag) → now --full | fixed | Only reth needed correction. Archive is the default (~2.8 TiB). --full prunes to ~1.2 TiB: full block/receipt history, but state changesets and indices retain only ~10k recent blocks. reth__prysm was relaunched. |
All seven ELs now use disk-optimal sync settings. Six were already correct; only archive-by-default reth changed. Configuration comparisons are therefore equivalent. Sync time still differs: full-sync ELs execute all ~25M blocks, unlike geth’s snap baseline. Final footprint remains comparable.
Final synced disk footprint (Stage B)
Every candidate has a final verdict: synced, capped, or no-sync. Runs were sequential. Footprint means EL + CL datadir size, excluding secrets/validator material; capped and no-sync rows are partial.
Full post-merge history converges at ~1.0–1.2 TiB. History retention sets size across clients. ethrex and minimal-history nethermind are smaller only because they retain no history, not because they’re leaner. Choose on snap-sync speed and restart-resume; see recommendations.
- nimbus_eth1
- ~40 GiB · partial · ~21.6%, full-sync-only
- nethermind (minimal, default)
- ~265 GiB · synced · no history
- ethrex
- ~475 GiB · synced · no history
- reth
- ~980 GiB · partial · projected ~1.1–1.2 TiB
- besu
- ~1080 GiB · synced
- nethermind (full-history)
- ~1088 GiB · synced
- geth
- ~1130 GiB · synced · baseline
- erigon
- ~1210 GiB · no-sync · frozen partial
Full Stage B footprint table — per-candidate bytes, verdicts, and notes
| Candidate | Result | Sync time | Final disk footprint (EL + CL) | Notes |
|---|---|---|---|---|
geth__prysm | synced | ~8h28m | 1.13 TiB — geth 1,245 GB + prysm 655 MB | Baseline: clean, no crash. The snap-synced EL gives Prysm an already-validated head, avoiding a large optimistic gap. |
erigon__prysm | no-sync | n/a | ~1.21 TiB* — erigon 1,333 GB + prysm 1,646 MB | *Partial at a near-tip frozen head; never a synced datadir. erigon3 OtterSync deadlocked against checkpoint-synced Prysm. EL head froze thousands of blocks behind tip; beacon stayed is_optimistic=true. Neither issued the gap-closing forkchoiceUpdated. Raising CL CPU 200%→600% advanced head ~5k blocks, then it froze again. This was a deadlock, not resource starvation. Terminated. |
reth__prysm | capped (72h) | n/a | ~0.98 TiB* — reth 1,065 GB + prysm 12.5 GB | *Partial at block 11,970,965 of 25,395,872 after 72h: 47% by block count, ~21% gas-weighted. reth --full is the only no-snap EL here. Sequential execution is too slow to finish inside the cap. Clean stop, no crash. Size is reconstructed from the last sample before the cap, not a final disk scan. At ~21% gas-executed, size was already ~87% of geth’s 1.13 TiB, projecting ~1.1–1.2 TiB finished. |
nethermind__prysm | synced | ~14.5h | ~1.06 TiB steady-state (re-measured 2026-08-01: ~1,088 GiB — state ~226–230 GiB compact flat storage + ~843 GiB post-merge bodies/receipts + ~19 GiB headers/code) — ~251 GiB at snap-sync, before FastBlocks backfilled post-merge history (268 GB at that point) + prysm 1,431 MB | Snap-synced to head 25,428,620 with 49 peers, no crash. State uses ~226–230 GiB of compact flat storage. Full post-merge history backfills to ~1.06 TiB, comparable to geth. Update 2026-08-03: shipped default is minimal-history (NETHERMIND_FULL_HISTORY=false): fresh sync stays at ~250–280 GiB, state only, no bodies/receipts. ~1.06 TiB is the full-history opt-in for RPC providers. The first attempt stalled 13.3h with zero peers: P2P bound to 127.0.0.1, frozen execution head, apparently healthy beacon. This exposed triage’s blind spot. Advertising a routable external IP fixed it; the re-run synced cleanly. |
besu__prysm | synced; pruned re-run abandoned | ~19h18m | ~1.08 TiB — besu 1,190 GB + prysm 1,682 MB | besu synced cleanly to a fully validating head (~50 peers) in ~19h18m: a production-viable node. Its ~1.08 TiB is the same magnitude as geth (1.13 TiB) and nethermind (~1.06 TiB) with full post-merge history: comparable, not an outlier. A further-pruning re-run deadlocked twice and was abandoned. A stale CL stall caused it, not a besu fault. The operational caveat is snap sync’s fragility during prolonged CL outages, not disk size; see gotchas. |
ethrex__prysm | synced | ~2h16m–4h10m | ~470–476 GiB steady-state plateau (as of 2026-07-31: drifting 470.2 → 475.5 GiB over ~42 hours at +0.13 GiB/hr) — ~286 GiB at first sync (307 GB, 2026-07-06); ~300 GiB at a later re-sync (4h09m56s) | Fastest EL sync: ~2h16m to a fully validating head on v19.0.0. v22.0.0 re-sync took 4h09m56s on a different day/host load; this does not show regression. 50 peers throughout, no crash. One automatic stale-pivot update self-healed in ~4 min. This un-pruned node serves ~no history: live eth_getBlockByNumber returned null below head. After sync, growth fell ~300× from +43 GiB/hr to +0.13 GiB/hr: 470.2 → 475.5 GiB over ~42 hours, verified clean. The earlier ~467 GiB reading caught this plateau mid-climb; growth is not unbounded. Smaller only because it retains no history, not because it’s leaner. nethermind’s state alone is ~226–230 GiB, roughly half ethrex’s total. Encodings differ, and ethrex’s total includes headers/recent blocks, so this comparison is not perfectly controlled. See limitations and gotchas for the unchanged restart cliff and no-history RPC cost. |
Fresh-sync vs. steady-state disk footprint
“Synced” and “settled” can have different sizes. nethermind’s pre-backfill ~251 GiB was not final. ethrex’s mid-climb ~467 GiB did not prove continuing growth. Both fresh and steady measurements appear below where available; missing phases are explicit.
- nethermind
- ~251 GiB fresh → ~1,088 GiB steady (full-history)
- ethrex
- ~286 GiB fresh → ~475 GiB steady (no-history plateau)
Fresh and steady measurements — all recorded phases
| EL | At snap-sync (fresh) | Steady-state | Note |
|---|---|---|---|
| nethermind | ~251 GiB (268 GB) | ~1.06 TiB (re-measured 2026-08-01: ~1,088 GiB — state ~226–230 GiB + ~843 GiB post-merge bodies/receipts + ~19 GiB headers/code) | Grew after sync as FastBlocks backfilled post-merge history. |
| ethrex | ~286 GiB (307 GB, 2026-07-06, v19.0.0); ~300 GiB (2026-07-28, v22.0.0, at 4h09m56s) | ~470–476 GiB plateau (drifts 470.2 → 475.5 GiB over ~42h) | +43 GiB/hr during post-sync settling, then collapsed ~300× to +0.13 GiB/hr, drifting for ~42 hours. No-history node — not pruned-comparable to the rows below. The two runs used different ethrex versions (v19.0.0 → v22.0.0) as well as different days/host load. |
| geth | not separately captured | 1.13 TiB (1,245 GB) | --history.chain postmerge. |
| besu | not separately captured | 1.08 TiB (1,190 GB) | |
| reth | — | ~0.98 TiB partial @72h cap (projected ~1.1–1.2 TiB finished) | Full-sync-only; never reaches a moment distinct from its capped steady state. |
Only nethermind and ethrex have distinct fresh/steady measurements. geth and besu were measured only at finished size. reth never finished within 72h.
Consensus-client matrix across three anchors
The same three CL tiers appear on every execution anchor. We held the EL constant and cycled CLs: ethrex first, already at tip, saving a multi-day re-sync; then geth and nethermind cross-checks. Each CL footprint is under ~1.1% of its anchor’s EL datadir. Engine API decoupling means anchor choice does not bias the comparison. Every EL datadir stayed untouched at tip; runs cycled only CL and validator keys.
All five CLs reached a fully validating head on every anchor, without crashes. Sync took ~22–23 min on ethrex, already at tip, versus ~6–10 min on fresh geth/nethermind. Time tracks anchor freshness, rather than CL choice, so footprint is the real differentiator. Figures below are decimal MB/GB.
Full per-client table — all three anchors
| CL | Ethrex anchor | Geth anchor | Nethermind anchor | Disk-optimal lever |
|---|---|---|---|---|
| lighthouse | ~22m 773 MB (smallest) | ~8m54s 542 MB | ~10m07s 492 MB | checkpoint-sync-url (blob pruning by default) |
| lodestar | ~22m 868 MB | ~6m27s 185 MB (smallest) | ~7m36s 186 MB (smallest) | pruneHistory=true |
| grandine | ~22m ~992 MB (actual) | ~8m50s ~760 MB (actual) | ~9m58s ~765 MB (actual) | --prune-storage — required, or it stores every state |
| teku | ~22m 2,161 MB | ~8m52s 977 MB | ~10m07s 875 MB | data-storage-mode=minimal |
| nimbus | ~23m 5,302 MB (largest (6.9×)) | ~7m58s 1,198 MB (largest) | ~10m13s 1,338 MB (largest) | history=prune |
- ethrex anchor
- ~22m · 773 MB (smallest)
- geth anchor
- ~8m54s · 542 MB
- nethermind anchor
- ~10m07s · 492 MB
- Lever
- checkpoint-sync-url (blob pruning by default)
- ethrex anchor
- ~22m · 868 MB
- geth anchor
- ~6m27s · 185 MB (smallest)
- nethermind anchor
- ~7m36s · 186 MB (smallest)
- Lever
- pruneHistory=true
- ethrex anchor
- ~22m · ~992 MB (actual)
- geth anchor
- ~8m50s · ~760 MB (actual)
- nethermind anchor
- ~9m58s · ~765 MB (actual)
- Lever
- --prune-storage — required, or it stores every state
- ethrex anchor
- ~22m · 2,161 MB
- geth anchor
- ~8m52s · 977 MB
- nethermind anchor
- ~10m07s · 875 MB
- Lever
- data-storage-mode=minimal
- ethrex anchor
- ~23m · 5,302 MB (largest (6.9×))
- geth anchor
- ~7m58s · 1,198 MB (largest)
- nethermind anchor
- ~10m13s · 1,338 MB (largest)
- Lever
- history=prune
Disk ranking, smallest → largest:
- Ethrex anchor: lighthouse (773 MB) < lodestar (868 MB) < grandine (~992 MB) < teku (2,161 MB) < nimbus (5,302 MB)
- Geth anchor: lodestar (185 MB) < lighthouse (542 MB) < grandine (~760 MB) < teku (977 MB) < nimbus (1,198 MB)
- Nethermind anchor: lodestar (186 MB) < lighthouse (492 MB) < grandine (~765 MB) < teku (875 MB) < nimbus (1,338 MB)
CL measurement caveats — heap, watchdog, and sparse files
- teku needed an ethrex-anchor re-run. An undersized JVM heap on the shared host caused 64-min sync; that reading was discarded.
TEKU_CACHE=8192mfixed it: clean sync in 22 min. Size the heap generously to keep garbage-collection (GC) pressure off co-resident services. - lodestar’s first nethermind-anchor run (~76m) was a fluke, not a lodestar property. The EL was importing an unrelated ~2-day gap from a separate crash-loop incident. After recovery: ~7m36s / 186 MB, matching the other four CLs and lodestar’s geth result.
- teku’s two false “not synced” nethermind readings were watchdog bugs. The watchdog latched onto early warmup and never re-checked the recovered anchor. Both footprints are valid: first ~700 MB, published clean re-read 875 MB.
- grandine’s sparse DB files overstate usage. Raw size is ~1,074 MB on geth/nethermind and ~1,344 MB on ethrex. Longer post-sync runtime increased sparse pre-allocation; this is not measurement error. Published actual disk usage: ~760–765 MB on geth/nethermind, ~992 MB on ethrex.
Cross-anchor verdict — the tiers reproduce, not an identical order:
- nimbus is the largest CL on all three anchors — the one ranking that holds without exception.
- {lodestar, lighthouse} are smallest on every anchor. lighthouse wins on ethrex; lodestar wins on geth/nethermind. Within-tier order depends on measurement time.
- {teku, grandine} form the middle tier. teku’s ~700 → 875 MB nethermind readings cross grandine’s ~765 MB. Taking the clean re-read as authoritative, grandine < teku holds on all three anchors. That pair’s order is measurement-sensitive, not a stable client property.
- Observation time affects absolute size. geth/nethermind sweeps measured minutes after checkpoint-sync; ethrex ran longer. Hence nimbus ~1.2–1.3 GB vs ~5.3 GB. Tiers hold; exact within-tier order does not.
- Three anchors reproduce three tiers: lightweight {lodestar, lighthouse}, mid {teku, grandine}, heavy {nimbus}. This supports EL/CL decoupling without an identical total order.
The tier story across anchors
- Lodestar
- ~868 / 185 / 186 MB (ethrex / geth / nethermind anchor)
- Lighthouse
- ~773 / 542 / 492 MB (ethrex / geth / nethermind anchor)
- Grandine
- ~992 / 760 / 765 MB (ethrex / geth / nethermind anchor)
- Teku
- ~2161 / 977 / 875 MB (ethrex / geth / nethermind anchor)
- Nimbus
- ~5302 / 1198 / 1338 MB (ethrex / geth / nethermind anchor)
Reader Q&A: does ethrex serve a usable RPC?
Live probes: 2026-07-29/30, our synced ethrex v22.0.0 node. Head advanced between probes.
Usable from the sync point onward. Earlier blocks are unavailable. The boundary is when you synced, rather than “wallet vs DeFi”.
Before pivot: unavailable
25,634,444 → null
Pivot onward: blocks served
25,634,445 → head ~25,639,228
4,783 blocks (~16h), growing forward
State: ~128 recent blocks (~25 min)
eth_call: head−100 works; head−500 fails
What works (verified)
Current-state DeFi reads work: swap quotes, balances, allowances, and pool state. Every frontend method tested at latest passed: eth_call, eth_getCode, eth_getStorageAt, eth_estimateGas, eth_getBalance, eth_getTransactionCount, eth_gasPrice, eth_feeHistory, eth_maxPriorityFeePerGas, web3_clientVersion, net_version. eth_sendRawTransaction exists and validates input: malformed input returned Invalid params: InvalidLength.
The deploy-today test
Every dapp read worked for a recent deployment. At head ~25,646,566, we tested a real contract-creation transaction 17 blocks back: block 25,646,549, tx 0xc909b51c…, contract 0x227efd38ef38a798ae5ec9af062c437ee4bbef35. eth_getCode returned 8,043 bytes of bytecode. eth_getStorageAt, eth_getTransactionByHash, eth_getTransactionReceipt, and eth_getLogs from deploy block to latest also worked.
ethrex can serve a dapp frontend for a contract deployed today, including current-state reads.
What does NOT work
Every pre-merge probe returns null:
| Block probed | Era | eth_getBlockByNumber |
|---|---|---|
| 1 | genesis, 2015 | null |
| 4,374,488 | 2017 — ICO era | null |
| 12,000,000 | 2021 — DeFi summer | null |
| 15,537,394 | the merge block itself | null |
History starts exactly at the snap-sync pivot. pivot−1 (25,634,444) returned null; pivot+0 (25,634,445) was served. At this earlier probe, head ~25,639,228, the window held 4,783 blocks (~16h). It grows forward but never backfills.
Historical state limit
Historical state is roughly the last ~128 blocks (~25 minutes). eth_call succeeds at head−100; head−500 fails: Vm execution error: DB error: state root missing for block N. Older “balance at block X” requests fail.
Consequently broken:
- Indexer/subgraph backfill
- Portfolio history
- Tax and accounting exports
- Historical charts
- "Show me my transaction from last month"
- Any analytics that replays past logs
eth_getLogs requires topics. Omitting it returns Expected parameter: topics is missing; geth treats it as optional. Conformant tooling can fail even inside ethrex’s served window.
The shipped nginx/Caddy RPC setup inherits these limits. geth with --history.chain postmerge serves post-merge history. ethrex handles current-state/wallet traffic but returns null/errors outside its history window. If exposing an endpoint is the goal, that’s an independent reason to prefer geth or nethermind. ethrex is not a drop-in public RPC for users expecting history.
ethrex’s ~470–476 GiB plateau is not a disk win. The ~600 GiB difference is history other clients retain. Footprint tracks retention.
Recommendation & operational viability
geth and nethermind clear the durability bar. All 12 pairs installed, checkpoint-synced, and passed Stage A; full-history disk sizes converge. Restarts, upgrades, and weeks of uptime separate them. ELs carry the operational risk. CLs look solved only for measured sync and footprint; only prysm was restart-tested.
- geth — conservative default, largest ecosystem, cleanest ~8h28m snap; resumes downtime by importing missed blocks.
- nethermind — client diversity, compact flat-storage state; restart-resume bisected 12 min → ~35 h with no cliff.
- besu — synced cleanly (~19h18m); snap sync is fragile to a prolonged CL outage, fine if you keep it current.
- ethrex — fastest cold sync (~2h16m) but a ~25-min restart cliff; serves no history.
- reth, nimbus_eth1 — full-sync-only; cannot finish a mainnet sync inside 72h on this host.
- erigon — structural OtterSync + prysm deadlock; no synced datadir.
EL summary; CL guidance and caveats follow.
Execution clients
- History retention sets disk size. Full-history ELs converge at ~1.0–1.2 TiB: geth 1.13, nethermind ~1.06, besu 1.08, reth ~1.1–1.2 projected. Update 2026-08-03: nethermind ships minimal-history by default, staying at ~250–280 GiB, state only. Pre-sync blocks return
null, like ethrex. For public RPC history, setNETHERMIND_FULL_HISTORY=trueon a fresh datadir. Its no-history size beats ethrex’s ~470 GiB and reaches a floor geth cannot. Calling it “4× leaner than geth” would compare unlike history settings. With history retained, choose on snap speed and restart-resume. - geth and nethermind both reached validating tip and resumed cleanly. They are the two most battle-tested codebases here, with comparable full-history disk (~1.06 vs ~1.13 TiB). geth offers the largest ecosystem, conservative default, and cleanest ~8h28m snap. nethermind offers diversity and compact flat-storage state. Every tested gap from 12 min to ~35h resumed through ordinary import, without re-snap or cliff. Either supports long-running operation.
- besu is a viable enterprise third. It fully validated; ~1.08 TiB is the same magnitude as geth/nethermind: comparable, not an outlier. A prolonged CL outage can age its snap pivot beyond servable state. The sync thread then dies, observed twice and unrecoverable. Keep the CL current; this is not a set-and-forget solo-staker choice.
- Each remaining EL missed a specific operational requirement:
- ethrex syncs fastest (~2h16m), but 26-minute downtime stalled; 1.5–2h gaps triggered full re-snap. Its datadir settles at ~470–476 GiB, rather than growing unbounded. It is smaller only because it retains no history, not because it’s leaner. nethermind’s ~226–230 GiB state is smaller, though different encodings prevent a perfectly controlled comparison. Fast setup remains painful to operate without restart recovery. This young client moved v19.0.0 → v22.0.0 during the campaign and may improve.
- reth, nimbus_eth1 cannot reach tip practically on this host: full-sync-only, no snap. This misses our snap-to-tip bar; it is not a universal client verdict. reth is widely run elsewhere.
- erigon — deadlocked against checkpoint-synced prysm on this host (structural, reproducible), so no synced datadir.
- Stage-A note: geth, besu, nimbus_eth1, ethrex passed Stage A with zero installer changes and zero REST contention — the cleanest out-of-the-box ELs against Prysm.
Consensus clients
Every swept CL worked: lighthouse, lodestar, grandine, teku, nimbus. All checkpoint-synced to validating head in ~22–23 min without crashes. Within measured sync/footprint, choose by size and preference; only prysm has resume evidence.
- Disk order (the only differentiator): on ethrex, lighthouse (~773 MB) < lodestar (~868 MB) < grandine (~992 MB with
--prune-storage) < teku (~2,161 MB) < nimbus (~5,302 MB). Recommended: lighthouse, ~22-min checkpoint-sync, blob pruning enabled by default. lodestar is smaller on geth/nethermind, down to ~185 MB. - Size teku’s JVM heap generously: undersizing caused GC pressure on co-resident services and invalidated the first run. Enable grandine
--prune-storage, or every state is stored. nimbus is otherwise clean but heaviest, ~6.9× lighthouse. - Keep the CL binary current. Stale prysm v7.1.5 stalled ~28h on a PeerDAS/data-column bug, aging out besu’s pivot. Binary freshness is an operational requirement.
Choose geth or nethermind for long-running EL operation; besu qualifies where operators keep the CL healthy. Any swept CL works on measured axes; lighthouse is the lean default. Initial speed and size alone do not prove durability across restarts and uptime. That deciding risk sits in the EL layer.
Gotchas & lessons learned
Verify EL progress and restart recovery, even when RPC answers. Dates and measured thresholds follow by theme.
ethrex restart gaps · 2026-07-10 · v19.0.0
Sync signals that lie
Stage-A triage misses stalled ELs. It checks CL tip and Engine-API JWT authentication. Optimistic checkpoint-sync passes even with 0 EL peers and frozen execution head: nethermind hid a 13.3h stall. Require peer-count>0 + advancing EL head + beacon
sync_distance, neversync_distancealone.eth_syncing=falsedoes not prove sync. It appears before snap starts (no pivot), after completion, and even mid-sync on besu. Use prysmis_optimistic=false: the EL validated the head payload.Synced nethermind returns an
eth_syncingOBJECT, withcurrentBlock==highestBlock, rather than boolean false. The harness now accepts either condition.besu sync has two tracks. Block import reaches head first, giving a premature “done” signal. Bonsai world-state download/heal remains the bottleneck and expands disk usage.
Restart & resume — the deciding axis
besu snap-sync deadlocked twice after a long CL stall; the pruned re-run was abandoned. Stale prysm v7.1.5 hit a PeerDAS bug and stalled ~28h. Without
forkchoiceUpdated, besu’s pivot aged beyond peers’ servable state: ~128 recent blocks, ≈25 min.World-state heal could not finish. besu threw
IllegalStateException: The pivot block number has not increased; its sync thread died without restarting. The process still answered RPC, but the datadir stayed untouched with zero DB writes. Restarting reused the stale pivot and deadlocked again.Keep the CL current; watch DB writes.
eth_blockNumberresponses do not prove progress. The harness now offers an opt-in stall-watchdog: after bounded polls without progress, restart the unit a few times, then fail the row instead of reaching the cap.ethrex stalls past ~25 min; ~1.5–2h gaps triggered a full re-snap (v19.0.0). A routine restart with a ~1.5–2h gap made ethrex discard its fully synced 286 GiB state and restart near genesis: datadir 286 GiB → ~9 GiB → climbing. Journal:
SNAP SYNC STARTED→PHASE 1/8: BLOCK HEADERSfrom ~198k/25.47M;eth_blockNumber=0x0throughout.Its old head exceeded peers’ ~128-block (~25 min) servable-state window. When prysm sent
forkchoiceUpdated, ethrex re-pivoted instead of importing missed blocks. geth preserves state and imports the gap. Measured costs: ~2h16m cold + ~2h11m post-downtime re-snap. The rebuilt datadir exceeded 286 GiB.This operational pain is a strong candidate explanation for ethrex’s ~0% adoption despite its fastest cold sync.
ethrex’s cliff is bracketed at ~128 blocks / ~24–25 min (2026-07-10). With live prysm driving forkchoice, controlled
systemctl stop eth1→ wait →starttests resumed at 12 min / 68 blk, 20 min / 108 blk, 23 min / 124 blk. Missed blocks imported; datadir survived; canonical head returned to tip.26 min / 132 blk stalled.
eth_blockNumberstayed at the pre-stop head for 12+ min;eth_syncing.currentBlock=0x0. Logs:FCU head state not reachable from DB state … Starting sync toward head;Failed to fetch headers for sync head — peer(s) queried but did not serve headers. The gap widened as tip advanced. No datadir collapse occurred within the 12-min watch; larger ~1.5–2h gaps escalated to full re-snap.The edge matches the servable window: peers serve gap headers inside it, but not beyond it. This young v19.0.0 client may improve. The recorded 2h16m sync-time result remains unchanged.
geth resumed after ~52h (multi-day) downtime, measured 2026-07-10. It was ~15,400 blocks behind, starting at
eth_syncing.startingBlock=25,487,154, not genesis; no snap-pivot reset. It kept its multi-hundred-GB datadir and applied trie diffs during sequential block import. Every journal segment showedImported new chain segment … triediffs=… triedirty=….No datadir collapse occurred, unlike ethrex’s 286 GiB → ~9 GiB.
eth_syncingreturned an import object, never0x0. State healing completed (healingTrienodes=0x0), then the node reached validating tip:eth_syncing=falseat block 25,502,592. This preservation-and-import mechanism clears the operational bar for regular upgrades/restarts. Shared-host conditions prevent a clean wall-clock bound; only mechanism and datadir preservation are claimed.nethermind showed no servable-window cliff in 2026-08-01 to 08-03 tests. Unlike ethrex’s ~128-block cliff or besu’s pivot-aging deadlock, it resumed ordinary import. First, a CL restart left it 10,607 blocks (~35h) behind external tip. It closed the gap in 35m09s (~302 blocks/min). Datadir grew 1.165 → 1.178 TB (+1.1%), matching imported bodies/receipts: no state wipe or re-snap.
Controlled stop→wait→start tests confirmed ordinary Engine-API import at every gap: 12 min / 30 min / 1 h / 4 h, or 69 / 151 / 301 / 1,196 blocks. Every rung verified clean, without re-pivot or snap/state-sync.
State-directory growth stayed ~1.0–1.3 MiB per imported block across rungs. A re-snap would rewrite the whole ~238 GiB directory. Resume time scaled gently: 121s @ 12 min → 186s @ 1 h → 483s @ 4 h → 35m09s @ ~35 h. CL catch-up on missed slots dominated, rather than EL import.
A separate nethermind fresh sync took 1h52m51s on 2026-07-31: ~280 GiB at snap, zero restarts. This supplements the ~14.5h Stage-B figure. Its minutes-old near-tip pivot and different network conditions explain the faster result.
prysm resumed from its own DB in four clean events (n=4, 2026-08-02 to 08-03). After a deliberate 30-min CL stop, it drove the EL at
sync_distance=0within ~2m44s, without re-checkpoint-sync. During the stop, EL head froze undriven, beyond ethrex’s ~25-min cliff. The 12 min / 1 h / 4 h bisection rungs repeated re-checkpoint-free resume.Only prysm was resume-tested. Other CLs remain untested; prysm’s intermittent discovery-listener wedge was absent here, not disproven. Beacon-only operation, without validator keys, made deliberate stops zero-risk.
Client-specific & measurement notes
ethrex serves nothing before its snap-sync pivot, measured live 2026-07-29. At head ~25,639,228, pivot 25,634,445 resolved at pivot+0 and pivot+200; pivot−1 (25,634,444) returned
null. The exact window held 4,783 blocks (~16h), grew forward, and never backfilled. Blocks 1, 1,000,000, 21,600,000, and merge block 15,537,394 returnednull.Current-state RPC works.
eth_chainId,eth_gasPrice,eth_getBalance/eth_getTransactionCount@ latest, andeth_call@ latest resolved. Post-pivot blocks/receipts/logs worked: 615 logs over a 2-block USDC range at head−1000. Balances, current quotes, and allowances work for wallet traffic.Pre-pivot blocks/logs/receipts fail with
Internal Error: Could not get body for block N, excluding effectively all Ethereum history. This breaks indexer/subgraph backfill, portfolio history, and tax/accounting exports. The Q&A details the tighter state window,eth_getLogsdeviation, and nginx/Caddy endpoint tradeoff against geth--history.chain postmerge.Market share weakly predicts syncability. erigon deadlocked; full-sync-only reth/nimbus_eth1 could not finish in 72h. Yet ethrex (~0% share, Lambda Class) synced fastest (~2h16m). Snap availability and client reliability explain more: ethrex has snap and clock-based stale-pivot self-healing. Its measured ~470 GiB no-history plateau is not a disk win; speed remains its supported claim.
Loopback P2P broke peering. besu and nethermind advertised
127.0.0.1, causing degraded/zero peering. Removing loopbackp2p-host/ injecting routableExternalIpfixed them. geth/erigon/reth/ethrex/nimbus_eth1 bind externally by default.erigon3 OtterSync + checkpoint-synced prysm deadlock — the one structural no-sync (see the erigon row): EL head freezes behind tip while the beacon stays optimistic; neither issues the
forkchoiceUpdatedthat would close the gap. Raising CPU caps advanced it ~5k blocks then re-froze.reth is
--full-only here (archive was the disk-hostile default; switched to--full); sequential full block execution can't finish a mainnet sync inside the 72h cap.Sampler timestamp skew (~2h): samples label local CEST times as
Z. Trust file mtime for wall-clock, not the sample'stimestamp_utcstring.
Bottom line
Across six weeks and 12 pairs, sync speed and restart recovery separated clients. Disk size reflected history settings.
- Disk converges, it doesn’t rank: ~1.0–1.2 TiB for full-history ELs (geth 1.13, nethermind ~1.06, besu 1.08, reth ~1.1–1.2 projected). Minimal-history nethermind (~250–280 GiB default) and ethrex (~470–476 GiB plateau) are smaller because they retain no history, not because they’re leaner.
- Speed has a real order: ethrex (~2h16m) < geth (~8h28m) < nethermind (~14.5h) < besu (~19h18m). reth and nimbus_eth1 never reached tip in the 72h cap (full-sync-only), and erigon deadlocked entirely against a checkpoint-synced CL.
- Restart-resume decides. geth and nethermind imported missed blocks cleanly across tested gaps: 12 min up to ~52h/~35h. Neither re-snapped. ethrex stalled past ~25 min/~128 blocks; ~1.5–2h gaps triggered re-snap near scratch. besu’s snap sync deadlocks if a CL outage ages its pivot out of the ~128-block servable-state window.
- Pick geth or nethermind for long-running nodes; besu qualifies for enterprise operators keeping the CL current. Any swept CL works, with lighthouse the lean default. ethrex’s fastest cold sync cannot offset its restart cliff and missing history: watch, rather than deploy, today.