Removes the entire fallback submitter mechanism: ckpool patch 0004,
the blocksubmit package, the wiring in main.go, the config fields,
the aggregator's fallback counters and snapshot fields, the healthz
fallback fields, and the TS type fields plus the HealthBanners
fallback alert.
Reasoning: ckpool's primary bitcoind submission must remain the
single source of truth, and getting the parallel "race a fallback
during the submit" semantics right is more architectural complexity
than the marginal reliability gain justifies. The original upstream
behavior — submit to bitcoind, retry indefinitely if unavailable —
is what we want.
Kept intact:
* Submit-attempt vs confirmed counters (block_submit_attempts /
block_submits_confirmed). Useful on their own as a "did bitcoind
confirm the submission?" signal.
* HealthBanners shows submit_gap and zmq_stale only.
* /healthz exposes submit_gap, zmq_stale, etc.
* All P0 reliability work (tailer cursor, reconcile loop, reorg
detection, multi-solve guard) and other P1 (RPC retry, WS
back-pressure, ZMQ tracking, startup readiness gate).
P1 audits / fixes:
* Bitcoin Core RPC now retries up to 3 times with linear backoff on
transport errors, 5xx responses, and warm-up/loading RPC errors
(code -28). Hard "no" answers (block-not-found etc.) bubble up
immediately so we don't mask real errors.
* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
(~30s with the default poll cadence). Stuck readers no longer hold
stale snapshots indefinitely or freeze hub state.
* ZMQ subscriber freshness: aggregator records the last-event
timestamp, surfaces zmq_enabled / has_last_zmq_event /
last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
the gap exceeds 30 minutes.
* /healthz expanded with submit_attempts / submits_confirmed /
submit_gap, fallback_submits_total + last_fallback_*, and the zmq
staleness check. Now usable as a real-world ops dashboard signal.
Block-broadcast fallback (new feature):
* ckpool patch 0004: hooks local_block_submit to write the raw block
hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
invoking generator_submitblock. Unlinks on success. ckpool's normal
flow is otherwise untouched.
* api/internal/blocksubmit: watcher polls the dir every 5s. Files
sitting longer than the grace window (default 30s, configurable)
are re-broadcast through operator-supplied backup RPC URLs in
sequence. Treats both null and any "duplicate*" reject reason as
success (the block landed). Pre-checks the primary chain first so
a stale file from a successful-but-unlinked submit gets cleaned
up without bothering fallbacks.
* Aggregator records each successful fallback submission as a
persistent counter and surfaces it in the snapshot so the UI can
show a "primary bitcoind isn't accepting submits" alert.
* Config: BACKUP_RPC_URLS (comma- or newline-separated, with
optional inline credentials) plus PENDING_BLOCKS_DIR and
PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
https://user:pass@host:port/ works cleanly.
The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
Upstream ckpool tracks user/worker best_ever across block solves but
only emits it in the on-disk users.json/workers.json persistence files,
not in the runtime socket API (userinfo() / workerinfo() in
stratifier.c). Consumers that poll the socket — like kamado-api — see
bestdiff reset to zero on every block solve via reset_bestshares()
with no all-time field to fall back on. That is the "best share stuck
at zero after a block" UX we want to fix in Kamado.
0001-expose-bestever-in-runtime-json.patch adds bestever to the two
JSON_CPACK calls so the UI can show current-round and all-time side
by side. No behavioral change, no impact on share validation or block
handling. Candidate for upstreaming.
Also surfaces BestEver in api/internal/ckpool types (User, Worker).
Gracefully degrades to 0 on an unpatched ckpool.
Forks CKPool-solo at upstream commit cfb0f83b (v1.0) via a multi-stage
Docker build, with an env-driven entrypoint that renders ckpool.conf from
a template. The pinned commit already includes every fix referenced in
Bassin issue #29 (workbase_id double increment, extended low-power
timeouts, configurable dropidle, vardiff burst handling).
No patches are applied yet — the patches/ directory holds the workflow
and build wiring so Kamado-specific patches can be added incrementally.
Build is portable across aarch64/x86_64: yasm is intentionally omitted
so CKPool falls back to its C SHA256, and CFLAGS override drops upstream's
default -march=native. Runtime image ships ckpool and ckpmsg for socket
debugging; share logging (-L) is enabled by default.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
satoshi