P1 reliability + block-broadcast fallback path

P1 audits / fixes:

* Bitcoin Core RPC now retries up to 3 times with linear backoff on
  transport errors, 5xx responses, and warm-up/loading RPC errors
  (code -28). Hard "no" answers (block-not-found etc.) bubble up
  immediately so we don't mask real errors.

* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
  (~30s with the default poll cadence). Stuck readers no longer hold
  stale snapshots indefinitely or freeze hub state.

* ZMQ subscriber freshness: aggregator records the last-event
  timestamp, surfaces zmq_enabled / has_last_zmq_event /
  last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
  the gap exceeds 30 minutes.

* /healthz expanded with submit_attempts / submits_confirmed /
  submit_gap, fallback_submits_total + last_fallback_*, and the zmq
  staleness check. Now usable as a real-world ops dashboard signal.

Block-broadcast fallback (new feature):

  * ckpool patch 0004: hooks local_block_submit to write the raw block
    hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
    invoking generator_submitblock. Unlinks on success. ckpool's normal
    flow is otherwise untouched.

  * api/internal/blocksubmit: watcher polls the dir every 5s. Files
    sitting longer than the grace window (default 30s, configurable)
    are re-broadcast through operator-supplied backup RPC URLs in
    sequence. Treats both null and any "duplicate*" reject reason as
    success (the block landed). Pre-checks the primary chain first so
    a stale file from a successful-but-unlinked submit gets cleaned
    up without bothering fallbacks.

  * Aggregator records each successful fallback submission as a
    persistent counter and surfaces it in the snapshot so the UI can
    show a "primary bitcoind isn't accepting submits" alert.

  * Config: BACKUP_RPC_URLS (comma- or newline-separated, with
    optional inline credentials) plus PENDING_BLOCKS_DIR and
    PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
    https://user:pass@host:port/ works cleanly.

The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
This commit is contained in:
satoshi
2026-04-27 21:25:56 +03:00
parent df0dbf89e5
commit a4a894e196
10 changed files with 687 additions and 18 deletions
+20
View File
@@ -39,6 +39,22 @@ type Config struct {
// the UI uses the public mempool.space; non-empty means the user
// has pointed Kamado at their own instance via the StartOS config.
MempoolBaseURL string
// Directory ckpool's patched local_block_submit writes raw block
// hex into. Empty disables the fallback submitter.
PendingBlocksDir string
// Comma- or newline-separated list of fallback bitcoind RPC URLs
// that the submitter will try in order if a block stays pending
// past the grace window. Each entry can encode credentials inline:
// https://user:pass@host:8332/
// or without (e.g. an unauthenticated localhost backup node).
BackupRPCURLs string
// How long a pending block file must persist before the submitter
// will try fallback RPCs. Defaults to 30s — long enough for
// ckpool's own retry loop and a fast ckpool->bitcoind round-trip.
PendingBlocksGrace time.Duration
}
func FromEnv() (*Config, error) {
@@ -54,6 +70,10 @@ func FromEnv() (*Config, error) {
DBPath: getenv("DB_PATH", "/var/lib/kamado/kamado.db"),
PollInterval: getenvDuration("POLL_INTERVAL", 5*time.Second),
MempoolBaseURL: os.Getenv("MEMPOOL_BASE_URL"),
PendingBlocksDir: os.Getenv("PENDING_BLOCKS_DIR"),
BackupRPCURLs: os.Getenv("BACKUP_RPC_URLS"),
PendingBlocksGrace: getenvDuration("PENDING_BLOCKS_GRACE", 30*time.Second),
}
if cfg.BitcoinRPCURL == "" {