P1 reliability + block-broadcast fallback path
P1 audits / fixes:
* Bitcoin Core RPC now retries up to 3 times with linear backoff on
transport errors, 5xx responses, and warm-up/loading RPC errors
(code -28). Hard "no" answers (block-not-found etc.) bubble up
immediately so we don't mask real errors.
* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
(~30s with the default poll cadence). Stuck readers no longer hold
stale snapshots indefinitely or freeze hub state.
* ZMQ subscriber freshness: aggregator records the last-event
timestamp, surfaces zmq_enabled / has_last_zmq_event /
last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
the gap exceeds 30 minutes.
* /healthz expanded with submit_attempts / submits_confirmed /
submit_gap, fallback_submits_total + last_fallback_*, and the zmq
staleness check. Now usable as a real-world ops dashboard signal.
Block-broadcast fallback (new feature):
* ckpool patch 0004: hooks local_block_submit to write the raw block
hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
invoking generator_submitblock. Unlinks on success. ckpool's normal
flow is otherwise untouched.
* api/internal/blocksubmit: watcher polls the dir every 5s. Files
sitting longer than the grace window (default 30s, configurable)
are re-broadcast through operator-supplied backup RPC URLs in
sequence. Treats both null and any "duplicate*" reject reason as
success (the block landed). Pre-checks the primary chain first so
a stale file from a successful-but-unlinked submit gets cleaned
up without bothering fallbacks.
* Aggregator records each successful fallback submission as a
persistent counter and surfaces it in the snapshot so the UI can
show a "primary bitcoind isn't accepting submits" alert.
* Config: BACKUP_RPC_URLS (comma- or newline-separated, with
optional inline credentials) plus PENDING_BLOCKS_DIR and
PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
https://user:pass@host:port/ works cleanly.
The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
This commit is contained in:
@@ -90,14 +90,42 @@ func writeJSON(w http.ResponseWriter, status int, v any) {
|
||||
func (s *Server) health(w http.ResponseWriter, r *http.Request) {
|
||||
snap := s.Agg.Snapshot()
|
||||
status := http.StatusOK
|
||||
|
||||
// Submit-attempt gap: if we've tried to submit more blocks than have
|
||||
// been confirmed, surface the gap. A non-zero gap is a strong signal
|
||||
// even if everything else looks healthy.
|
||||
submitGap := snap.BlockSubmitAttempts - snap.BlockSubmitsConfirmed
|
||||
if submitGap < 0 {
|
||||
submitGap = 0
|
||||
}
|
||||
|
||||
// ZMQ freshness: only meaningful if the operator enabled it. Stale
|
||||
// = no event in 30 minutes (typical mainnet block interval is 10
|
||||
// min, but spikes happen).
|
||||
zmqStale := false
|
||||
if snap.ZMQEnabled && snap.HasLastZMQEvent && snap.LastZMQEventAge > 1800 {
|
||||
zmqStale = true
|
||||
}
|
||||
|
||||
overallOK := snap.CKPoolOK && snap.BitcoinOK && submitGap == 0 && !zmqStale
|
||||
if !snap.CKPoolOK || !snap.BitcoinOK {
|
||||
status = http.StatusServiceUnavailable
|
||||
}
|
||||
writeJSON(w, status, map[string]any{
|
||||
"ok": snap.CKPoolOK && snap.BitcoinOK,
|
||||
"ckpool": snap.CKPoolOK,
|
||||
"bitcoin": snap.BitcoinOK,
|
||||
"last_error": snap.LastError,
|
||||
"ok": overallOK,
|
||||
"ckpool": snap.CKPoolOK,
|
||||
"bitcoin": snap.BitcoinOK,
|
||||
"submit_attempts": snap.BlockSubmitAttempts,
|
||||
"submits_confirmed": snap.BlockSubmitsConfirmed,
|
||||
"submit_gap": submitGap,
|
||||
"fallback_submits_total": snap.FallbackSubmitsTotal,
|
||||
"last_fallback_submit_at": snap.LastFallbackSubmitAt,
|
||||
"last_fallback_via": snap.LastFallbackVia,
|
||||
"zmq_enabled": snap.ZMQEnabled,
|
||||
"zmq_event_age_seconds": snap.LastZMQEventAge,
|
||||
"zmq_has_event": snap.HasLastZMQEvent,
|
||||
"zmq_stale": zmqStale,
|
||||
"last_error": snap.LastError,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user