P1 reliability + block-broadcast fallback path
P1 audits / fixes:
* Bitcoin Core RPC now retries up to 3 times with linear backoff on
transport errors, 5xx responses, and warm-up/loading RPC errors
(code -28). Hard "no" answers (block-not-found etc.) bubble up
immediately so we don't mask real errors.
* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
(~30s with the default poll cadence). Stuck readers no longer hold
stale snapshots indefinitely or freeze hub state.
* ZMQ subscriber freshness: aggregator records the last-event
timestamp, surfaces zmq_enabled / has_last_zmq_event /
last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
the gap exceeds 30 minutes.
* /healthz expanded with submit_attempts / submits_confirmed /
submit_gap, fallback_submits_total + last_fallback_*, and the zmq
staleness check. Now usable as a real-world ops dashboard signal.
Block-broadcast fallback (new feature):
* ckpool patch 0004: hooks local_block_submit to write the raw block
hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
invoking generator_submitblock. Unlinks on success. ckpool's normal
flow is otherwise untouched.
* api/internal/blocksubmit: watcher polls the dir every 5s. Files
sitting longer than the grace window (default 30s, configurable)
are re-broadcast through operator-supplied backup RPC URLs in
sequence. Treats both null and any "duplicate*" reject reason as
success (the block landed). Pre-checks the primary chain first so
a stale file from a successful-but-unlinked submit gets cleaned
up without bothering fallbacks.
* Aggregator records each successful fallback submission as a
persistent counter and surfaces it in the snapshot so the UI can
show a "primary bitcoind isn't accepting submits" alert.
* Config: BACKUP_RPC_URLS (comma- or newline-separated, with
optional inline credentials) plus PENDING_BLOCKS_DIR and
PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
https://user:pass@host:port/ works cleanly.
The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
This commit is contained in:
@@ -42,8 +42,18 @@ type wsClient struct {
|
||||
conn net.Conn
|
||||
send chan []byte // serialized JSON frames pending write
|
||||
writeMu sync.Mutex // guards writes to conn (writer + reader-pong)
|
||||
|
||||
// dropMisses counts consecutive Broadcast calls where this client's
|
||||
// send channel was full. After maxDropMisses we close the connection
|
||||
// instead of letting a stuck reader hold an old snapshot forever.
|
||||
dropMisses int
|
||||
}
|
||||
|
||||
// maxDropMisses bounds how many back-to-back broadcasts we let the
|
||||
// hub skip for one client before forcibly disconnecting it. With the
|
||||
// poll cadence at 5s this is roughly 30s of unresponsiveness.
|
||||
const maxDropMisses = 6
|
||||
|
||||
// writeFrameLocked writes one frame, serializing writers on the client.
|
||||
func (c *wsClient) writeFrameLocked(opcode byte, payload []byte) error {
|
||||
c.writeMu.Lock()
|
||||
@@ -79,22 +89,33 @@ func (h *Hub) remove(c *wsClient) {
|
||||
|
||||
// Broadcast serializes the snapshot once and enqueues it for every
|
||||
// subscribed client. Slow clients are dropped rather than blocking
|
||||
// the hub.
|
||||
// the hub. After maxDropMisses consecutive drops for a single client,
|
||||
// we forcibly close its connection so a stuck consumer doesn't hold
|
||||
// resources indefinitely or freeze on a stale snapshot.
|
||||
func (h *Hub) Broadcast(snap state.Snapshot) {
|
||||
payload, err := json.Marshal(snap)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
h.mu.RLock()
|
||||
defer h.mu.RUnlock()
|
||||
var toClose []*wsClient
|
||||
h.mu.Lock()
|
||||
for c := range h.clients {
|
||||
select {
|
||||
case c.send <- payload:
|
||||
c.dropMisses = 0
|
||||
default:
|
||||
// Drop — the client's reader goroutine will clean up on
|
||||
// the next write failure or close frame.
|
||||
c.dropMisses++
|
||||
if c.dropMisses >= maxDropMisses {
|
||||
toClose = append(toClose, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
h.mu.Unlock()
|
||||
for _, c := range toClose {
|
||||
// Closing the conn unblocks the writer goroutine, which
|
||||
// removes the client from the hub through wsReader's defer.
|
||||
_ = c.conn.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// handleWS upgrades an HTTP request to a WebSocket connection and
|
||||
|
||||
Reference in New Issue
Block a user