satoshi/KamadoPool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

P1 reliability + block-broadcast fallback path

Browse Source 
P1 audits / fixes:

* Bitcoin Core RPC now retries up to 3 times with linear backoff on
  transport errors, 5xx responses, and warm-up/loading RPC errors
  (code -28). Hard "no" answers (block-not-found etc.) bubble up
  immediately so we don't mask real errors.

* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
  (~30s with the default poll cadence). Stuck readers no longer hold
  stale snapshots indefinitely or freeze hub state.

* ZMQ subscriber freshness: aggregator records the last-event
  timestamp, surfaces zmq_enabled / has_last_zmq_event /
  last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
  the gap exceeds 30 minutes.

* /healthz expanded with submit_attempts / submits_confirmed /
  submit_gap, fallback_submits_total + last_fallback_*, and the zmq
  staleness check. Now usable as a real-world ops dashboard signal.

Block-broadcast fallback (new feature):

  * ckpool patch 0004: hooks local_block_submit to write the raw block
    hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
    invoking generator_submitblock. Unlinks on success. ckpool's normal
    flow is otherwise untouched.

  * api/internal/blocksubmit: watcher polls the dir every 5s. Files
    sitting longer than the grace window (default 30s, configurable)
    are re-broadcast through operator-supplied backup RPC URLs in
    sequence. Treats both null and any "duplicate*" reject reason as
    success (the block landed). Pre-checks the primary chain first so
    a stale file from a successful-but-unlinked submit gets cleaned
    up without bothering fallbacks.

  * Aggregator records each successful fallback submission as a
    persistent counter and surfaces it in the snapshot so the UI can
    show a "primary bitcoind isn't accepting submits" alert.

  * Config: BACKUP_RPC_URLS (comma- or newline-separated, with
    optional inline credentials) plus PENDING_BLOCKS_DIR and
    PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
    https://user:pass@host:port/ works cleanly.

The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
This commit is contained in:
satoshi
2026-04-27 21:25:56 +03:00
parent df0dbf89e5
commit a4a894e196
10 changed files with 687 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ckpool/patches/0004-dump-pending-block-for-fallback.patch
+80
View File
@@ -0,0 +1,80 @@
diff --git a/src/stratifier.c b/src/stratifier.c
index 52da790..815eb01 100644
--- a/src/stratifier.c
+++ b/src/stratifier.c
@@ -2069,16 +2069,64 @@ process_block(const workbase_t *wb, const char *coinbase, const int cblen,
return gbt_block;
}
+/* Write the raw block hex to a sidecar file under <logdir>/pending-blocks/
+ * before submitting, so an external watcher (kamado-api) can re-broadcast
+ * via fallback RPC nodes if our primary bitcoind doesn't accept it. The
+ * file is unlinked once generator_submitblock returns success. Best-
+ * effort: any error here is logged at INFO and never blocks the submit. */
+static void kamado_dump_pending_block(ckpool_t *ckp, const char *gbt_block,
+ const char *rhash, int height,
+ char *out_path, size_t out_path_len)
+{
+ char dir[512] = {};
+ int fd;
+ size_t blen;
+ ssize_t w;
+
+ out_path[0] = '\0';
+ if (!ckp || !ckp->logdir || !gbt_block)
+ return;
+ snprintf(dir, sizeof(dir), "%spending-blocks", ckp->logdir);
+ if (mkdir(dir, 0750) < 0 && errno != EEXIST) {
+ LOGINFO("kamado: mkdir %s failed: %s", dir, strerror(errno));
+ return;
+ }
+ snprintf(out_path, out_path_len, "%s/%d-%.16s.hex", dir, height, rhash);
+ fd = open(out_path, O_WRONLY | O_CREAT | O_TRUNC, 0640);
+ if (fd < 0) {
+ LOGINFO("kamado: open %s failed: %s", out_path, strerror(errno));
+ out_path[0] = '\0';
+ return;
+ }
+ blen = strlen(gbt_block);
+ w = write(fd, gbt_block, blen);
+ close(fd);
+ if (w != (ssize_t)blen) {
+ LOGINFO("kamado: short write to %s (%zd/%zu)", out_path, w, blen);
+ unlink(out_path);
+ out_path[0] = '\0';
+ return;
+ }
+ LOGNOTICE("kamado: dumped pending block height %d to %s (%zu bytes)",
+ height, out_path, blen);
+}
+
/* Submit block data locally, absorbing and freeing gbt_block */
static bool local_block_submit(ckpool_t *ckp, char *gbt_block, const uchar *flip32, int height)
{
- bool ret = generator_submitblock(ckp, gbt_block);
char heighthash[68] = {}, rhash[68] = {};
+ char pending_path[512] = {};
uchar swap256[32];
+ bool ret;
- free(gbt_block);
swap_256(swap256, flip32);
__bin2hex(rhash, swap256, 32);
+ kamado_dump_pending_block(ckp, gbt_block, rhash, height,
+ pending_path, sizeof(pending_path));
+
+ ret = generator_submitblock(ckp, gbt_block);
+
+ free(gbt_block);
generator_preciousblock(ckp, rhash);
/* Check failures that may be inconclusive but were submitted via other
@@ -2099,6 +2147,8 @@ static bool local_block_submit(ckpool_t *ckp, char *gbt_block, const uchar *flip
height, ret ? "ACCEPTED" : "REJECTED");
}
}
+ if (ret && pending_path[0])
+ unlink(pending_path);
return ret;
}
ckpool/patches/README.md
+28 -4
View File
@@ -9,11 +9,15 @@ Patches are applied in alphabetical order by filename. Use a numeric prefix to e
## Current state
One Kamado patch is applied on top of the pinned upstream commit:
Four Kamado patches are applied on top of the pinned upstream commit, in
alphabetical order:
| Patch | What it does |
| --------------------------------------------- | -------------------------------------------------------------------------------------- |
| `0001-expose-bestever-in-runtime-json.patch` | Adds `bestever` field to the `users` / `workers` runtime socket JSON |
| Patch | What it does |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `0001-expose-bestever-in-runtime-json.patch` | Adds `bestever` field to the `users` / `workers` runtime socket JSON |
| `0002-enable-socket-api-responses.patch` | Always reply on the listener socket so kamado-api gets responses even with `btcsolo: true` |
| `0003-share-error-as-stratum-array.patch` | Maps `share_err` to Stratum spec error codes; emits `[code, msg, null]` per Slush |
| `0004-dump-pending-block-for-fallback.patch` | Writes the raw block hex to `<logdir>/pending-blocks/` before submit; unlinks on success |
### Why 0001 matters
@@ -30,6 +34,26 @@ This patch adds `bestever` to the runtime JSON so the UI can show both
"this round" and "all-time" best share side by side. No behavioral change
to share validation or block handling. Candidate for upstreaming.
### Why 0004 matters
Block submission to bitcoind is the most revenue-critical RPC call ckpool
makes. If bitcoind is unreachable when a share meets network difficulty,
ckpool's `generator` thread retries indefinitely against the same single
endpoint — and the raw block data lives only in stratifier memory, so a
ckpool crash before bitcoind comes back permanently loses the block.
This patch hooks `local_block_submit` to write the raw block hex to
`<logdir>/pending-blocks/<height>-<rhash>.hex` *before* invoking
`generator_submitblock`, and unlinks the file on success. `kamado-api`
runs a watcher over that directory: if a file persists past a grace
period (default 30 s), it submits the block via fallback RPC URLs the
operator has configured. Multiple fallbacks are tried in sequence; the
file is unlinked when any fallback returns success or "duplicate"
(meaning the block already landed).
This is a Kamado-specific integration hook — almost certainly not
upstreamable, but minimal-impact on existing ckpool behavior.
Beyond this patch, the pinned upstream commit (`cfb0f83b`, tagged as
version 1.0) already includes every fix that Bassin issue #29 asked to
backport, plus several improvements: