Submit-first ordering. Patch 0004 now calls generator_submitblock
BEFORE writing the pending-block hex to disk. The happy path adds zero
disk I/O — we only dump when the primary returned false. The same
patch bounds generator_submitblock's "no live current_si" spin to
~3s instead of the original infinite loop, so a permanently-down
primary doesn't pin the stratifier; the bounded spin lets the caller
return false and lets local_block_submit dump for kamado-api to take
over.
Default grace lowered from 30s to 3s. With ckpool's bounded spin and
sub-second sweep cadence, the fallback now reacts within ~4s of a
failed primary submit — fast enough that the work is still relevant
for the current chain tip. The submitter's sweep poll dropped to 1s
to match.
UI HealthBanners. New top-of-page strip surfaces:
* Fallback used (red banner, 24h after most recent event):
"primary bitcoind didn't accept; backup X took over Y ago"
* Submit gap (orange banner, only when no recent fallback):
"N blocks attempted but unconfirmed — configure backups"
* ZMQ stale (orange banner): no hashblock frame in 30+ minutes
Operators see degraded-but-not-fatal states without checking logs.
Startup readiness gate. main now waits up to 8s on agg.Ready() before
starting the HTTP server so the very first /api/snapshot doesn't show
all-zero state during the aggregator's first refresh. Capped so a
permanently-down bitcoind can't block startup; /healthz is honest
about the degraded state once we do start serving.
P1 audits / fixes:
* Bitcoin Core RPC now retries up to 3 times with linear backoff on
transport errors, 5xx responses, and warm-up/loading RPC errors
(code -28). Hard "no" answers (block-not-found etc.) bubble up
immediately so we don't mask real errors.
* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
(~30s with the default poll cadence). Stuck readers no longer hold
stale snapshots indefinitely or freeze hub state.
* ZMQ subscriber freshness: aggregator records the last-event
timestamp, surfaces zmq_enabled / has_last_zmq_event /
last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
the gap exceeds 30 minutes.
* /healthz expanded with submit_attempts / submits_confirmed /
submit_gap, fallback_submits_total + last_fallback_*, and the zmq
staleness check. Now usable as a real-world ops dashboard signal.
Block-broadcast fallback (new feature):
* ckpool patch 0004: hooks local_block_submit to write the raw block
hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
invoking generator_submitblock. Unlinks on success. ckpool's normal
flow is otherwise untouched.
* api/internal/blocksubmit: watcher polls the dir every 5s. Files
sitting longer than the grace window (default 30s, configurable)
are re-broadcast through operator-supplied backup RPC URLs in
sequence. Treats both null and any "duplicate*" reject reason as
success (the block landed). Pre-checks the primary chain first so
a stale file from a successful-but-unlinked submit gets cleaned
up without bothering fallbacks.
* Aggregator records each successful fallback submission as a
persistent counter and surfaces it in the snapshot so the UI can
show a "primary bitcoind isn't accepting submits" alert.
* Config: BACKUP_RPC_URLS (comma- or newline-separated, with
optional inline credentials) plus PENDING_BLOCKS_DIR and
PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
https://user:pass@host:port/ works cleanly.
The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
Closes the silent-failure modes between "ckpool logs a solve" and
"block correctly displayed":
* Difficulty estimate matched mempool.space — the projection now uses
(inEpoch + 1) intervals so it converges on Bitcoin Core's eventual
retarget formula at end-of-epoch instead of undershooting by ~0.05–
0.10 % throughout.
* Tailer resumes mid-log on restart — persists (inode, offset) to kv
every EOF + on shutdown, and replays the unread tail next time. Any
solve line written while kamado-api was down would previously be
invisible forever.
* Background reconcile loop (60 s) retries hash/reward enrichment for
blocks the original RPC missed, so a transient bitcoind-index race no
longer permanently leaves a block hashless.
* Reorg detection: same loop compares each recent stored hash against
getblockhash(height); a mismatch stamps orphaned_at. UI renders these
strikethrough with a red "orphaned" tag instead of showing illusory
rewards forever.
* InsertBlock now reports whether a row was actually inserted; the
caller WARN-logs duplicate-height ignores so a re-mined orphaned
height can't disappear silently.
* Submit-attempt vs confirmed counters surface failed submissions:
every "Possible/Submitting block solve" log line increments
block_submit_attempts; "Solved and confirmed" increments
block_submits_confirmed. A growing gap means bitcoind is rejecting
our submissions — previously invisible.
* share_err patch refreshed against pinned ckpool source: added
SE_NO_JOBID -> 21 and SE_WORKER_MISMATCH -> 24 mappings, kept
SE_INVALID_NONCE2 in 20 (it's a malformed-input error, not low-diff).
AxeOS users now see actionable Stratum codes instead of
"unknown error".
UI gets new orphaned_at + block_submit_attempts/confirmed fields on
the snapshot type and a strikethrough-with-tag rendering for orphaned
blocks in BlocksTable.
ckpool's share-rejection response sets the "error" field to a bare
JSON string ("Stale", "Above target", "Duplicate", ...). The
Stratum mining v1 spec, and every miner firmware that follows it,
expects that field to be a [code, message, traceback] array. AxeOS
parses with cJSON_GetArrayItem after a cJSON_IsArray check, sees a
string, falls back to "unknown error", and the rejection reason
disappears from the dashboard. Bassin / public-pool sends the
array form, which is why those rejections render as "stale" there.
Patch the JSON_ERR macro in stratifier.c to wrap the existing
share_errs[] string in json_pack("[isn]", ...) and add a small
share_err_code() helper mapping ckpool's enum to the standard
Slush stratum codes:
21 stale (SE_STALE, SE_INVALID_JOBID, SE_NTIME_INVALID)
22 duplicate (SE_DUPE)
23 low diff (SE_HIGH_DIFF)
24 unauthorized (SE_NO_USERNAME)
20 other (everything else)
All 14 JSON_ERR call sites in parse_submit pick this up
automatically — no other call site changes needed. Generated with
git diff against the pinned upstream commit so format is correct;
round-trip-tested with git apply --check.
Upstream ckpool-solo stubs out send_api_response as a no-op, so the
stratifier socket accepts commands like poolstats/users/workers but
never writes anything back — kamado-api always gets EOF.
Replace the stub with a real implementation: serialize the json_t
with json_dumps, write via send_unix_msg, and free. The stratum_loop
retry label handles socket cleanup.
Upstream ckpool tracks user/worker best_ever across block solves but
only emits it in the on-disk users.json/workers.json persistence files,
not in the runtime socket API (userinfo() / workerinfo() in
stratifier.c). Consumers that poll the socket — like kamado-api — see
bestdiff reset to zero on every block solve via reset_bestshares()
with no all-time field to fall back on. That is the "best share stuck
at zero after a block" UX we want to fix in Kamado.
0001-expose-bestever-in-runtime-json.patch adds bestever to the two
JSON_CPACK calls so the UI can show current-round and all-time side
by side. No behavioral change, no impact on share validation or block
handling. Candidate for upstreaming.
Also surfaces BestEver in api/internal/ckpool types (User, Worker).
Gracefully degrades to 0 on an unpatched ckpool.
Forks CKPool-solo at upstream commit cfb0f83b (v1.0) via a multi-stage
Docker build, with an env-driven entrypoint that renders ckpool.conf from
a template. The pinned commit already includes every fix referenced in
Bassin issue #29 (workbase_id double increment, extended low-power
timeouts, configurable dropidle, vardiff burst handling).
No patches are applied yet — the patches/ directory holds the workflow
and build wiring so Kamado-specific patches can be added incrementally.
Build is portable across aarch64/x86_64: yasm is intentionally omitted
so CKPool falls back to its C SHA256, and CFLAGS override drops upstream's
default -march=native. Runtime image ships ckpool and ckpmsg for socket
debugging; share logging (-L) is enabled by default.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
satoshi