Submit-first ordering. Patch 0004 now calls generator_submitblock
BEFORE writing the pending-block hex to disk. The happy path adds zero
disk I/O — we only dump when the primary returned false. The same
patch bounds generator_submitblock's "no live current_si" spin to
~3s instead of the original infinite loop, so a permanently-down
primary doesn't pin the stratifier; the bounded spin lets the caller
return false and lets local_block_submit dump for kamado-api to take
over.
Default grace lowered from 30s to 3s. With ckpool's bounded spin and
sub-second sweep cadence, the fallback now reacts within ~4s of a
failed primary submit — fast enough that the work is still relevant
for the current chain tip. The submitter's sweep poll dropped to 1s
to match.
UI HealthBanners. New top-of-page strip surfaces:
* Fallback used (red banner, 24h after most recent event):
"primary bitcoind didn't accept; backup X took over Y ago"
* Submit gap (orange banner, only when no recent fallback):
"N blocks attempted but unconfirmed — configure backups"
* ZMQ stale (orange banner): no hashblock frame in 30+ minutes
Operators see degraded-but-not-fatal states without checking logs.
Startup readiness gate. main now waits up to 8s on agg.Ready() before
starting the HTTP server so the very first /api/snapshot doesn't show
all-zero state during the aggregator's first refresh. Capped so a
permanently-down bitcoind can't block startup; /healthz is honest
about the degraded state once we do start serving.
P1 audits / fixes:
* Bitcoin Core RPC now retries up to 3 times with linear backoff on
transport errors, 5xx responses, and warm-up/loading RPC errors
(code -28). Hard "no" answers (block-not-found etc.) bubble up
immediately so we don't mask real errors.
* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
(~30s with the default poll cadence). Stuck readers no longer hold
stale snapshots indefinitely or freeze hub state.
* ZMQ subscriber freshness: aggregator records the last-event
timestamp, surfaces zmq_enabled / has_last_zmq_event /
last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
the gap exceeds 30 minutes.
* /healthz expanded with submit_attempts / submits_confirmed /
submit_gap, fallback_submits_total + last_fallback_*, and the zmq
staleness check. Now usable as a real-world ops dashboard signal.
Block-broadcast fallback (new feature):
* ckpool patch 0004: hooks local_block_submit to write the raw block
hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
invoking generator_submitblock. Unlinks on success. ckpool's normal
flow is otherwise untouched.
* api/internal/blocksubmit: watcher polls the dir every 5s. Files
sitting longer than the grace window (default 30s, configurable)
are re-broadcast through operator-supplied backup RPC URLs in
sequence. Treats both null and any "duplicate*" reject reason as
success (the block landed). Pre-checks the primary chain first so
a stale file from a successful-but-unlinked submit gets cleaned
up without bothering fallbacks.
* Aggregator records each successful fallback submission as a
persistent counter and surfaces it in the snapshot so the UI can
show a "primary bitcoind isn't accepting submits" alert.
* Config: BACKUP_RPC_URLS (comma- or newline-separated, with
optional inline credentials) plus PENDING_BLOCKS_DIR and
PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
https://user:pass@host:port/ works cleanly.
The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
Closes the silent-failure modes between "ckpool logs a solve" and
"block correctly displayed":
* Difficulty estimate matched mempool.space — the projection now uses
(inEpoch + 1) intervals so it converges on Bitcoin Core's eventual
retarget formula at end-of-epoch instead of undershooting by ~0.05–
0.10 % throughout.
* Tailer resumes mid-log on restart — persists (inode, offset) to kv
every EOF + on shutdown, and replays the unread tail next time. Any
solve line written while kamado-api was down would previously be
invisible forever.
* Background reconcile loop (60 s) retries hash/reward enrichment for
blocks the original RPC missed, so a transient bitcoind-index race no
longer permanently leaves a block hashless.
* Reorg detection: same loop compares each recent stored hash against
getblockhash(height); a mismatch stamps orphaned_at. UI renders these
strikethrough with a red "orphaned" tag instead of showing illusory
rewards forever.
* InsertBlock now reports whether a row was actually inserted; the
caller WARN-logs duplicate-height ignores so a re-mined orphaned
height can't disappear silently.
* Submit-attempt vs confirmed counters surface failed submissions:
every "Possible/Submitting block solve" log line increments
block_submit_attempts; "Solved and confirmed" increments
block_submits_confirmed. A growing gap means bitcoind is rejecting
our submissions — previously invisible.
* share_err patch refreshed against pinned ckpool source: added
SE_NO_JOBID -> 21 and SE_WORKER_MISMATCH -> 24 mappings, kept
SE_INVALID_NONCE2 in 20 (it's a malformed-input error, not low-diff).
AxeOS users now see actionable Stratum codes instead of
"unknown error".
UI gets new orphaned_at + block_submit_attempts/confirmed fields on
the snapshot type and a strikethrough-with-tag rendering for orphaned
blocks in BlocksTable.
ckpool's stratum_instance.address (exposed as the "address" field in
the runtime JSON) is the SOURCE IP of the connection — set from
inet_ntop in connector.c — not the BTC payout address. The miner's
BTC payout comes from the stratum username, which ckpool stores in
worker.user (and in the dotted prefix of worker.workername).
Every "user" reference in the dashboard was reading c.address and
treating it as the BTC. Effects:
• clicking an online miner navigated to #/user/<source-ip>; the
UserDetailPage filters never matched and the page rendered with
junk values (or the user struct's stale residual hashrate when
the BTC happened to come from a sibling row).
• offline-miner clicks worked, but totals came from user.dsps*
which decay slowly inside ckpool, so a miner that had just
disconnected still showed positive hashrate for several minutes.
• the "Best (ever)" tile fell back to bestdiff (session) when
bestever was zero, so it lied about its semantics.
Fixes:
• Add btcAddressOf() helper and use w.user (or it) when extracting
the BTC for the user-link button. The source IP gets its own
sub-line under the worker name, clearly labelled.
• Redesign UserDetailPage: filter clients by workername prefix
against the BTC, never by c.address; compute hashrate totals by
SUMMING the user's currently-connected clients (so 0 online
clients => 0 hashrate, no stale decay artifacts); compute
best_ever as max across the user's worker.bestever values; show
online/total worker counts and a per-worker status pill.
• Add an explorer link (mempool.space) for the user's BTC.
TLS detection moves to a clean signal: ckpool now binds two stratum
sockets — public plaintext and loopback-only. stunnel forwards to
the loopback bind, so TLS clients arrive with c.server == 1. The
dashboard reads that and renders a green TLS pill next to the
worker name. No source-IP heuristics needed.
Open-source mark: new isOpenSource() heuristic over the stratum
useragent matches Bitaxe family (NerdAxe / NerdQAxe / NerdMiner /
NerdOctaxe / Lucky / QAxe / MCCM), Braiins OS, cgminer / bfgminer /
ckminer, and ESP32 builds. Renders as an orange ★ next to the
hardware label, matching public-pool's convention.
types.ts: document StratumClient.address (source IP, not BTC) and
add the previously-undeclared `server` field. Surfacing the runtime
value that has been there all along since cfb0f83.
ckpool's share-rejection response sets the "error" field to a bare
JSON string ("Stale", "Above target", "Duplicate", ...). The
Stratum mining v1 spec, and every miner firmware that follows it,
expects that field to be a [code, message, traceback] array. AxeOS
parses with cJSON_GetArrayItem after a cJSON_IsArray check, sees a
string, falls back to "unknown error", and the rejection reason
disappears from the dashboard. Bassin / public-pool sends the
array form, which is why those rejections render as "stale" there.
Patch the JSON_ERR macro in stratifier.c to wrap the existing
share_errs[] string in json_pack("[isn]", ...) and add a small
share_err_code() helper mapping ckpool's enum to the standard
Slush stratum codes:
21 stale (SE_STALE, SE_INVALID_JOBID, SE_NTIME_INVALID)
22 duplicate (SE_DUPE)
23 low diff (SE_HIGH_DIFF)
24 unauthorized (SE_NO_USERNAME)
20 other (everything else)
All 14 JSON_ERR call sites in parse_submit pick this up
automatically — no other call site changes needed. Generated with
git diff against the pinned upstream commit so format is correct;
round-trip-tested with git apply --check.
Upstream ckpool-solo stubs out send_api_response as a no-op, so the
stratifier socket accepts commands like poolstats/users/workers but
never writes anything back — kamado-api always gets EOF.
Replace the stub with a real implementation: serialize the json_t
with json_dumps, write via send_unix_msg, and free. The stratum_loop
retry label handles socket cleanup.
Upstream ckpool tracks user/worker best_ever across block solves but
only emits it in the on-disk users.json/workers.json persistence files,
not in the runtime socket API (userinfo() / workerinfo() in
stratifier.c). Consumers that poll the socket — like kamado-api — see
bestdiff reset to zero on every block solve via reset_bestshares()
with no all-time field to fall back on. That is the "best share stuck
at zero after a block" UX we want to fix in Kamado.
0001-expose-bestever-in-runtime-json.patch adds bestever to the two
JSON_CPACK calls so the UI can show current-round and all-time side
by side. No behavioral change, no impact on share validation or block
handling. Candidate for upstreaming.
Also surfaces BestEver in api/internal/ckpool types (User, Worker).
Gracefully degrades to 0 on an unpatched ckpool.
Forks CKPool-solo at upstream commit cfb0f83b (v1.0) via a multi-stage
Docker build, with an env-driven entrypoint that renders ckpool.conf from
a template. The pinned commit already includes every fix referenced in
Bassin issue #29 (workbase_id double increment, extended low-power
timeouts, configurable dropidle, vardiff burst handling).
No patches are applied yet — the patches/ directory holds the workflow
and build wiring so Kamado-specific patches can be added incrementally.
Build is portable across aarch64/x86_64: yasm is intentionally omitted
so CKPool falls back to its C SHA256, and CFLAGS override drops upstream's
default -march=native. Runtime image ships ckpool and ckpmsg for socket
debugging; share logging (-L) is enabled by default.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
satoshi