Submit-first ordering. Patch 0004 now calls generator_submitblock
BEFORE writing the pending-block hex to disk. The happy path adds zero
disk I/O — we only dump when the primary returned false. The same
patch bounds generator_submitblock's "no live current_si" spin to
~3s instead of the original infinite loop, so a permanently-down
primary doesn't pin the stratifier; the bounded spin lets the caller
return false and lets local_block_submit dump for kamado-api to take
over.
Default grace lowered from 30s to 3s. With ckpool's bounded spin and
sub-second sweep cadence, the fallback now reacts within ~4s of a
failed primary submit — fast enough that the work is still relevant
for the current chain tip. The submitter's sweep poll dropped to 1s
to match.
UI HealthBanners. New top-of-page strip surfaces:
* Fallback used (red banner, 24h after most recent event):
"primary bitcoind didn't accept; backup X took over Y ago"
* Submit gap (orange banner, only when no recent fallback):
"N blocks attempted but unconfirmed — configure backups"
* ZMQ stale (orange banner): no hashblock frame in 30+ minutes
Operators see degraded-but-not-fatal states without checking logs.
Startup readiness gate. main now waits up to 8s on agg.Ready() before
starting the HTTP server so the very first /api/snapshot doesn't show
all-zero state during the aggregator's first refresh. Capped so a
permanently-down bitcoind can't block startup; /healthz is honest
about the degraded state once we do start serving.
P1 audits / fixes:
* Bitcoin Core RPC now retries up to 3 times with linear backoff on
transport errors, 5xx responses, and warm-up/loading RPC errors
(code -28). Hard "no" answers (block-not-found etc.) bubble up
immediately so we don't mask real errors.
* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
(~30s with the default poll cadence). Stuck readers no longer hold
stale snapshots indefinitely or freeze hub state.
* ZMQ subscriber freshness: aggregator records the last-event
timestamp, surfaces zmq_enabled / has_last_zmq_event /
last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
the gap exceeds 30 minutes.
* /healthz expanded with submit_attempts / submits_confirmed /
submit_gap, fallback_submits_total + last_fallback_*, and the zmq
staleness check. Now usable as a real-world ops dashboard signal.
Block-broadcast fallback (new feature):
* ckpool patch 0004: hooks local_block_submit to write the raw block
hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
invoking generator_submitblock. Unlinks on success. ckpool's normal
flow is otherwise untouched.
* api/internal/blocksubmit: watcher polls the dir every 5s. Files
sitting longer than the grace window (default 30s, configurable)
are re-broadcast through operator-supplied backup RPC URLs in
sequence. Treats both null and any "duplicate*" reject reason as
success (the block landed). Pre-checks the primary chain first so
a stale file from a successful-but-unlinked submit gets cleaned
up without bothering fallbacks.
* Aggregator records each successful fallback submission as a
persistent counter and surfaces it in the snapshot so the UI can
show a "primary bitcoind isn't accepting submits" alert.
* Config: BACKUP_RPC_URLS (comma- or newline-separated, with
optional inline credentials) plus PENDING_BLOCKS_DIR and
PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
https://user:pass@host:port/ works cleanly.
The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
Closes the silent-failure modes between "ckpool logs a solve" and
"block correctly displayed":
* Difficulty estimate matched mempool.space — the projection now uses
(inEpoch + 1) intervals so it converges on Bitcoin Core's eventual
retarget formula at end-of-epoch instead of undershooting by ~0.05–
0.10 % throughout.
* Tailer resumes mid-log on restart — persists (inode, offset) to kv
every EOF + on shutdown, and replays the unread tail next time. Any
solve line written while kamado-api was down would previously be
invisible forever.
* Background reconcile loop (60 s) retries hash/reward enrichment for
blocks the original RPC missed, so a transient bitcoind-index race no
longer permanently leaves a block hashless.
* Reorg detection: same loop compares each recent stored hash against
getblockhash(height); a mismatch stamps orphaned_at. UI renders these
strikethrough with a red "orphaned" tag instead of showing illusory
rewards forever.
* InsertBlock now reports whether a row was actually inserted; the
caller WARN-logs duplicate-height ignores so a re-mined orphaned
height can't disappear silently.
* Submit-attempt vs confirmed counters surface failed submissions:
every "Possible/Submitting block solve" log line increments
block_submit_attempts; "Solved and confirmed" increments
block_submits_confirmed. A growing gap means bitcoind is rejecting
our submissions — previously invisible.
* share_err patch refreshed against pinned ckpool source: added
SE_NO_JOBID -> 21 and SE_WORKER_MISMATCH -> 24 mappings, kept
SE_INVALID_NONCE2 in 20 (it's a malformed-input error, not low-diff).
AxeOS users now see actionable Stratum codes instead of
"unknown error".
UI gets new orphaned_at + block_submit_attempts/confirmed fields on
the snapshot type and a strikethrough-with-tag rendering for orphaned
blocks in BlocksTable.
isOpenSource was matching things like Braiins OS, cgminer, bfgminer,
and a generic "esp32" — software that's open but runs on closed
hardware (Antminer, unknown rigs). public-pool's UI reserves the
star for open-source HARDWARE, with open firmware on top, and that's
the convention to match. New regex covers the family verbatim from
public-pool-ui's user-agent-link switch table:
bitaxe, bitaxeHex, NerdMiner, NerdNOS, NerdAxe, NerdAxeGamma,
NerdOCTAXE, NerdEKO, NerdQAxe+, NerdQAxe++, PiAxe, QAxe, QAxe+,
0xAxe, LeafMiner
Also extend detectHardware so all those user-agents get a recognised
label instead of "esp32" / raw token.
Replace the squished "TLS" pill with an inline-flex badge: 0.78em
text, 5px corners, padlock SVG, generous left/right padding so it
reads at a glance instead of looking like a typo.
Custom block explorer: new optional union under StartOS Advanced
config ("Block Explorer" — defaults to "mempool.space"). Picking
"Custom URL" surfaces the value as MEMPOOL_BASE_URL on the
container env. config.Config picks it up, the aggregator copies
it into every Snapshot (mempool_base_url field), and the UI's
shared explorerBaseFor() helper trusts the custom URL verbatim
when present (no /testnet4 / /signet path appended — a self-hosted
instance is presumably single-network already). Falls back to the
public mempool.space mirrors per chain when unset, which is the
default behaviour.
New internal/zmqmon package subscribes to bitcoind's hashblock
ZMQ topic and emits TipEvents. Uses github.com/go-zeromq/zmq4
(pure Go, builds with CGO_ENABLED=0). Exponential backoff on
connection failure so bitcoind restarts don't kill the
subscriber permanently; event channel drops rather than blocks
if the consumer is slow (signals are advisory, not logs).
Aggregator.Run now takes a <-chan TipEvent; when a tip arrives
it fires an immediate refresh() outside the normal ticker
cadence. With a 5s poll interval and 0.5-1s ZMQ latency from
bitcoind, dashboards now reflect new tips roughly 4x faster.
Endpoint comes from BITCOIN_ZMQ_BLOCK — empty disables ZMQ
entirely and the aggregator just runs on the ticker alone.
New internal/store package wraps modernc.org/sqlite (pure Go,
no CGO) with a BlockStore that exposes Open/Close/InsertBlock/
Recent. The aggregator now accepts an optional *store.BlockStore;
on Run() it loads up to maxBlockHistory rows from the store before
the first refresh, and each ingested block gets written to the
DB before being appended to the in-memory ring.
main.go opens the store at cfg.DBPath and logs a warning + falls
back to in-memory-only if the file can't be created — a broken
data volume shouldn't stop the pool from running.
InsertBlock uses INSERT OR IGNORE on the height primary key so
replayed log events after a restart are harmless.
Adds real-time block detection via ckpool log tailing and a push
channel for the upcoming Svelte UI, all stdlib-only:
- logmon.Tailer follows ckpool.log with rotation/truncation survival
(inode + size tracking) and parses "Solved and confirmed block N"
into BlockEvent values.
- state.Aggregator grows a 256-entry block ring, an OnRefresh hook,
and IngestBlockEvents which best-effort enriches events with the
block hash via bitcoind getblockhash.
- httpapi.Hub implements RFC 6455 from scratch (SHA1 handshake,
unmasked text frames out, masked frames in, ping keepalive,
per-client write mutex, slow-client drop) so we don't pull in a
ws dependency before we can go mod tidy.
- New routes: GET /api/blocks and GET /api/ws. Snapshot pushes fire
on every poll tick and immediately on block-solve.
ZMQ hashblock subscription and SQLite persistence are deferred to
Phase 2b.5 once the s9pk packaging repo exists and we have a real
build environment for adding Go deps.
Go 1.22 module that polls CKPool's Unix socket control API, queries
Bitcoin Core over JSON-RPC, merges both into a thread-safe snapshot, and
serves it over REST. Layout:
api/
├── cmd/kamado-api/main.go signal-aware entrypoint
└── internal/
├── config/ env var loader with validation
├── ckpool/ socket client (4-byte LE length-prefixed wire
│ protocol verified against libckpool.c), typed
│ response models for poolstats/users/workers/
│ clients/uptime, + unit tests using a fake
│ unix socket server
├── bitcoind/ minimal JSON-RPC client, getblockchaininfo
│ and getnetworkhashps
├── state/ Aggregator that refreshes a merged Snapshot
│ on a ticker; readers get a copy under RWMutex
└── httpapi/ REST handlers on Go 1.22 ServeMux:
/api/health /api/pool /api/users
/api/workers /api/clients /api/snapshot
CKPool stores hashrate as "dsps" (diff shares per second); we convert
to H/s via the 2^32 constant used by GoBrrr-Pool and other clients.
Every stat CKPool exposes to its socket API is surfaced — useragent,
IP, per-client diff, per-worker best diff — closing the gap against
Bassin which only reads the 60-second stats files.
Dockerfile does a CGO_ENABLED=0 static build on golang:1.22-bookworm
with -trimpath -ldflags=-s -w. docker-compose now runs both ckpool and
kamado-api, sharing a named volume for /run/ckpool so the API can
dial the stratifier socket directly.
Deferred to Phase 2b (called out in README):
- Bitcoin Core ZMQ hashblock subscriber
- WebSocket push for real-time UI updates
- SQLite persistence (block history, best-share history)
- CKPool log tailer for "Solved and confirmed block" detection
Tests and build NOT run in this commit — Go isn't installed in the
dev environment. Run `make api-test` or `make api` to verify.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
satoshi