When bitcoind is unreachable for 3+ consecutive polls and no block
submission is pending, SIGTERM the ckpool process so miners disconnect
and failover to backup pools. Adds a red dashboard banner when
Bitcoin Core is down.
On ZMQ hashblock events, wait for the ckpool "Block update latency"
log line before firing dashboard RPC calls (getblockchaininfo,
getnetworkhashps, etc). This avoids competing with ckpool's
latency-critical getblocktemplate call for bitcoind RPC threads,
which matters when many services flood bitcoind on every new block.
- Best share page: hex + binary hash comparison against network target,
per-bit coloring showing exactly which bits prevented a valid block,
toggle between network diff at time of finding vs current diff
- Capture best share hash from ckpool logs with one-time backfill
- Persist network difficulty at time of best share for historical accuracy
- Add miner (worker) column to blocks table via coinbase address matching
- Truncate block hashes in table with full hash on hover
- Increase hashrate chart Y-axis to 7 ticks for better readability
Backend:
- Persist best_diff as a high-water mark in kv store so it survives
restarts and transient ckpool gaps
- Persist acked_best_diff so the "new best share" glow survives pool
restarts and works across multiple new bests before dismissal
- Add POST /api/admin/ack-best and /api/admin/reset-ack-best endpoints
Frontend:
- Best share card: golden glow, shimmer, sparkles, "NEW BEST SHARE!"
tag with previous value, "Nice" dismiss button
- Optimistic local override so Nice/test buttons respond instantly
instead of waiting for next WebSocket push
- Fix shimmer sweep to go left-to-right (not start from middle)
- Share animations: reject (red flash + shake) suppresses accept;
reject effect declared first so guard reads correct state
- Share tooltips show exact counts on hover
- Block reward tooltip shows full amount in satoshis
- Expected block uses 1h hashrate instead of 1m
- formatDuration uses full unit names; years always show 1 decimal
- formatWork always shows 2 decimal places
- Map chain "main" to "mainnet" in block height card
Replace the subtle petal animation with a full block-found party:
confetti rain, sakura petals, ember burst, and a dismissible banner
showing block height and reward. Triggers on pool block finds (not
every network block). Animations loop until the user clicks dismiss.
Fix ZMQ stale banner false-positiving during long block intervals by
comparing ZMQ age against tip-change age instead of a fixed 30-min
threshold. Add tip_changed_age to the snapshot so the UI can tell
"no blocks on the network" from "ZMQ is broken". Use solid background
colors on health banners instead of transparent rgba.
Zeroes block-update latency counters (count, sum, last, wasted work)
in both memory and the kv store. Used by the StartOS reset-latency
action to let operators start fresh after tuning.
Log ZMQ→mining.notify latency in ckpool (patch 0005), expose a raw
reject counter (patch 0006), and surface both in the dashboard:
- Block latency card shows avg/last ms, wasted work, and block count
- SharesBar component shows session + all-time accepted/rejected with
a Demon Slayer flame-slash animation on new shares
- Miners card info moved to MinersTable section header
- Latency stats and share counts persist across restarts via kv store
- Removed misleading pool-wide share stats from per-worker/user pages
(ckpool doesn't expose per-user raw counts)
- Change OrphanedAt from time.Time to *time.Time so JSON omitempty
correctly omits zero values (Go serializes zero time.Time as
"0001-01-01T00:00:00Z" which JS treats as truthy)
- Rewrite reconcileOnce with 4-pass architecture: enrichment, reorg
detection via GetBlock verification, false-positive un-orphaning,
and legacy block stamping
- Use inferOtherChain() to stamp cross-network blocks with the correct
chain identifier (testnet4 reports as "testnet4", not "test")
- Move tailer startup after aggregator's first refresh so blocks are
always stamped with the current chain
- Add StampChain store method, debug-blocks admin endpoint, and
comprehensive reconcile tests for cross-network scenarios
Tests cover the four subsystems that sit between a share submission and
a confirmed block reward:
bitcoind/rpc_test.go — isRetryable classification, 3-attempt backoff
on 503, no retry on semantic RPC errors (-5), retry on warm-up (-28),
CoinbaseReward calculation.
logmon/tailer_test.go — handleLine regex parsing, share-diff carry from
attempt line to block event, diff reset after use, unrelated-line
silence, file-read via goroutine, cursor save/resume across restart,
log rotation re-open.
store/blocks_test.go — InsertBlock roundtrip (all fields including
chain), idempotent dedup, MarkOrphaned, UpdateEnrichment,
BlocksNeedingEnrichment, KV get/set/overwrite, and migration of an
existing DB that lacks the chain column.
state/blocks_test.go — reconcileOnce chain-filter (testnet block not
orphaned on mainnet), genuine reorg detection (different canonical
hash → orphaned), transient RPC error (no orphan on -5), legacy
empty-chain block still checked for reorgs, IngestBlockEvents new
block persisted + confirmed counter incremented, dedup skips counter
and OnRefresh, chain stamped from current snapshot at ingest time.
Blocks are now stamped with the Bitcoin network name ("main", "test",
"signet") at ingest time. The reconcile loop's Pass 2 skips any stored
block whose chain differs from the node's current chain, preventing
testnet blocks from being falsely orphaned after switching back to
mainnet. Legacy rows with an empty chain field fall through unchanged.
The UI shows a blue "test" / "signet" badge next to blocks from a
non-current network so operators can distinguish cross-chain history
from genuine reorg-orphaned blocks.
Removes the entire fallback submitter mechanism: ckpool patch 0004,
the blocksubmit package, the wiring in main.go, the config fields,
the aggregator's fallback counters and snapshot fields, the healthz
fallback fields, and the TS type fields plus the HealthBanners
fallback alert.
Reasoning: ckpool's primary bitcoind submission must remain the
single source of truth, and getting the parallel "race a fallback
during the submit" semantics right is more architectural complexity
than the marginal reliability gain justifies. The original upstream
behavior — submit to bitcoind, retry indefinitely if unavailable —
is what we want.
Kept intact:
* Submit-attempt vs confirmed counters (block_submit_attempts /
block_submits_confirmed). Useful on their own as a "did bitcoind
confirm the submission?" signal.
* HealthBanners shows submit_gap and zmq_stale only.
* /healthz exposes submit_gap, zmq_stale, etc.
* All P0 reliability work (tailer cursor, reconcile loop, reorg
detection, multi-solve guard) and other P1 (RPC retry, WS
back-pressure, ZMQ tracking, startup readiness gate).
Submit-first ordering. Patch 0004 now calls generator_submitblock
BEFORE writing the pending-block hex to disk. The happy path adds zero
disk I/O — we only dump when the primary returned false. The same
patch bounds generator_submitblock's "no live current_si" spin to
~3s instead of the original infinite loop, so a permanently-down
primary doesn't pin the stratifier; the bounded spin lets the caller
return false and lets local_block_submit dump for kamado-api to take
over.
Default grace lowered from 30s to 3s. With ckpool's bounded spin and
sub-second sweep cadence, the fallback now reacts within ~4s of a
failed primary submit — fast enough that the work is still relevant
for the current chain tip. The submitter's sweep poll dropped to 1s
to match.
UI HealthBanners. New top-of-page strip surfaces:
* Fallback used (red banner, 24h after most recent event):
"primary bitcoind didn't accept; backup X took over Y ago"
* Submit gap (orange banner, only when no recent fallback):
"N blocks attempted but unconfirmed — configure backups"
* ZMQ stale (orange banner): no hashblock frame in 30+ minutes
Operators see degraded-but-not-fatal states without checking logs.
Startup readiness gate. main now waits up to 8s on agg.Ready() before
starting the HTTP server so the very first /api/snapshot doesn't show
all-zero state during the aggregator's first refresh. Capped so a
permanently-down bitcoind can't block startup; /healthz is honest
about the degraded state once we do start serving.
P1 audits / fixes:
* Bitcoin Core RPC now retries up to 3 times with linear backoff on
transport errors, 5xx responses, and warm-up/loading RPC errors
(code -28). Hard "no" answers (block-not-found etc.) bubble up
immediately so we don't mask real errors.
* WebSocket hub disconnects clients that miss 6 consecutive broadcasts
(~30s with the default poll cadence). Stuck readers no longer hold
stale snapshots indefinitely or freeze hub state.
* ZMQ subscriber freshness: aggregator records the last-event
timestamp, surfaces zmq_enabled / has_last_zmq_event /
last_zmq_event_age in the snapshot. /healthz flags zmq_stale when
the gap exceeds 30 minutes.
* /healthz expanded with submit_attempts / submits_confirmed /
submit_gap, fallback_submits_total + last_fallback_*, and the zmq
staleness check. Now usable as a real-world ops dashboard signal.
Block-broadcast fallback (new feature):
* ckpool patch 0004: hooks local_block_submit to write the raw block
hex to <logdir>/pending-blocks/<height>-<hash16>.hex right before
invoking generator_submitblock. Unlinks on success. ckpool's normal
flow is otherwise untouched.
* api/internal/blocksubmit: watcher polls the dir every 5s. Files
sitting longer than the grace window (default 30s, configurable)
are re-broadcast through operator-supplied backup RPC URLs in
sequence. Treats both null and any "duplicate*" reject reason as
success (the block landed). Pre-checks the primary chain first so
a stale file from a successful-but-unlinked submit gets cleaned
up without bothering fallbacks.
* Aggregator records each successful fallback submission as a
persistent counter and surfaces it in the snapshot so the UI can
show a "primary bitcoind isn't accepting submits" alert.
* Config: BACKUP_RPC_URLS (comma- or newline-separated, with
optional inline credentials) plus PENDING_BLOCKS_DIR and
PENDING_BLOCKS_GRACE. URLs are parsed via net/url so
https://user:pass@host:port/ works cleanly.
The fallback is opt-in and disabled by default. Once enabled with at
least one URL, a primary bitcoind outage at the moment of solving no
longer means a lost block — kamado-api re-broadcasts via whichever
backup the operator trusts (a second self-hosted node, an
authenticated public RPC service, etc.).
Closes the silent-failure modes between "ckpool logs a solve" and
"block correctly displayed":
* Difficulty estimate matched mempool.space — the projection now uses
(inEpoch + 1) intervals so it converges on Bitcoin Core's eventual
retarget formula at end-of-epoch instead of undershooting by ~0.05–
0.10 % throughout.
* Tailer resumes mid-log on restart — persists (inode, offset) to kv
every EOF + on shutdown, and replays the unread tail next time. Any
solve line written while kamado-api was down would previously be
invisible forever.
* Background reconcile loop (60 s) retries hash/reward enrichment for
blocks the original RPC missed, so a transient bitcoind-index race no
longer permanently leaves a block hashless.
* Reorg detection: same loop compares each recent stored hash against
getblockhash(height); a mismatch stamps orphaned_at. UI renders these
strikethrough with a red "orphaned" tag instead of showing illusory
rewards forever.
* InsertBlock now reports whether a row was actually inserted; the
caller WARN-logs duplicate-height ignores so a re-mined orphaned
height can't disappear silently.
* Submit-attempt vs confirmed counters surface failed submissions:
every "Possible/Submitting block solve" log line increments
block_submit_attempts; "Solved and confirmed" increments
block_submits_confirmed. A growing gap means bitcoind is rejecting
our submissions — previously invisible.
* share_err patch refreshed against pinned ckpool source: added
SE_NO_JOBID -> 21 and SE_WORKER_MISMATCH -> 24 mappings, kept
SE_INVALID_NONCE2 in 20 (it's a malformed-input error, not low-diff).
AxeOS users now see actionable Stratum codes instead of
"unknown error".
UI gets new orphaned_at + block_submit_attempts/confirmed fields on
the snapshot type and a strikethrough-with-tag rendering for orphaned
blocks in BlocksTable.
isOpenSource was matching things like Braiins OS, cgminer, bfgminer,
and a generic "esp32" — software that's open but runs on closed
hardware (Antminer, unknown rigs). public-pool's UI reserves the
star for open-source HARDWARE, with open firmware on top, and that's
the convention to match. New regex covers the family verbatim from
public-pool-ui's user-agent-link switch table:
bitaxe, bitaxeHex, NerdMiner, NerdNOS, NerdAxe, NerdAxeGamma,
NerdOCTAXE, NerdEKO, NerdQAxe+, NerdQAxe++, PiAxe, QAxe, QAxe+,
0xAxe, LeafMiner
Also extend detectHardware so all those user-agents get a recognised
label instead of "esp32" / raw token.
Replace the squished "TLS" pill with an inline-flex badge: 0.78em
text, 5px corners, padlock SVG, generous left/right padding so it
reads at a glance instead of looking like a typo.
Custom block explorer: new optional union under StartOS Advanced
config ("Block Explorer" — defaults to "mempool.space"). Picking
"Custom URL" surfaces the value as MEMPOOL_BASE_URL on the
container env. config.Config picks it up, the aggregator copies
it into every Snapshot (mempool_base_url field), and the UI's
shared explorerBaseFor() helper trusts the custom URL verbatim
when present (no /testnet4 / /signet path appended — a self-hosted
instance is presumably single-network already). Falls back to the
public mempool.space mirrors per chain when unset, which is the
default behaviour.
Block-reward tile was cached for 60s, so users saw it sit still even
as bitcoind's CreateNewBlock fired every few seconds with updated
fee totals. Drop the template cache TTL to 15s — bitcoind caches
the template internally, so the extra RPC cost is trivial.
Hashrate chart was rendering 1440 raw per-minute points across a
~780px plot, which collapsed into a noisy smear. Bucket-average to
~96 points so the chart actually communicates a trend. Raw samples
under the target pass through unchanged (early process lifetime).
Layout: Blocks found is now full-width, with the Best shares
leaderboard stacked below it. The previous side-by-side was
squeezing both tables on typical displays.
New user-detail modal. Clicking a BTC address in the Miners table
opens a focused view: per-user aggregate hashrate (1m/5m/1h/24h),
best round / best ever, and a worker-level breakdown with the same
columns as the main table. Driven by a small selection store so any
component in the tree can open it. Closes on backdrop click or Esc.
New block-found animation. When the network tip advances, a brief
full-screen overlay plays: a radial Hinokami Kagura ember burst
above the header, a faint sun-ray sweep, and ~22 falling sakura
petals with randomised drift / rotation / delay so no two blocks
look identical. The existing block-height tile flash still fires
alongside; the overlay is pointer-events: none so nothing in the
UI becomes unreachable during the ~3.6s animation.
Total work was summing pool.Shares (ckpool's accounted_shares — raw
per-share count) and multiplying by 2^32, which is nonsense: each
share's actual difficulty was ignored, so a pool running at any
real hashrate would show a tiny number. Switch to pool.Accepted,
which ckpool exposes as accounted_diff_shares (sum of each accepted
share's difficulty in diff-1-normalized units). cumulative_shares *
2^32 is now actually total hashes.
Bump the kv key from "cumulative_shares" to "cumulative_work" so
any value saved under the old name is orphaned rather than mixed
into the new (correctly-unit'd) counter on upgrade.
Total work tile's sub-info now shows the round effort:
cumulative_shares / network_difficulty * 100. Matches ckpool's own
formula at stratifier.c:8201 for the percent-of-block display.
Drop the duplicate effort text from the Expected block tile, which
was an uptime-based approximation of the same thing.
The diff-adjustment tile was showing raw epoch progress (blocks-so-far
/ 2016 * 100%), which is never what anyone wants to see there. Compute
the real predicted change server-side: fetch the timestamp of the
first block in the current retarget epoch once per epoch, then each
refresh derive (expected_interval / actual_interval - 1) * 100,
clamped to Bitcoin's consensus bounds of [-75%, +300%]. The tile now
renders the signed percent, green when positive, red when negative.
Tile layout is forced to an explicit 5x2 grid so the number of
columns doesn't depend on viewport width. Reordering puts the
difficulty tile at (row 1, col 5) with the next-adjustment tile
directly below it at (row 2, col 5); block height, block reward,
total work, and expected block fill the rest of the second row.
Mobile breakpoints fall back to 3 and then 2 columns.
Header keeps only a live/connecting/offline status dot — the chain
name and block height badges were redundant now that the height tile
exists and animates on its own.
Stat-sub text bumped from 0.85em to 0.95em for readability.
Background: a warm ember-glow radial at the top of the viewport
layered with two accent radials and a low-opacity haori-checker tile
pattern (inline SVG data URI). Cards get a translucent backdrop so
the texture peeks through between them.
Dashboard now renders 10 tiles in a 5x2 overview: hashrate, best
share, miners, network hashrate, and expected block on the top row;
difficulty, block height, block reward, total work, and the
difficulty-adjustment countdown on the bottom row. Difficulty is
rendered with T/P suffixes instead of scientific notation, the main
hashrate card shows the 1-minute value, and the block-height tile
pulses orange when the network tip advances.
Added a 24-hour hashrate area chart below the overview, sampled
once per minute. Samples are persisted to a new hashrate_samples
SQLite table and restored on startup so the chart doesn't reset
every time kamado-api is restarted.
Cumulative pool work (sum of accepted diff-1-normalized shares) is
now tracked across ckpool restarts. The aggregator integrates only
positive deltas on pool.Shares — a regression means ckpool's
counter reset to zero and the baseline is refreshed without losing
the running total. A hasPoolSharesBaseline flag prevents double-
counting on the first refresh after a kamado-api restart. The
value is persisted to a new kv table once per minute.
Next-block reward (subsidy + fees) is fetched from bitcoind
getblocktemplate at most once per minute and surfaced as a tile.
Header's block-height badge now reads prevHeight via untrack() so
the effect doesn't form a dependency cycle with its own write.
Logmon now captures the share diff from ckpool's "Possible block
solve" line preceding the confirmation and attaches it to the
BlockEvent. Persisted as share_diff alongside height/hash/reward
and rendered as a new column in the dashboard block history.
Adds RPC.GetBlock(hash, verbosity=2) and a CoinbaseReward
helper that sums the first tx's outputs. IngestBlockEvents
now does getblockhash -> getblock -> sum(vout) so
BlockRecord.RewardBT carries the actual BTC paid out on
solve instead of always being zero. Both RPC calls share a
single 5s deadline and are best-effort — bitcoind being
down just leaves the reward at zero.
New internal/zmqmon package subscribes to bitcoind's hashblock
ZMQ topic and emits TipEvents. Uses github.com/go-zeromq/zmq4
(pure Go, builds with CGO_ENABLED=0). Exponential backoff on
connection failure so bitcoind restarts don't kill the
subscriber permanently; event channel drops rather than blocks
if the consumer is slow (signals are advisory, not logs).
Aggregator.Run now takes a <-chan TipEvent; when a tip arrives
it fires an immediate refresh() outside the normal ticker
cadence. With a 5s poll interval and 0.5-1s ZMQ latency from
bitcoind, dashboards now reflect new tips roughly 4x faster.
Endpoint comes from BITCOIN_ZMQ_BLOCK — empty disables ZMQ
entirely and the aggregator just runs on the ticker alone.
New internal/store package wraps modernc.org/sqlite (pure Go,
no CGO) with a BlockStore that exposes Open/Close/InsertBlock/
Recent. The aggregator now accepts an optional *store.BlockStore;
on Run() it loads up to maxBlockHistory rows from the store before
the first refresh, and each ingested block gets written to the
DB before being appended to the in-memory ring.
main.go opens the store at cfg.DBPath and logs a warning + falls
back to in-memory-only if the file can't be created — a broken
data volume shouldn't stop the pool from running.
InsertBlock uses INSERT OR IGNORE on the height primary key so
replayed log events after a restart are harmless.
The stratifier can accept a connection and close it without
writing a response during early startup — the refresh loop
then logs 'ckpool poolstats failed: read len: EOF' on every
tick until the stratifier is ready, which is noisy and looks
like a real fault.
- client.Send retries once after 200ms if the server closed
the connection before any bytes were read (io.EOF anywhere
in the wrapped chain).
- Aggregator tracks a ckFailStreak counter: first two
consecutive failures log at DEBUG, third and beyond escalate
to WARN. Successful refreshes reset the streak.
Adds real-time block detection via ckpool log tailing and a push
channel for the upcoming Svelte UI, all stdlib-only:
- logmon.Tailer follows ckpool.log with rotation/truncation survival
(inode + size tracking) and parses "Solved and confirmed block N"
into BlockEvent values.
- state.Aggregator grows a 256-entry block ring, an OnRefresh hook,
and IngestBlockEvents which best-effort enriches events with the
block hash via bitcoind getblockhash.
- httpapi.Hub implements RFC 6455 from scratch (SHA1 handshake,
unmasked text frames out, masked frames in, ping keepalive,
per-client write mutex, slow-client drop) so we don't pull in a
ws dependency before we can go mod tidy.
- New routes: GET /api/blocks and GET /api/ws. Snapshot pushes fire
on every poll tick and immediately on block-solve.
ZMQ hashblock subscription and SQLite persistence are deferred to
Phase 2b.5 once the s9pk packaging repo exists and we have a real
build environment for adding Go deps.
Go 1.22 module that polls CKPool's Unix socket control API, queries
Bitcoin Core over JSON-RPC, merges both into a thread-safe snapshot, and
serves it over REST. Layout:
api/
├── cmd/kamado-api/main.go signal-aware entrypoint
└── internal/
├── config/ env var loader with validation
├── ckpool/ socket client (4-byte LE length-prefixed wire
│ protocol verified against libckpool.c), typed
│ response models for poolstats/users/workers/
│ clients/uptime, + unit tests using a fake
│ unix socket server
├── bitcoind/ minimal JSON-RPC client, getblockchaininfo
│ and getnetworkhashps
├── state/ Aggregator that refreshes a merged Snapshot
│ on a ticker; readers get a copy under RWMutex
└── httpapi/ REST handlers on Go 1.22 ServeMux:
/api/health /api/pool /api/users
/api/workers /api/clients /api/snapshot
CKPool stores hashrate as "dsps" (diff shares per second); we convert
to H/s via the 2^32 constant used by GoBrrr-Pool and other clients.
Every stat CKPool exposes to its socket API is surfaced — useragent,
IP, per-client diff, per-worker best diff — closing the gap against
Bassin which only reads the 60-second stats files.
Dockerfile does a CGO_ENABLED=0 static build on golang:1.22-bookworm
with -trimpath -ldflags=-s -w. docker-compose now runs both ckpool and
kamado-api, sharing a named volume for /run/ckpool so the API can
dial the stratifier socket directly.
Deferred to Phase 2b (called out in README):
- Bitcoin Core ZMQ hashblock subscriber
- WebSocket push for real-time UI updates
- SQLite persistence (block history, best-share history)
- CKPool log tailer for "Solved and confirmed block" detection
Tests and build NOT run in this commit — Go isn't installed in the
dev environment. Run `make api-test` or `make api` to verify.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
satoshi