Tighten fallback latency + alert UI on degraded states

Submit-first ordering. Patch 0004 now calls generator_submitblock
BEFORE writing the pending-block hex to disk. The happy path adds zero
disk I/O — we only dump when the primary returned false. The same
patch bounds generator_submitblock's "no live current_si" spin to
~3s instead of the original infinite loop, so a permanently-down
primary doesn't pin the stratifier; the bounded spin lets the caller
return false and lets local_block_submit dump for kamado-api to take
over.

Default grace lowered from 30s to 3s. With ckpool's bounded spin and
sub-second sweep cadence, the fallback now reacts within ~4s of a
failed primary submit — fast enough that the work is still relevant
for the current chain tip. The submitter's sweep poll dropped to 1s
to match.

UI HealthBanners. New top-of-page strip surfaces:
  * Fallback used (red banner, 24h after most recent event):
    "primary bitcoind didn't accept; backup X took over Y ago"
  * Submit gap (orange banner, only when no recent fallback):
    "N blocks attempted but unconfirmed — configure backups"
  * ZMQ stale (orange banner): no hashblock frame in 30+ minutes
Operators see degraded-but-not-fatal states without checking logs.

Startup readiness gate. main now waits up to 8s on agg.Ready() before
starting the HTTP server so the very first /api/snapshot doesn't show
all-zero state during the aggregator's first refresh. Capped so a
permanently-down bitcoind can't block startup; /healthz is honest
about the degraded state once we do start serving.

ckpool/patches/0004-dump-pending-block-for-fallback.patch

@@ -1,16 +1,48 @@
+diff --git a/src/generator.c b/src/generator.c
+index 22e2e08..438b199 100644
+--- a/src/generator.c
++++ b/src/generator.c
+@@ -351,11 +351,25 @@ bool generator_submitblock(ckpool_t *ckp, const char *buf)
+ 	server_instance_t *si;
+ 	bool warn = false;
+ 	connsock_t *cs;
++	/* Bound the wait for current_si so a permanently-down primary
++	 * bitcoind does not pin the caller forever. After this many 10ms
++	 * sleeps (~3s), return false so the caller (in our fork:
++	 * local_block_submit) can hand off to kamado-api's fallback
++	 * broadcaster instead of blocking the stratifier on a dead RPC.
++	 * Original upstream behavior was to spin indefinitely; we trade
++	 * that for predictable latency. */
++	const int max_no_si_iters = 300;
++	int no_si_iters = 0;
+ 
+ 	while (unlikely(!(si = gdata->current_si))) {
+ 		if (!warn)
+-			LOGWARNING("No live current server in generator_blocksubmit! Resubmitting indefinitely!");
++			LOGWARNING("No live current server in generator_blocksubmit! Waiting up to ~3s before giving up so fallback can take over...");
+ 		warn = true;
++		if (++no_si_iters > max_no_si_iters) {
++			LOGWARNING("generator_submitblock: no live primary server after %d ms, returning false",
++				   no_si_iters * 10);
++			return false;
++		}
+ 		cksleep_ms(10);
+ 	}
+ 	cs = &si->cs;
 diff --git a/src/stratifier.c b/src/stratifier.c
-index 52da790..815eb01 100644
+index 52da790..7caf179 100644
 --- a/src/stratifier.c
 +++ b/src/stratifier.c
-@@ -2069,16 +2069,64 @@ process_block(const workbase_t *wb, const char *coinbase, const int cblen,
+@@ -2069,16 +2069,75 @@ process_block(const workbase_t *wb, const char *coinbase, const int cblen,
  	return gbt_block;
  }
  
+-/* Submit block data locally, absorbing and freeing gbt_block */
 +/* Write the raw block hex to a sidecar file under <logdir>/pending-blocks/
-+ * before submitting, so an external watcher (kamado-api) can re-broadcast
-+ * via fallback RPC nodes if our primary bitcoind doesn't accept it. The
-+ * file is unlinked once generator_submitblock returns success. Best-
-+ * effort: any error here is logged at INFO and never blocks the submit. */
++ * so the external watcher (kamado-api) can re-broadcast via fallback RPC
++ * nodes. Best-effort: any error here is logged at INFO and never blocks
++ * the caller. Called only after the primary submit fails so the happy
++ * path stays disk-free. */
 +static void kamado_dump_pending_block(ckpool_t *ckp, const char *gbt_block,
 +				      const char *rhash, int height,
 +				      char *out_path, size_t out_path_len)
@@ -44,11 +76,17 @@ index 52da790..815eb01 100644
 +		out_path[0] = '\0';
 +		return;
 +	}
-+	LOGNOTICE("kamado: dumped pending block height %d to %s (%zu bytes)",
++	LOGNOTICE("kamado: dumped pending block height %d to %s (%zu bytes) for fallback broadcast",
 +		  height, out_path, blen);
 +}
 +
- /* Submit block data locally, absorbing and freeing gbt_block */
++/* Submit block data locally, absorbing and freeing gbt_block.
++ *
++ * Order matters: we always try ckp's primary bitcoind FIRST so the
++ * happy path adds zero latency or disk I/O. Only when the primary
++ * rejects (or generator_submitblock returns false for any other
++ * reason) do we dump the raw hex to <logdir>/pending-blocks/ so the
++ * kamado-api watcher can re-broadcast via fallback RPC nodes. */
  static bool local_block_submit(ckpool_t *ckp, char *gbt_block, const uchar *flip32, int height)
  {
 -	bool ret = generator_submitblock(ckp, gbt_block);
@@ -60,19 +98,26 @@ index 52da790..815eb01 100644
 -	free(gbt_block);
  	swap_256(swap256, flip32);
  	__bin2hex(rhash, swap256, 32);
-+	kamado_dump_pending_block(ckp, gbt_block, rhash, height,
-+				  pending_path, sizeof(pending_path));
 +
++	/* Primary submit first — this is the latency-critical path. */
 +	ret = generator_submitblock(ckp, gbt_block);
 +
++	/* Only touch disk if the primary didn't accept. */
++	if (!ret) {
++		kamado_dump_pending_block(ckp, gbt_block, rhash, height,
++					  pending_path, sizeof(pending_path));
++	}
++
 +	free(gbt_block);
  	generator_preciousblock(ckp, rhash);
  
  	/* Check failures that may be inconclusive but were submitted via other
-@@ -2099,6 +2147,8 @@ static bool local_block_submit(ckpool_t *ckp, char *gbt_block, const uchar *flip
+@@ -2099,6 +2158,10 @@ static bool local_block_submit(ckpool_t *ckp, char *gbt_block, const uchar *flip
  				   height, ret ? "ACCEPTED" : "REJECTED");
  		}
  	}
++	/* Block ended up on chain (either initial submit or precious-block
++	 * recovery): the dump file is no longer needed. */
 +	if (ret && pending_path[0])
 +		unlink(pending_path);
  	return ret;