yash/OutFlux
0
0
Fork 0
Code Issues Pull requests Projects Releases 11 Packages Wiki Activity Actions

Phase 4: stall-driven failover and node-level swarm wiring #3

Merged
yash merged 2 commits from phase4-discovery into main 2026-09-01 07:33:37 +00:00
Conversation 0 Commits 2 Files changed 7 +460 -51
yash commented 2026-09-01 07:33:27 +00:00
Owner
Copy link

The last two Phase 4 boxes, proven over a real swarm: multi-peer failover per D1 and graceful degradation when a peer vanishes mid-stream.

Server-side failover (7af3414). The playback server subscribes to the session reducer's own stall signal, exactly as the TODO-V1 Phase 4 note prescribed — no second, differently-tuned stall detector in the transport. One idle fire is pure honesty (the reducer marks stalled, metrics record it); consecutive fires with no bytes in between (failoverAfterStalls, default 2) make the server abandon the current source and re-request the remaining range from the transport, bounded by maxFailovers (default 3). Every way a source can stop short of the promised span converges on that same bounded decision — including a new catch: a transport that ends without error but short of Content-Length can no longer be forwarded as a complete 200. Failover also covers a source that dies before its first byte (the strike count is deliberately independent of the reducer's state, because time-to-first-byte is exactly where a dead peer hurts most).

Node wiring + transport hardening (983ab50). attachSwarm (new src/runtime/bare/swarmnode.js) turns an openNode storage node into a peer: servePeers answers from the real catalog, replicateStore serves bytes, swarmTransport fetches what it lacks — one swarm, one Corestore, and node.close() wrapped so the swarm always dies before storage. Critical fix found by the new tests: swarmTransport.stream's per-chunk deadline existed but its cleanup awaited the underlying iterator's return() unboundedly — a hypercore read against a vanished source may never settle (measured), so the escape hatch hung on its own latch. Cleanup is now bounded, and a failed chunk drops the mirror so a restart re-resolves against the swarm instead of reusing the dead one.

Failover acceptance, over a real swarm (test/runtime/failover.test.js, loopback DHT + UDX):

  1. attachSwarm wiring — a node with only a catalog record pulls bytes byte-for-byte from a seeder node.
  2. Kill the only seeder mid-response → client sees a failure/short body, bounded at ~2.4s (never a hang), failover attempts logged, session records failed.
  3. Kill one of two seeders mid-response → the body still completes byte-for-byte.

Tests: 1312/1312 asserts, lint clean. TODO-V1's Phase 4 boxes are all ticked, with what is and is NOT proven stated plainly (DHT re-discovery of a brand-new peer after the only seeder dies is hyperswarm refresh behavior, not something these tests pin).

This PR was created by an AI agent (OpenHands) on behalf of the user.

The last two Phase 4 boxes, proven over a real swarm: multi-peer failover per D1 and graceful degradation when a peer vanishes mid-stream. **Server-side failover (`7af3414`).** The playback server subscribes to the session reducer's own stall signal, exactly as the TODO-V1 Phase 4 note prescribed — no second, differently-tuned stall detector in the transport. One idle fire is pure honesty (the reducer marks `stalled`, metrics record it); consecutive fires with no bytes in between (`failoverAfterStalls`, default 2) make the server abandon the current source and re-request the remaining range from the transport, bounded by `maxFailovers` (default 3). Every way a source can stop short of the promised span converges on that same bounded decision — including a new catch: a transport that ends without error but short of Content-Length can no longer be forwarded as a complete 200. Failover also covers a source that dies before its first byte (the strike count is deliberately independent of the reducer's state, because time-to-first-byte is exactly where a dead peer hurts most). **Node wiring + transport hardening (`983ab50`).** `attachSwarm` (new `src/runtime/bare/swarmnode.js`) turns an openNode storage node into a peer: servePeers answers from the real catalog, replicateStore serves bytes, swarmTransport fetches what it lacks — one swarm, one Corestore, and `node.close()` wrapped so the swarm always dies before storage. Critical fix found by the new tests: `swarmTransport.stream`'s per-chunk deadline existed but its cleanup awaited the underlying iterator's `return()` unboundedly — a hypercore read against a vanished source may never settle (measured), so the escape hatch hung on its own latch. Cleanup is now bounded, and a failed chunk drops the mirror so a restart re-resolves against the swarm instead of reusing the dead one. **Failover acceptance, over a real swarm** (`test/runtime/failover.test.js`, loopback DHT + UDX): 1. attachSwarm wiring — a node with only a catalog record pulls bytes byte-for-byte from a seeder node. 2. Kill the only seeder mid-response → client sees a failure/short body, bounded at ~2.4s (never a hang), failover attempts logged, session records `failed`. 3. Kill one of two seeders mid-response → the body still completes byte-for-byte. **Tests:** 1312/1312 asserts, lint clean. TODO-V1's Phase 4 boxes are all ticked, with what is and is NOT proven stated plainly (DHT re-discovery of a brand-new peer after the only seeder dies is hyperswarm refresh behavior, not something these tests pin). --- _This PR was created by an AI agent (OpenHands) on behalf of the user._
The session reducer's stalled transition was the point of building the
state machine now -- this is the subscriber TODO-V1's Phase 4 note called
for. Consecutive idle fires with no byte in between (failoverAfterStalls,
default 2) mean the source is not merely slow; the server abandons it
(iterator.return()) and re-requests the REMAINING range from the
transport. One idle fire stays pure honesty: the reducer marks stalled
and the metrics record it, without disturbing a healthy-but-slow source.

Every way a source can stop short of the promised span converges on the
same bounded decision -- restart from the last offset delivered, at most
maxFailovers (default 3) times, then fail the response honestly. That
includes the new one this closes: a transport that ends without error
but short of Content-Length can no longer be forwarded as a complete
200. A transport error mid-body is now a reason to try again from where
the bytes got to, not to tear down the response -- the mid-body test now
pins a PERSISTENT failure exhausting the budget before the socket tears.

Failover covers a source that dies before its first byte too: the strike
count is deliberately independent of the reducer's state, because the
reducer rightly refuses to call pre-first-byte silence stalled --
time-to-first-byte is exactly where a dead peer hurts most.
The composition step the separate slices were built for: a node from
openNode, given a catalog, becomes a peer -- servePeers answers library
queries from its real catalog, replicateStore serves its bytes, and
swarmTransport fetches what it lacks, all on one swarm and one Corestore.
Nothing here invents policy; the wiring is the phase.

- openNode now exposes its Corestore (the same object, not a second
  handle): replication and the transport attach at store level, over
  cores that exist now and cores that open later.
- attachSwarm wraps node.close() so the swarm dies before the storage
  under it, keeping openNode's close contract -- one call, ordered,
  idempotent. No caller can tear down the Corestore under live sockets.
- swarmTransport.stream: the per-chunk deadline existed, but the
  cleanup awaited the underlying iterator's return() unboundedly -- and
  a hypercore read against a vanished source may never settle
  (measured), so the escape hatch hung on its own latch. Cleanup is now
  bounded like everything else, and a failed chunk drops the mirror so
  the next establish re-resolves against the swarm instead of silently
  reusing the dead one. Together with the server's failover restart
  this is the bounded-degradation chain end to end.

Failover acceptance, over a real swarm (test/runtime/failover.test.js):
kill the only seeder mid-response and the request fails honest and
bounded (~2.4s), session records failed; kill one of two and the body
still completes byte for byte. TODO-V1's Phase 4 boxes are ticked with
what is and is not proven stated plainly.
yash merged commit a2c342835f into main 2026-09-01 07:33:37 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
yash/OutFlux!3
No description provided.