Phase 4: swarm discovery, peer protocol, drive transfer, swarm ContentTransport #2
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "phase4-discovery"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase 4 of the V1 roadmap: the peer network, end to end, on the transport seam.
Slice 1 (
fa54b6d) — swarm setup, DHT lifecycle, D3-gated announce.createSwarmwrapper with announce cap, connection wrapping, forceful destroy; D3 boundary: only explicitseed(hash)announces, lookups stay invisible.Slice 2 (
a84ec03) — record-query protocol over protomux.outflux/record/1channel on inbound+outbound connections; wire whitelist enforced at the serialization boundary; malformed queries refused;PeerQueryTimeoutErrorwhile the connection survives.Slice 3 (
b83cfe9) — blob fetch channel.replicateStore()attaches hypercore replication to every swarm connection;mirrorDriveBlobs()is the read-only clamped view of a peer's drive; the clamp/read discipline now lives in one shareddriveReader. Proven end to end: 256KB blob byte-for-byte over a real swarm connection (noise over UDX through the DHT), full/sparse/tail.Slice 4 (
b7a03f9) — swarm ContentTransport.swarmTransport()behind thesrc/core/transport.jsseam: available/size/fetch/stream/warm against peers, per-drive-key LRU mirror cache, and every wait bounded (lookup, per-peer query, per-chunk read). Failover honestly scoped: a cached mirror survives its serving connection dying (swarm re-dials the announced peer, fresh stream re-attaches the open cores) — proven by test; mid-chunk peer switching is deliberately out of V1.createSwarmgainstakeConnections().Tests: 366/366 tests, 1295/1295 asserts, lint clean. New:
test/runtime/swarmtransport.test.js(byte-for-byte swarm fetch, sparse ranged stream, honest absence, bounded failure on dead peer, connection-loss failover).This PR was created by an AI agent (OpenHands) on behalf of the user.
TODO-V1 Phase 4 boxes 1-2: "Hyperswarm setup, DHT bootstrap, connection lifecycle" and "announce on topic = hash(contentHash) -- only for opted-in seeded content". - src/core/discovery.js: pure topic derivation (BLAKE2b-256 over the hex content hash, injected crypto, same contract as src/core/hash.js). The topic is hashed rather than leaked: it is announced into a monitorable DHT, and handing that audience the library's raw content hashes would be a privacy bug nothing downstream can undo (spec ss9). - src/runtime/bare/swarm.js: Hyperswarm wrapper. D3 is enforced here so no caller can bypass it: seed(hash) is the ONLY server:true path, lookup() joins client-only and never announces, and a fetched hash stays invisible until someone explicitly seeds it. Announce cap (256 topics) bounds both the public advertisement surface and the number of live backoff loops. maxPeers defaults to 16 -- upstream's 64 is a desktop assumption. - test/helpers/dht.js: throwaway loopback DHT bootstrap for tests, with the bootstrapper registered to die after every swarm (brittle order: 100). Destroying it first leaves live discovery queries waiting on a dead node -- ~5s of retry timeouts per swarm, measured. - Two measured hyperswarm behaviours recorded in the wrapper: * destroy({ force: true }) skips clear(), leaving PeerDiscovery objects running -- the event loop never exits after every promise resolves. Our force path clears first, then closes hard (~2ms, clean exit). * teardown ECONNRESETs rethrow from streamx's destroy path as uncaught errors; the wrapper attaches no-op error handlers to every connection, server-side included, until the protocol layer replaces them. Tests: 11 new. Discovery over a real local DHT (seed -> lookup -> connection with the seeder's key), lookup-never-announces, unseed idempotence, the 256-topic cap, idempotent force destroy, and the pure topic contract. Suite: 348/348, lint clean. Co-Authored-By: openhands <openhands@all-hands.dev>'m outflux/record/1': { hash } -> { found, record?, driveKey? } on every swarm connection, inbound and outbound alike. - servePeers() answers from the injected catalog + blob store; records are re-filtered through the wire whitelist at the serialization boundary, so no path/mtime can reach a peer even from a polluted store, and driveKey is disclosed only for hashes whose bytes this node actually holds. - queryPeer() is one channel, one question, one answer, with a hard timeout (unref'd timer, always cleared) -- a silent peer costs the asker a bounded wait, never a hang, and never the connection itself. - malformed hashes are refused at the answer boundary and, on the asker side, before the wire. TODO-V1 Phase 4 'Peer handshake' box.The fetch side of D1, behind the seam the playback server already speaks: - swarmTransport({ swarm, corestore }): available/size/fetch/stream/warm against the peer network. A lookup per unknown hash, one query per candidate connection, a mirror per drive key cached LRU -- a seek within a track reuses the mirror instead of re-querying the swarm. - Every wait is bounded: lookupTimeoutMs for the DHT, queryTimeoutMs per peer question, readTimeoutMs per CHUNK of a stream. A vanished peer is a bounded disappointment, not a hang; a failed read drops its mirror so the next request re-establishes from the swarm. - Failover, honestly scoped: the mirror is a Hypercore fed by every replication stream the node's wiring attaches, so a read survives the serving connection dying when the swarm re-dials the announced peer -- proven by a test that kills the connection mid-flight. Mid-chunk peer switching is NOT in V1; the player re-requesting is the recovery path, and the header comment says so. - createSwarm gains takeConnections(): a snapshot for callers who arrive after the fact instead of racing their own onConnection against flush(). Tests: full fetch byte-for-byte over a real swarm; sparse ranged stream; honest absence when no peer answers; bounded failure when the only peer dies; cached mirror surviving a connection loss. No debug scripts were committed with this slice.