feat(polymarket)!: two connections per market, first one wins

Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
This commit is contained in:
pascal committed 2026-11-11 15:18:48 +00:00
1 parent 8114a51dc2
commit 02747b882f
24 files changed
+8984 -1214

No files matched your search

+26 -29
View File
@@ -40,31 +40,25 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange.
4. **Extension 1 — redundant websockets** — *not started*
Multiple connections per symbol (several connections in a lane, its book
thread choosing among them, the orchestrator coordinating); choose a
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
every source switch. Recovery from errors, with the lane budget scaling
itself down when connections still get 1013 closes (no rate is safe:
[notes.md](notes.md), so a budget set too high must correct itself).
**Async re-evaluation checkpoint:** before building this, reassess whether the
socket-management complexity (reconnect/timeout/select, redundancy) warrants
swapping the thread-only runtime for tokio (see ADR 0001).
The lane that holds markets busy together (the 5- and 15-minute crypto
markets: closed 16 times in 75 minutes, in bursts a minute's load
average doesn't show) is the case to beat.
Also: surviving a hard crash without losing the writer's in-memory chunk,
a move's Remove reaching the old lane before the new lane's first
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
over 75 minutes), and closing the subscription gap (the few ms of a
symbol's changes the server doesn't send between our unsubscribe and
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
switch.
4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
Every symbol on two connections chosen independently, its book taking each
change from whichever delivers it first (first one wins); connections
separated from book threads; a connection the server keeps closing with
1013 is split, so a budget set too high corrects itself. The async
checkpoint was held: threads stay
([ADR 0001](adr/0001-concurrency-model.md)).
Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
Left as ceilings: the writer's in-memory chunk on a hard crash, and the
subscription gap when a symbol opens.
*Ends:* redundancy demonstrably reduces data loss: fewer losses and
subscription gaps per million changes with two subscriptions than with one.
Result: losses a quarter lower in each pair of alternating runs; losses and
gaps together lower in one pair, even in the other (sprint 04's Outcome).
5. **Extension 2 — subscription monitoring** — *not started*
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
resubscribe symbols that go too quiet; periodic full-socket recycle
resubscribe symbols that go too quiet (with two connections, first one wins
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
*Ends:* monitoring + recycling running.
@@ -108,16 +102,19 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
- Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging.
- Counting a reconnect's differing books as losses rather than subscription
gaps when the symbol was resyncing or not yet verified since its opening
book (a reconnect marker through the frames channel; they are labelled
gaps).
- A `reconcile` that reads the collector's log, to tell subscription gaps from
losses in its own report.
- Polymarket's `new_market` websocket event as a faster discovery signal (it
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
needed; `market_resolved` was never seen).
- Closing lanes that drain.
- Closing connections that drain.
- Verifying a symbol again from the trade event's price after a trade, with
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
- Telling a real loss from a resync book merely ahead of our stream, so the
loss count measures data missed (most of sprint 04's "losses" follow
trades, as often with one connection as with two).
- Closing the opening subscription gap with the second connection's opening
book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until
its copies are counted (sprint 04 reports an upper bound).
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.