Archived
feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
This commit is contained in:
24 files changed
+8984
-1214
No files matched your search
+26
-29
@@ -40,31 +40,25 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
4. **Extension 1 — redundant websockets** — *not started*
|
||||
Multiple connections per symbol (several connections in a lane, its book
|
||||
thread choosing among them, the orchestrator coordinating); choose a
|
||||
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
|
||||
every source switch. Recovery from errors, with the lane budget scaling
|
||||
itself down when connections still get 1013 closes (no rate is safe:
|
||||
[notes.md](notes.md), so a budget set too high must correct itself).
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
The lane that holds markets busy together (the 5- and 15-minute crypto
|
||||
markets: closed 16 times in 75 minutes, in bursts a minute's load
|
||||
average doesn't show) is the case to beat.
|
||||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||||
a move's Remove reaching the old lane before the new lane's first
|
||||
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
|
||||
over 75 minutes), and closing the subscription gap (the few ms of a
|
||||
symbol's changes the server doesn't send between our unsubscribe and
|
||||
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
|
||||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||||
switch.
|
||||
4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
|
||||
Every symbol on two connections chosen independently, its book taking each
|
||||
change from whichever delivers it first (first one wins); connections
|
||||
separated from book threads; a connection the server keeps closing with
|
||||
1013 is split, so a budget set too high corrects itself. The async
|
||||
checkpoint was held: threads stay
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
|
||||
Left as ceilings: the writer's in-memory chunk on a hard crash, and the
|
||||
subscription gap when a symbol opens.
|
||||
*Ends:* redundancy demonstrably reduces data loss: fewer losses and
|
||||
subscription gaps per million changes with two subscriptions than with one.
|
||||
Result: losses a quarter lower in each pair of alternating runs; losses and
|
||||
gaps together lower in one pair, even in the other (sprint 04's Outcome).
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||||
resubscribe symbols that go too quiet; periodic full-socket recycle
|
||||
resubscribe symbols that go too quiet (with two connections, first one wins
|
||||
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
|
||||
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
|
||||
*Ends:* monitoring + recycling running.
|
||||
|
||||
@@ -108,16 +102,19 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
- Counting a reconnect's differing books as losses rather than subscription
|
||||
gaps when the symbol was resyncing or not yet verified since its opening
|
||||
book (a reconnect marker through the frames channel; they are labelled
|
||||
gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell subscription gaps from
|
||||
losses in its own report.
|
||||
- Polymarket's `new_market` websocket event as a faster discovery signal (it
|
||||
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
|
||||
needed; `market_resolved` was never seen).
|
||||
- Closing lanes that drain.
|
||||
- Closing connections that drain.
|
||||
- Verifying a symbol again from the trade event's price after a trade, with
|
||||
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
|
||||
- Telling a real loss from a resync book merely ahead of our stream, so the
|
||||
loss count measures data missed (most of sprint 04's "losses" follow
|
||||
trades, as often with one connection as with two).
|
||||
- Closing the opening subscription gap with the second connection's opening
|
||||
book, when the first one's chain breaks before any change verified.
|
||||
- An exact rescued count: a change remembered past the window of 128 until
|
||||
its copies are counted (sprint 04 reports an upper bound).
|
||||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||||
minute: ~130 MB a minute at full coverage).
|
||||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|
||||
Reference in new issue
Block a user