Archived
feat(polymarket): track every live market over load-balanced lanes
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
18 files changed
+2826
-320
No files matched your search
+33
-22
@@ -30,33 +30,35 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
|
||||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||||
|
||||
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
|
||||
Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or
|
||||
Polymarket's `new_market` / `market_resolved` events — using subscribe /
|
||||
unsubscribe on open connections; load-balance all live regular-market symbols
|
||||
across N sockets.
|
||||
**Opening questions (from M2):** what "live" means, and so how many symbols:
|
||||
Gamma lists at least 100,000 markets `active` and not `closed` (2026-11-04,
|
||||
count capped), at least 13,000 of them traded in the last 24 h, against the
|
||||
40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each)
|
||||
would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and
|
||||
memory (~10 KB a book), sockets per connection, resyncs per minute and the
|
||||
book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s
|
||||
of load) all need measuring. Gamma pages past 10,000 markets only through
|
||||
`/markets/keyset` with `after_cursor`.
|
||||
3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)*
|
||||
Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a
|
||||
connection and a book thread of its own; an orchestrator keeps the tracked
|
||||
set equal to Gamma's live set (a scan at startup and every hour, new and
|
||||
closed markets every minute) through subscribe / unsubscribe on open
|
||||
connections.
|
||||
Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md).
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
4. **Extension 1 — redundant websockets** — *not started*
|
||||
Multiple connections per symbol; choose a trusted source and switch on
|
||||
loss/quiet/delay; emit a fresh snapshot on every source switch.
|
||||
Multiple connections per symbol (several connections in a lane, its book
|
||||
thread choosing among them, the orchestrator coordinating); choose a
|
||||
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
|
||||
every source switch. Recovery from errors, with the lane budget scaling
|
||||
itself down when connections still get 1013 closes (no rate is safe:
|
||||
[notes.md](notes.md), so a budget set too high must correct itself).
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
The lane that holds markets busy together (the 5- and 15-minute crypto
|
||||
markets: closed 16 times in 75 minutes, in bursts a minute's load
|
||||
average doesn't show) is the case to beat.
|
||||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||||
and closing the resync gap (the few ms of a symbol's changes the server
|
||||
doesn't send between our unsubscribe and subscribe), both accepted as
|
||||
ceilings in M2.
|
||||
a move's Remove reaching the old lane before the new lane's first
|
||||
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
|
||||
over 75 minutes), and closing the subscription gap (the few ms of a
|
||||
symbol's changes the server doesn't send between our unsubscribe and
|
||||
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
|
||||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||||
switch.
|
||||
|
||||
@@ -106,7 +108,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
- Counting a reconnect's differing books as losses rather than resync gaps
|
||||
(a reconnect marker through the frames channel; M2 labels them gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell resync gaps from
|
||||
- Counting a reconnect's differing books as losses rather than subscription
|
||||
gaps when the symbol was resyncing or not yet verified since its opening
|
||||
book (a reconnect marker through the frames channel; they are labelled
|
||||
gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell subscription gaps from
|
||||
losses in its own report.
|
||||
- Polymarket's `new_market` websocket event as a faster discovery signal (it
|
||||
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
|
||||
needed; `market_resolved` was never seen).
|
||||
- Closing lanes that drain.
|
||||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||||
minute: ~130 MB a minute at full coverage).
|
||||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|
||||
Reference in new issue
Block a user