feat(polymarket)!: two connections per market, first one wins

Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
This commit is contained in:
pascal committed 2026-11-11 15:18:48 +00:00
1 parent 8114a51dc2
commit 02747b882f
24 files changed
+8984 -1214

No files matched your search

+9 -5
View File
@@ -1,8 +1,8 @@
# Test fixtures
A recording of Polymarket's market channel, replayed by the tests in
`src/polymarket/feed.rs`, and Gamma pages read by `src/polymarket/gamma.rs`
(no network in tests).
Two recordings of Polymarket's market channel, replayed by the tests in
`src/polymarket/feed.rs` (the first also by `socket.rs`'s routing test), and
Gamma pages read by `src/polymarket/gamma.rs` (no network in tests).
- `gamma-markets.json`: Gamma's top 20 markets by 24 h volume on 2026-11-04,
trimmed to the fields the collector reads. `conditionId`, `active`, `closed`
@@ -15,5 +15,9 @@ A recording of Polymarket's market channel, replayed by the tests in
minute boundaries) of the market channel for those markets' 40 tokens, one
line per frame: `{"received_at_us": <receive time, µs>, "text": <frame>}`.
Opening books, 3,850 changes, 28 mid-stream books, 14 trades, 8 `PONG`s.
- `record.py`: the recorder, if the fixtures need redoing:
`uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90`.
- `polymarket-frames-two.jsonl`: 95 s (2026-11-11, 12:19:58 to 12:21:33 UTC)
of the same tokens on two connections at once, in receive order, each line
with its `connection` (0 or 1). 20 of the 40 tokens still had an order book.
- `record.py`: the recorder, if the fixtures need redoing, on CONNECTIONS
connections at once (default 1):
`uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90 [CONNECTIONS]`.
File diff suppressed because it is too large. Load diff
+9 -4
View File
@@ -1,10 +1,13 @@
# Records the market channel for the markets in gamma-markets.json: one JSON
# line per frame, {"received_at_us": <micros>, "text": <frame text>}.
# line per frame, in receive order,
# {"received_at_us": <micros>, "connection": <number>, "text": <frame text>}.
# CONNECTIONS (default 1) are opened at once, all on the same tokens.
# uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90 [CONNECTIONS]
import asyncio, json, sys, time
import websockets
markets = json.load(open(sys.argv[1]))
tokens = [t for m in markets if m["enableOrderBook"] for t in json.loads(m["clobTokenIds"])]
async def main(out, seconds):
async def record(connection, out, seconds):
async with websockets.connect("wss://ws-subscriptions-clob.polymarket.com/ws/market", max_size=None) as ws:
await ws.send(json.dumps({"type": "market", "assets_ids": tokens}))
loop = asyncio.get_event_loop(); end = loop.time() + seconds; last_ping = loop.time()
@@ -14,6 +17,8 @@ async def main(out, seconds):
try: text = await asyncio.wait_for(ws.recv(), 1)
except asyncio.TimeoutError: continue
received_at_us = time.time_ns() // 1000
out.write(json.dumps({"received_at_us": received_at_us, "text": text}) + "\n")
out.write(json.dumps({"received_at_us": received_at_us, "connection": connection, "text": text}) + "\n")
async def main(out, seconds, connections):
await asyncio.gather(*(record(connection, out, seconds) for connection in range(connections)))
with open(sys.argv[2], "w") as out:
asyncio.run(main(out, float(sys.argv[3])))
asyncio.run(main(out, float(sys.argv[3]), int(sys.argv[4]) if len(sys.argv) > 4 else 1))