Archived
feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
This commit is contained in:
24 files changed
+8984
-1214
No files matched your search
Vendored
+9
-4
@@ -1,10 +1,13 @@
|
||||
# Records the market channel for the markets in gamma-markets.json: one JSON
|
||||
# line per frame, {"received_at_us": <micros>, "text": <frame text>}.
|
||||
# line per frame, in receive order,
|
||||
# {"received_at_us": <micros>, "connection": <number>, "text": <frame text>}.
|
||||
# CONNECTIONS (default 1) are opened at once, all on the same tokens.
|
||||
# uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90 [CONNECTIONS]
|
||||
import asyncio, json, sys, time
|
||||
import websockets
|
||||
markets = json.load(open(sys.argv[1]))
|
||||
tokens = [t for m in markets if m["enableOrderBook"] for t in json.loads(m["clobTokenIds"])]
|
||||
async def main(out, seconds):
|
||||
async def record(connection, out, seconds):
|
||||
async with websockets.connect("wss://ws-subscriptions-clob.polymarket.com/ws/market", max_size=None) as ws:
|
||||
await ws.send(json.dumps({"type": "market", "assets_ids": tokens}))
|
||||
loop = asyncio.get_event_loop(); end = loop.time() + seconds; last_ping = loop.time()
|
||||
@@ -14,6 +17,8 @@ async def main(out, seconds):
|
||||
try: text = await asyncio.wait_for(ws.recv(), 1)
|
||||
except asyncio.TimeoutError: continue
|
||||
received_at_us = time.time_ns() // 1000
|
||||
out.write(json.dumps({"received_at_us": received_at_us, "text": text}) + "\n")
|
||||
out.write(json.dumps({"received_at_us": received_at_us, "connection": connection, "text": text}) + "\n")
|
||||
async def main(out, seconds, connections):
|
||||
await asyncio.gather(*(record(connection, out, seconds) for connection in range(connections)))
|
||||
with open(sys.argv[2], "w") as out:
|
||||
asyncio.run(main(out, float(sys.argv[3])))
|
||||
asyncio.run(main(out, float(sys.argv[3]), int(sys.argv[4]) if len(sys.argv) > 4 else 1))
|
||||
Reference in new issue
Block a user