An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
2.7 KiB
Order-book collector
The collector's language: what it tracks on each exchange and what it writes for the backtests that read its output.
Exchange side
Market: A tradable question on an exchange. On Polymarket it has two outcomes (Yes, No), each traded as its own symbol, and their books mirror each other. Avoid: Event, contract
Symbol: The unit that has one order book: on Polymarket a token (one outcome of a market), named by its token id. Avoid: Asset, instrument, ticker (for Polymarket)
Live: Said of a market (and its symbols) the exchange lists as open for trading on its order book. A market stops being live when it closes or stops accepting orders, not at its end date. Avoid: Active (Gamma's flag, only one input of being live)
Book hash: The exchange's fingerprint of a symbol's full book state, sent with every full book and every change.
Collector side
Book: The collector's in-memory order book for one symbol: its bid and ask price levels, each with its absolute quantity. Avoid: Order book state, depth
Lane: A set of symbols whose books one book thread owns, fed by its own connections: one path from the wire to the writer. Every live symbol is in exactly one lane. Avoid: Shard (a technique, not this thing), group, partition
Load: How much a market weighs on its connection: the bytes per second of its frames. A lane's load is the sum of its markets'. Avoid: Activity, volume (the exchange's traded amount, unrelated)
Connection: One websocket to the exchange, carrying the symbols of one lane. Avoid: Socket (for the concept; fine for the code), stream
Receive time: When a message came off the socket, by the local clock. Avoid: Arrival time, local timestamp
Loss: Any message for a symbol that the collector did not apply, which leaves its book silently wrong. Avoid: Gap (implies sequence numbers)
Resync: Asking the exchange for a fresh full book of a symbol and replacing ours with it, whenever our book can't be verified. Our book was wrong only if the fresh one differs: a loss, or a subscription gap. Avoid: Re-snapshot, rebuild, refresh
Subscription gap: The changes our own subscribe keeps from us, whether it opens, resyncs or moves a symbol: the exchange holds them in the book it sends but never sends them as changes. Avoid: Resync gap (the older, narrower name); calling it a loss (the state stays right; only those updates are missing)
Output
Snapshot: The output message holding one symbol's whole book at a point in time. Avoid: using it for the action of fetching a fresh book (that is a resync)
Delta:
The output message changing one price level of one symbol's book to a new
absolute quantity (PriceLevelUpdate in the schema).
Avoid: Increment, diff