The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
4.5 KiB
Order-book collector
The collector's language: what it tracks on each exchange and what it writes for the backtests that read its output.
Exchange side
Market: A tradable question on an exchange. On Polymarket it has two outcomes (Yes, No), each traded as its own symbol, and their books mirror each other. Avoid: Event, contract
Symbol: The unit that has one order book: on Polymarket a token (one outcome of a market), named by its token id. Avoid: Asset, instrument, ticker (for Polymarket)
Live: Said of a market (and its symbols) the exchange lists as open for trading on its order book. A market stops being live when it closes or stops accepting orders, not at its end date. Avoid: Active (Gamma's flag, only one input of being live)
Book hash: The exchange's fingerprint of a symbol's full book state, sent with every full book and every change.
Collector side
Book: The collector's in-memory order book for one symbol: its bid and ask price levels, each with its absolute quantity. Avoid: Order book state, depth
Connection: One websocket to the exchange. Every live symbol is subscribed on two connections, chosen independently of each other. Avoid: Socket (for the concept; fine for the code), stream, lane (the retired pairing of one connection with one book thread)
Load: How much a market weighs on a connection: the bytes per second of its frames there. A connection's load is the sum of its markets'. Avoid: Activity, volume (the exchange's traded amount, unrelated)
First one wins: The rule for a symbol's two connections: a change is taken from whichever delivers it first, with that receive time, and the other's copy is dropped. Avoid: Source, primary, standby, failover (no connection has a rank)
Duplicate: A change another connection already delivered.
Verified: Said of a symbol whose book gives the exchange's book hash after every change. It stops being verified when a change fits on no connection in time.
Held: Said of a change kept unwritten because it doesn't fit the book yet: a change before it is still to come, from this connection or the other. Avoid: Buffered, pending, queued
Rescued: Said of a change one connection missed and the other delivered: the output has it, where a single connection would have had a loss.
Follow: To apply one connection's changes unchecked, while a symbol can't be verified and until a fresh book says where it stands.
Split: Moving half the markets off a connection the exchange keeps closing.
Quiet: Said of a symbol's subscription on a connection that is up, while the exchange has stopped sending that symbol's changes there. Nothing tells us: it has to be looked for. Avoid: Stale, dead (a connection can be fine while one of its subscriptions is quiet)
Resubscribe: To unsubscribe a symbol on one connection and subscribe it again there, which makes the exchange start its subscription afresh. A resync is a resubscribe whose book we asked for.
Silent: Said of a connection that is open but delivers nothing at all, not even the answer to its keepalive.
Recycle: To close a working connection on purpose and reconnect it with the same symbols, to shed what has gone wrong on the exchange's side of it. Avoid: Restart, rotate
Receive time: When a message came off the socket, by the local clock. Avoid: Arrival time, local timestamp
Loss: A change for a symbol that no connection delivered in time, which leaves its book silently wrong. A change only one connection missed is rescued, not lost. Avoid: Gap (implies sequence numbers)
Resync: Asking the exchange for a fresh full book of a symbol and replacing ours with it, whenever our book can't be verified. Our book was wrong only if the fresh one differs: a loss, or a subscription gap. Avoid: Re-snapshot, rebuild, refresh
Subscription gap: The changes our own subscribe keeps from us, whether it opens or resyncs a symbol: the exchange holds them in the book it sends but never sends them as changes. A connection that stays subscribed meanwhile has no such gap. Avoid: Resync gap (the older, narrower name); calling it a loss (the state stays right; only those updates are missing)
Output
Snapshot: The output message holding one symbol's whole book at a point in time. Avoid: using it for the action of fetching a fresh book (that is a resync)
Delta:
The output message changing one price level of one symbol's book to a new
absolute quantity (PriceLevelUpdate in the schema).
Avoid: Increment, diff