This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
pascal cf6c16c9cf feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone.

- scripts/watch.sh runs the collector with a live view of its log, stops
  cleanly on Ctrl-C, then checks the output (reconcile) and shows one
  symbol's book as text (show, a new binary).
- The README for the solution, with the final runs: Polymarket through an
  outage it recovered from on its own, Kalshi's demo with two slow
  connections recycled.
- From a review of the whole codebase: Kalshi takes RSA keys too
  (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's
  next turn, which it nearly never got before; a market tracked again no
  longer fails reconcile; Book::add checks for overflow; usage and --help
  name the key variables; the first scan says how far it got.
- Tests for every gap the reviews found, an offline run of the whole
  orchestrator among them; ShellCheck in CI.
2026-11-12 22:32:39 +01:00

161 lines
5.8 KiB
Markdown

# Order-book collector
The collector's language: what it tracks on each exchange and what it writes
for the backtests that read its output.
## Exchange side
**Market**:
A tradable question on an exchange. On Polymarket it has two outcomes (Yes,
No), each traded as its own symbol, and their books mirror each other. On
Kalshi it has one book, holding bids for Yes and bids for No.
_Avoid_: Event, contract
**Symbol**:
The unit that has one order book: on Polymarket a token (one outcome of a
market), named by its token id; on Kalshi the market itself, named by its
ticker, a bid for No at a price being an ask at one minus that price.
_Avoid_: Asset, instrument, ticker (for Polymarket)
**Live**:
Said of a market (and its symbols) the exchange lists as open for trading on
its order book. A market stops being live when it closes or stops accepting
orders, not at its end date. A Kalshi market about to open is tracked ahead
of being live, its book empty.
_Avoid_: Active (Gamma's flag, only one input of being live)
**Book hash**:
Polymarket's fingerprint of a symbol's full book state, sent with every
full book and every change.
**Sequence number**:
Kalshi's count of the messages it sent on one connection, all its markets
together: one skipped means the connection lost a message, for a market it
doesn't name.
**Clock**:
Kalshi's counter for one market, the same on every connection, carried by
every book and change. Several changes can share one clock, always in the
same order.
## Collector side
**Book**:
The collector's in-memory order book for one symbol: its bid and ask price
levels, each with its absolute quantity.
_Avoid_: Order book state, depth
**Connection**:
One websocket to the exchange. Every live symbol is subscribed on two
connections, chosen independently of each other.
_Avoid_: Socket (for the concept; fine for the code), stream (that is one
connection's changes of a symbol), lane (the retired pairing of one
connection with one book thread)
**Stream**:
One connection's changes of one symbol, in the order it delivers them. The
two streams of a symbol carry the same changes, each at its own pace.
_Avoid_: Feed (the book thread's code), using it for the connection
**Load**:
How much a market weighs on a connection: the bytes per second of its frames
there. A connection's load is the sum of its markets'.
_Avoid_: Activity, volume (the exchange's traded amount, unrelated)
**First one wins**:
The rule for a symbol's two connections: a change is taken from whichever
delivers it first, with that receive time, and the other's copy is dropped.
_Avoid_: Source, primary, standby, failover (no connection has a rank)
**Duplicate**:
A change another connection already delivered.
**Lateness**:
How long after a change's first copy a connection delivered it; 0 for the
connection that delivered first. Needs no clock.
_Avoid_: Lag, delay
**Lag**:
A change's receive time minus the exchange's timestamp for it. It carries
the local clock's offset.
_Avoid_: Latency (the general idea, not a number)
**Speed**:
A connection's median lateness over a minute.
**Verified**:
Said of a symbol whose book gives the exchange's book hash after every
change. It stops being verified when a change fits on no connection in time.
**Held**:
Said of a change kept unwritten because it doesn't fit the book yet: a change
before it is still to come, from this connection or the other.
_Avoid_: Buffered, pending, queued
**Rescued**:
Said of a change one connection missed and the other delivered: the output
has it, where a single connection would have had a loss.
**Follow**:
To apply one connection's changes unchecked, while a symbol can't be
verified and until a fresh book says where it stands.
**Split**:
Moving half the markets off a connection the exchange keeps closing.
**Quiet**:
Said of a symbol's subscription on a connection that is up, while the
exchange has stopped sending that symbol's changes there. Nothing tells us:
it has to be looked for.
_Avoid_: Stale, dead (a connection can be fine while one of its
subscriptions is quiet)
**Resubscribe**:
To unsubscribe a symbol on one connection and subscribe it again there, which
makes the exchange start its subscription afresh. A resync is a resubscribe
whose book we asked for.
**Silent**:
Said of a connection that is open but delivers nothing at all, not even the
answer to its keepalive.
**Recycle**:
To close a working connection on purpose and reconnect it with the same
symbols, to shed what has gone wrong on the exchange's side of it.
_Avoid_: Restart, rotate
**Receive time**:
When a message came off the socket, by the local clock.
_Avoid_: Arrival time, local timestamp
**Loss**:
A change for a symbol that no connection delivered in time, which leaves its
book silently wrong. A change only one connection missed is rescued, not
lost.
_Avoid_: Gap (a skipped sequence number is one connection's, and may be no
loss at all)
**Resync**:
Asking the exchange for a fresh full book of a symbol and replacing ours with
it, whenever our book can't be verified. Our book was wrong only if the fresh
one differs: a loss, or a subscription gap.
_Avoid_: Re-snapshot, rebuild, refresh
**Subscription gap**:
The changes our own subscribe keeps from us, whether it opens or resyncs a
symbol: the exchange holds them in the book it sends but never sends them as
changes. A connection that stays subscribed meanwhile has no such gap.
_Avoid_: Resync gap (the older, narrower name); calling it a loss (the state
stays right; only those updates are missing)
## Output
**Snapshot**:
The output message holding one symbol's whole book at a point in time.
_Avoid_: using it for the action of fetching a fresh book (that is a resync)
**Delta**:
The output message changing one price level of one symbol's book to a new
absolute quantity (`PriceLevelUpdate` in the schema).
_Avoid_: Increment, diff