This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/notes.md
T
pascal 515e85941d feat(latency): latency experiments, LATENCY.md, Kalshi's slow-connection recycle
Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16
connections to the same busy markets and writes every copy of every change;
`scripts/latency/analyse.py` gives each connection's speed, whether it
persists, what keeping the fastest would gain, lateness by address, and lag
with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed
persists but is worth under a millisecond, at the median only; on Kalshi's
demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold
agreed before the runs.

So the collector now recycles a Kalshi connection whose mean lateness stays,
3 minutes running, in the slowest quarter and above twice everyone's, ahead
of its hourly turn. Kalshi's rule measures each duplicate's lateness; the
orchestrator judges once a minute. A duplicate's lag in Polymarket's minute
line is now its "lateness".

Roadmap milestone 7, roadmap/07.
2026-11-12 15:05:25 +01:00

340 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Challenge notes
## Deltas are absolute, not incremental
A delta carries the **new total quantity** at a price level, not a change to it.
`quantity: 400` means "this level is now 400," not "+400". Overwrite the level;
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
available quantity".) That is the output, and Polymarket's wire. Kalshi's wire
is relative (`delta_fp`: "+400"): the book adds it, and the Delta written
still carries the level's new total.
## A missed message means a wrong book: resync it
If a message is lost, the in-memory book is silently wrong and cannot be
patched — throw it away and resync that symbol from a fresh full book. This is the
"completeness" requirement made mechanical. *How* a loss is detected depends on
the exchange: a sequence-number gap where one exists (Kalshi: per connection,
below), something else where it doesn't — Polymarket has no sequence number,
but its book hash does the job (below).
The challenge brief prescribes no mechanism; it only asks for complete data.
## Polymarket market channel (observed 2026-11-04)
From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
[market channel docs](https://docs.polymarket.com/market-data/websocket/market-channel.md)
(read 2026-11-04).
- **Endpoint:** `wss://ws-subscriptions-clob.polymarket.com/ws/market`, public,
no auth. Subscribe: `{"type":"market","assets_ids":["<token id>", ...]}`.
On an open connection, `{"assets_ids":[...],"operation":"subscribe"}` (or
`"unsubscribe"`) changes the token set without reconnecting.
- **`custom_feature_enabled: true`** in the subscribe message adds
`best_bid_ask`, `new_market` and `market_resolved` events — candidate signals
for symbol discovery (M3), possibly replacing REST polling.
- **Symbol = token.** A market (`market`, a `0x…` condition id) has two tokens
(`asset_id`: Yes, No), each with its own book. The two books are **exact
mirrors**: Yes had only bids (best 0.999), No only asks (best 0.001), and
every change arrives as a pair in one message — Yes `BUY p` with No `SELL 1−p`,
same size.
- **Frames:** on subscribe, one frame holding a JSON **array** of `book`
messages (one per token); afterwards, one **object** per frame. A frame's text
can end with a newline.
- **Full books also arrive mid-stream.** After a trade the server sent, all with
one timestamp: the `price_change`, then a full `book` object for each token,
then a `last_trade_price` — a natural resync point.
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
of `{price, size}`, bids sorted **ascending** and asks **descending** by
price, so the best level of each side comes *last*. Its `timestamp` is the
time of the book's **last change**, not the send time: 19 s old on receipt.
Books sent mid-stream omit `tick_size` and `last_trade_price`.
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
level size (docs: `"0"` removes the level).
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
into fixed point (`1_000_000` = 1), never through a float. Prices and sizes
are canonical (no trailing zeros: every one in 40 REST books, 2026-11-04);
`last_trade_price` is not (`"0.080"`).
- **Timestamps are milliseconds** since the epoch (the schema stores
microseconds).
- **No sequence number**, confirmed by the docs: none, and no option enables
one, for the market channel (the `seq` in Polymarket's docs belongs to
PolyBolt, a separate price feed). The docs describe no gap detection or
resync either.
- **The book hash identifies the full book state, and we can reproduce it**
(found 2026-11-04; the docs don't describe it). It is the SHA-1, as 40 lowercase
hex chars, of this compact JSON (no spaces), keys in this order:
`{"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"}`
— the REST `/book` response with `hash` emptied. Bids ascending, asks
descending, prices and sizes as the exchange writes them. On a change, the
`timestamp` is the change message's. `min_order_size` (`"5"`) and `neg_risk`
(`true`) are not on the websocket: Gamma's `orderMinSize` / `negRisk` give
them (matched REST in 40 of 40 markets). Checked live: 14 books and 182
changes on 6 tokens, every one matched after folding the changes into our
own book. A change's `hash` therefore also equals that of the next `book`.
- **One update can span several frames** (seen in a 90 s recording of 40
tokens): `price_change` frames with the same `timestamp`, one change each
for the same token, all carrying the hash of the book after the *whole*
batch. Check a hash once its batch is over.
- **`last_trade_price` belongs to the market**: both tokens of a market hash
the same value (20 of 20 markets). A trade changes it, and the hashes use the
new value up to ~680 ms before the `last_trade_price` event arrives; that
event names one token, and the hashed value was its `price` (16 cases), its
complement 1 − `price` (8 cases), or neither (1 case, a trade that probably
filled at two prices). So a hash check fails after most trades with nothing
lost; a fresh book (from a resync) carries the right value.
- **Resync on the open connection:** `unsubscribe` then `subscribe` for a token
makes the server send its full `book` at once. That book can lag the changes
already sent by a few ms: in a 2-minute run, 14 of 14 fresh books that
differed from ours had the hash of one of our own recent changes, i.e. a
state we had already passed. Its timestamp and `last_trade_price` can be
newer than its levels: in a 15-minute run, 16 of 27 fresh books that
differed from ours equalled an earlier state of ours by levels, while their
hash matched none of our changes (two of them with a timestamp ending in
`.000`, as if cut to the second). So recognise a lagging book by its levels.
It can also be *ahead*: between handling the
`unsubscribe` and the `subscribe`, the server sends no changes for the
token, so the book holds changes we never receive (6 of 302 resyncs in a
10-minute run, 1 to 48 ms ahead of our last change; 39 others were behind). A `subscribe` for a token
already subscribed is answered with the plain text `NO NEW ASSETS` (not
JSON).
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
answers `PONG`. The spike survived 60 s without it, but it is required.
- **Server restarts:** the server can close the socket with code `Away`,
reason `server shutting down` (seen once in about 8 minutes of connections).
- **`last_trade_price`:** `market`, `asset_id`, `price`, `size`, `side`,
`fee_rate_bps`, `timestamp`, `transaction_hash` (one per trade, for one
token).
- **Not observed:** `market_resolved`, in 5 minutes with
`custom_feature_enabled` on 10 tokens. The docs now show it only in an
SDK's wrapped form (`topic` / `payload`, camelCase), not the raw wire
shape above.
- **`new_market` is broadcast** (with `custom_feature_enabled`, 2026-11-04):
every market created, subscribed or not (69 in 5 minutes), with
`condition_id`, `assets_ids` / `clob_token_ids`, `question`, `active`
(`false` for markets created ahead), but no `order_min_size` or `neg_risk`,
which the book hash needs.
- **Tokens per connection (2026-11-04):** no cap seen: one connection served
50,000 tokens. But a first `{"type":"market"}` message with more than 500
tokens gets no opening books (1,000 to 10,000 tried; changes still flow),
while `subscribe` operations of 500 or 1,000 get theirs. A connection opened
with `"assets_ids": []` stays up and serves a later `subscribe`.
- **Slow consumers are closed:** a connection the server can't send to fast
enough is closed with code 1013, reason `slow consumer: send buffer full`
(3 of 5 test connections, with a Python reader parsing every frame at up to
~10,000 frames/s). The collector's reader, which only stamps, got it too:
six times in 8 minutes on the one connection carrying ~5,000 changes a
second (NFL quarter spreads, little traded); none on the others, the next
at ~1,300. No rate is safe (2026-11-10, server, 5 ms from the edge): four
connections ramped on random live tokens were closed at 0.7, 2.1, 2.3 and
5.1 MB/s (minute averages of frame bytes), with subscribes of 5,000 or 500
tokens alike. 5,000 random tokens now carry ~380 frames/s, ~200 KB/s.
Closes come together: 5 of 20 connections (~1 MB/s each) were closed
within 45 s of each other, one at 0.46 MB/s.
- **A subscribe hides a few ms of changes** (smoke run, 2026-11-04): after
subscribing a market on a busy connection, the first check of its opening
book often failed, and the fresh book differed from ours at levels none of
our changes had touched, though the opening book matched its own hash. The
server never sent those changes: like the resync's unsubscribe/subscribe,
any subscribe opens this gap (glossary: subscription gap).
- **`tick_size_change`** (recorded 2026-11-10 on every live token: 62 markets
in 10 minutes, all `0.01` to `0.001`): one object per token, both tokens of
the market, each sent twice on the same connection, identical:
`{"market", "asset_id", "old_tick_size", "new_tick_size", "timestamp",
"event_type": "tick_size_change"}`.
- **Load (2026-11-04):** a random 5,000 of the live tokens gave 133 frames/s
and 266 changes/s; their books had a median of 5 levels (mean 10.9, p90 29,
max 136), 15% empty. The top 50,000 by volume gave ~10,000 frames/s.
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
Fetch with curl in throwaway scripts.
## Two connections on the same symbols (observed 2026-11-11)
- **Same changes, same order**: two connections subscribed to the same 8,000
tokens for 335 s delivered the same 616,708 changes, none on only one, in
the same order per token. So the book hash orders changes across
connections too.
- **The second delivery's lag**: 0.5 ms at the median, over 10 ms for 6.5% of
changes, over 100 ms for 1%, 706 ms at worst; up to 82 changes of its token
behind. Neither connection leads (one was first 34% of the time).
- **Batches**: 95% of updates are one change. The others come one change per
frame, all with the hash of the whole batch, within 1 ms (40 ms at worst).
- **The server repeats some changes** on one connection (104 in that
recording, the same on both): harmless, quantities being absolute.
- **Connections carrying the same tokens are closed together**: two ramped on
the same tokens got their 1013 close 3 s apart. Over 20 minutes on 48,038
tokens with two subscriptions each, twins in creation order had both
subscriptions down for 33,345 token-seconds, twins dealt at random 3,787,
two independent random deals 272.
- **300 connections** opened at once from one address all stayed up.
- **A connection whose every subscribed market has resolved is closed** by the
server, code 1000, reason `all subscribed assets resolved` (seen on a
connection left with one market, 2026-11-11).
- **A change's `timestamp` never goes back within a token** on a connection
(0 of 660,084 changes on each of the two):
one older than the last applied is a late copy.
## Quiet subscriptions and reconnects (observed 2026-11-11)
From a 25-minute live run on every live market, two connections each, one
connection recycled every 60 s (sprint 05).
- **Quiet subscriptions are rare**: in 21 minutes, one market once had 3 or
more changes on one connection and none on the other; none had 10. Both
tokens of a market go one-sided together.
- **A reconnect's opening book can lag by minutes**: for symbols unchanged
for 10 s or more, 514 of 19,379 differed from ours, 512 of them equal to
one of our earlier states (1 to 4 levels behind, some over 10 minutes),
none with a newer timestamp. Books of symbols never changed all matched
(42,096). Under 10 s, some ran ahead of our stream.
- **With the monitoring on** (30 minutes, 107 recycles): 26 subscriptions
resubscribed for being quiet, none quiet again the next minute; of 335,488
reconnect books of unchanged symbols, 7,240 fresh books in all were behind
ours; no connection silent for 30 s.
- **A reconnect takes ~1.2 s** from close to up (1 s of it our wait); the
opening books of ~4,500 tokens come 0.3 s later on most connections, 20 s
on one; 2 of 4,490 never came on another.
## Gamma, the market listing (observed 2026-11-04)
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
`enableOrderBook` and `acceptingOrders` (~476k tokens). 32,092 still accept
orders past their `endDate`, waiting to be resolved: the end date doesn't
mean closed. ~20k markets are created a day, mostly crypto "Up or Down"
markets created a day ahead and accepting orders at once.
- **Pages of 100**, whatever `limit` asks. `/markets` refuses offsets past
2,000 ("use /markets/keyset"); `/markets/keyset` pages with `after_cursor`
(the `next_cursor` of the last page) and has no such limit. A full keyset
scan of the live markets took 6 min 41 s.
- **Sorting works, date filters don't:** `order=createdAt&ascending=false`
(newest first) and `closed=true&order=closedTime&ascending=false` sort both
endpoints, and keyset pages keep the order across cursors. `created_at_min`
and `closed_time_min` are ignored. 2–20 markets were created and 10–18
closed a minute; more than 420 a minute changed `updatedAt`.
- **`conditionId`** is the websocket's `market`; `clobTokenIds` (a JSON array
inside a string) are its symbols.
- **Null hash inputs:** a null `orderMinSize` (31 live markets) hashes as
`"0"`, what REST's `/book` reports; a null `negRisk` (230) as `false`.
- **Scan speed:** the full keyset scan took 6 min 41 s from a Python script
spawning curl per page, ~2 minutes from the collector (`ureq`, one agent),
under a minute from the server (2026-11-10).
- **Indexing delay (2026-11-10):** new markets appear in the newest-first
listing 219 to 296 s after their `createdAt` (8 markets, polled every
20 s): a check reaching back 5 minutes misses some. 195,130 markets were
live that evening.
## Kalshi's websocket (demo, observed 2026-11-11)
From Kalshi's docs and four recordings of the demo with a throwaway recorder
(`../kalshi-spike/`, outside the repo). Production is untried where a number
is the demo's.
- **Hosts:** production `wss://external-api-ws.kalshi.com/trade-api/ws/v2`
and `https://external-api.kalshi.com/trade-api/v2`; demo
`wss://external-api-ws.demo.kalshi.co/trade-api/ws/v2` and
`https://external-api.demo.kalshi.co/trade-api/v2`. The older
`api.elections.kalshi.com` and `demo-api.kalshi.co` still answer.
- **Auth:** the websocket handshake carries `KALSHI-ACCESS-KEY` (the key id),
`KALSHI-ACCESS-TIMESTAMP` (ms) and `KALSHI-ACCESS-SIGNATURE`: base64 of the
Ed25519 signature of `<timestamp>GET/trade-api/ws/v2`. Without them: 401.
The market listing needs no key. Keys are created per environment
(Account → Profile → API Keys); the private key is a PKCS#8 PEM shown once.
- **Subscribe:** `{"id":1,"cmd":"subscribe","params":{"channels":
["orderbook_delta"],"market_tickers":[...]}}` is answered
`{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}`,
then one `orderbook_snapshot` per market. A connection has one such
subscription: a second `subscribe`, or `{"cmd":"update_subscription",
"params":{"sids":[1],"market_tickers":[...],"action":"add_markets"}}`, adds
markets and sends their books; `delete_markets` stops a market's changes;
`get_snapshot` sends books again (carrying the command's `id`). Each `ok`
answer lists every market of the subscription unless `skip_ticker_ack:
true` is sent. An empty ticker list is error 14.
- **`orderbook_snapshot`:** `{"type","sid","seq","sending_ts_ms","msg":
{"market_ticker","market_id","clock","yes_dollars_fp":[["0.4800","80.00"],
...],"no_dollars_fp":[...]}}`. A side with no level is omitted. Bids only:
YES bids and NO bids. No time of its own.
- **`orderbook_delta`:** `msg` is `{"market_ticker","market_id","clock",
"price_dollars":"0.9600","delta_fp":"-54.00","side":"yes"|"no","ts",
"ts_ms"}`: a signed change of the level's quantity. Folding them gave the
exchange's book every time (1,352 `get_snapshot` books equal to ours, no
level below zero).
- **`use_yes_price: true`** in the subscribe gives NO levels at `1 − q`, the
YES ask they are (books and changes alike); the default, `false`, gives
`q` and is announced to flip.
- **`seq`** is contiguous per connection across all its markets, counting
books, changes, `ok` and `error` (not `subscribed`): no gap in 343,000
messages.
- **`clock`** is per market, the same on every connection, never going back,
and shared by 2 to 4 changes for 12% of its values (some identical in
content). A book's clock is its last change's: none of 1,378 books was
followed by a change of its own clock or an older one. Two connections
deliver a market's changes in the same order (2,643 changes on 26 markets),
the second copy 2 ms later at the median, 344 ms at worst.
- **No limit met per connection:** 30,000 markets in one subscribe message,
or in 30 messages 20 ms apart; every book within 1.7 s; ~1,560 changes and
566 KB a second. The docs give 200 connections per user and unnumbered
limits (errors 26 and 27).
- **Keepalive:** Kalshi sends a Ping frame (`heartbeat`) to a connection it
has sent nothing else for 10 s; a busy connection gets none. tungstenite
answers it by itself.
- **An unopened market can be subscribed:** its book is empty with `clock: 0`
and `market_id: ""`, and its changes flow from its first order after the
opening, with no new subscribe.
- **Time:** changes arrived 88 to 174 ms after their `ts_ms` (median 90),
from the server.
- **Listing:** `GET /markets?status=open&mve_filter=exclude&limit=1000`,
paged by `cursor`: 177,966 open markets on production, about 4 minutes
unauthenticated. `status=open` returns `active` markets, `unopened`
`initialized` ones, `closed` also `determined` ones. `min_created_ts`
(with no status: open and unopened alike), `status=closed&min_close_ts`
and `tickers=a,b` all work without a key. Markets open at their
`open_time` (none of the demo's 20,504 unopened ones was overdue), often
long after creation: of 15,000 open markets, 82% opened more than a minute
after, 19% more than 15 minutes.
- **Commands on an open subscription** (tried 2026-11-11): `skip_ticker_ack`
works on `update_subscription` too (the `ok` comes with an empty `msg`);
a subscription emptied by `delete_markets` stays and takes markets again,
with their books; subscribing a market already subscribed answers `ok` and
sends no book, so a fresh book takes `delete_markets` first; deleting an
unknown ticker answers `ok`.
- **A market's close sends its subscribers a fresh book**, unasked and
without the changes that lead to it (20-minute demo run, 2026-11-11: 54
markets closing at 20:00 UTC, each a newer, different book at 20:00:01 to
20:00:04). Taken as a book past ours that differs, it is logged as a loss;
the Snapshot is right.
- **A closed market answers like an unopened one**: an empty book, `clock:
0`, `market_id: ""`.
- **Handshakes are rate limited**: opening 110 connections at once, 54
handshakes were refused with 429; each got in on its retry a second later.
- **Production lists unopened markets past their `open_time`** (seen
2026-11-11: `initialized` 16 minutes after it); the demo had none.
- **The demo's size:** 246,365 markets open or opening within two hours
(2026-11-11, 19:29 UTC).
- **Maintenance (docs):** Thursdays 3:00 to 5:00 AM ET; connections drop.
## Latency, from the server (observed 2026-11-12)
- **The server's clock is synced**: SNTP against `time.cloudflare.com` read
−0.25 to +0.69 ms, 3 to 4 ms round trips (sprint 07's probe).
- **What the exchange sees isn't ours to choose**: a NAT rewrites every
connection's source port (not even stable for one local port), and there
is no IPv6 route; Polymarket's websocket resolves to 4 IPv6 addresses
beside its 4 IPv4 ones (CloudFront).
- **Where the hosts are** (AWS's published ranges): Kalshi production's
websocket in us-east-2, the demo's two addresses in us-west-2; lag about
14 ms to Polymarket, about 100 ms to Kalshi's demo.
- **Polymarket repeats a change on one connection**, rarely: 22 of 29,922
changes came twice on every connection in a minute.
- **Kalshi's listing order moves**: its first 30,000 open markets held the
98 busiest at 11:12 and none of them at 11:50. Pick by counting every
market, not by the listing's first pages. The demo reports
`volume_24h_fp` 0.00 (267,691 open markets, a full listing in 75 s).
## Clock accuracy bounds time accuracy
`received_at` comes from the local clock, so it carries the clock's error. On
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
made raw `received_at − exchange timestamp` read −14 to +41 ms; corrected, the
real lag was ~+16 to +71 ms. Latency figures need a synced clock, or the offset
measured and reported alongside them.