feat!: a generic core, and Kalshi through it

The orchestrator, the connection loop and the book thread's shell move to
src/core/, generic over an Exchange and its loss rule (ADR 0004).
Polymarket runs through it unchanged. Kalshi joins: a signed handshake
(KALSHI_KEY_ID_FILE, KALSHI_PRIVATE_KEY_FILE), a reconnect on a skipped
sequence number, first one wins by clock and rank, and its market listing
with markets tracked from three hours before they open.

Breaking: the exchange is now the first argument,
`engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]`.
This commit is contained in:
pascal committed 2026-11-11 20:54:17 +00:00
1 parent 480ee3ad2d
commit 1a5bbcd602
33 files changed
+8150 -1846

No files matched your search

+2 -1
View File
@@ -44,4 +44,5 @@ Trade-off: the routing step 0002 avoided is back, parsing moves into the
connection thread, and the book thread's rules get subtler (held changes, a
ceiling, a chain declared broken). The design leans on an exchange that lets
the same change be recognised across connections: Polymarket's hash does;
Kalshi's `clock` looks as if it would (to confirm in M6).
on Kalshi a change's `clock` and its rank among the changes of that clock do
(recorded in M6, [sprint 06](../../roadmap/06.%20generic%20core%20and%20kalshi.md)).
+50
View File
@@ -0,0 +1,50 @@
# 0004. The exchange seam: a codec and a loss rule per exchange
## Context
Extension 3 asks for a generic service where "the only thing we need to
change is the encoding/decoding on the websocket layer". By M5 everything
lived in `src/polymarket/`, and M6 adds Kalshi. Most of the code is the same
for any exchange (the orchestrator, the connection loop, the book thread's
minute Snapshots, quiet watch and moves, the writer) or plainly per exchange
(URL, handshake, message formats, market listing). One part is neither: how
a loss is detected and how a symbol's two connections are merged
([0003](0003-first-one-wins.md)). Polymarket sends a hash of the whole book
with every change, which both verifies our book and says where a change from
either connection belongs. Kalshi has no hash: it numbers each connection's
messages and gives each market a clock. These are different mechanisms, not
two encodings of one.
## Decision
The seam has three parts. A **shared core** (`src/core/`) that names no
exchange. An **`Exchange` trait** for the websocket layer: the request, the
encoding of subscribe commands, the decoding of frames, the keepalive, and
the market fetcher. And, inside the book thread, a **rule supplied by the
exchange**: what a symbol's book does with an event from one of its
connections (a duplicate, the next change, held, a loss, a resync to ask
for), with its own per-symbol state.
The core understands of an event only the market it belongs to, which
routes it to a book thread, and whether it is a full book, which ends a move.
The event itself is the exchange's own type, handed to the exchange's rule
unread: the rule folds it into the symbol's book, which is the core's and
offers both ways a level changes (set to a quantity, moved by one).
## Why
Decode-only, as the brief's sentence reads, would put both loss rules in the
core, picked by exchange: the "generic" core would know each exchange's
quirks and a third exchange would mean editing it. A whole book thread per
exchange would keep the rules apart but write the minute Snapshots, the
quiet watch and the moves twice. With the rule behind the seam, the core is
really exchange-agnostic and the shell is written once. What the two rules
still share in shape (asking for a fresh book, and again when its connection
leaves or the minute passes) stays in each: their bookkeeping differs, and
Polymarket's was moved, not rewritten.
Trade-off: an exchange is more to write than a codec, and the seam is wider
than the brief asks. The book thread's shell and the rule also meet at more
points than a codec would (the minute's per-symbol questions, a connection
going down), so the rule's interface is the part most likely to move when a
third exchange arrives.
+95 -4
View File
@@ -4,15 +4,17 @@
A delta carries the **new total quantity** at a price level, not a change to it.
`quantity: 400` means "this level is now 400," not "+400". Overwrite the level;
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
available quantity".)
available quantity".) That is the output, and Polymarket's wire. Kalshi's wire
is relative (`delta_fp`: "+400"): the book adds it, and the Delta written
still carries the level's new total.
## A missed message means a wrong book: resync it
If a message is lost, the in-memory book is silently wrong and cannot be
patched — throw it away and resync that symbol from a fresh full book. This is the
"completeness" requirement made mechanical. *How* a loss is detected depends on
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
something else where it doesn't — Polymarket has no sequence number, but its
book hash does the job (below).
the exchange: a sequence-number gap where one exists (Kalshi: per connection,
below), something else where it doesn't — Polymarket has no sequence number,
but its book hash does the job (below).
The challenge brief prescribes no mechanism; it only asks for complete data.
## Polymarket market channel (observed 2026-11-04)
@@ -222,6 +224,95 @@ connection recycled every 60 s (sprint 05).
20 s): a check reaching back 5 minutes misses some. 195,130 markets were
live that evening.
## Kalshi's websocket (demo, observed 2026-11-11)
From Kalshi's docs and four recordings of the demo with a throwaway recorder
(`../kalshi-spike/`, outside the repo). Production is untried where a number
is the demo's.
- **Hosts:** production `wss://external-api-ws.kalshi.com/trade-api/ws/v2`
and `https://external-api.kalshi.com/trade-api/v2`; demo
`wss://external-api-ws.demo.kalshi.co/trade-api/ws/v2` and
`https://external-api.demo.kalshi.co/trade-api/v2`. The older
`api.elections.kalshi.com` and `demo-api.kalshi.co` still answer.
- **Auth:** the websocket handshake carries `KALSHI-ACCESS-KEY` (the key id),
`KALSHI-ACCESS-TIMESTAMP` (ms) and `KALSHI-ACCESS-SIGNATURE`: base64 of the
Ed25519 signature of `<timestamp>GET/trade-api/ws/v2`. Without them: 401.
The market listing needs no key. Keys are created per environment
(Account → Profile → API Keys); the private key is a PKCS#8 PEM shown once.
- **Subscribe:** `{"id":1,"cmd":"subscribe","params":{"channels":
["orderbook_delta"],"market_tickers":[...]}}` is answered
`{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}`,
then one `orderbook_snapshot` per market. A connection has one such
subscription: a second `subscribe`, or `{"cmd":"update_subscription",
"params":{"sids":[1],"market_tickers":[...],"action":"add_markets"}}`, adds
markets and sends their books; `delete_markets` stops a market's changes;
`get_snapshot` sends books again (carrying the command's `id`). Each `ok`
answer lists every market of the subscription unless `skip_ticker_ack:
true` is sent. An empty ticker list is error 14.
- **`orderbook_snapshot`:** `{"type","sid","seq","sending_ts_ms","msg":
{"market_ticker","market_id","clock","yes_dollars_fp":[["0.4800","80.00"],
...],"no_dollars_fp":[...]}}`. A side with no level is omitted. Bids only:
YES bids and NO bids. No time of its own.
- **`orderbook_delta`:** `msg` is `{"market_ticker","market_id","clock",
"price_dollars":"0.9600","delta_fp":"-54.00","side":"yes"|"no","ts",
"ts_ms"}`: a signed change of the level's quantity. Folding them gave the
exchange's book every time (1,352 `get_snapshot` books equal to ours, no
level below zero).
- **`use_yes_price: true`** in the subscribe gives NO levels at `1 − q`, the
YES ask they are (books and changes alike); the default, `false`, gives
`q` and is announced to flip.
- **`seq`** is contiguous per connection across all its markets, counting
books, changes, `ok` and `error` (not `subscribed`): no gap in 343,000
messages.
- **`clock`** is per market, the same on every connection, never going back,
and shared by 2 to 4 changes for 12% of its values (some identical in
content). A book's clock is its last change's: none of 1,378 books was
followed by a change of its own clock or an older one. Two connections
deliver a market's changes in the same order (2,643 changes on 26 markets),
the second copy 2 ms later at the median, 344 ms at worst.
- **No limit met per connection:** 30,000 markets in one subscribe message,
or in 30 messages 20 ms apart; every book within 1.7 s; ~1,560 changes and
566 KB a second. The docs give 200 connections per user and unnumbered
limits (errors 26 and 27).
- **Keepalive:** Kalshi sends a Ping frame (`heartbeat`) to a connection it
has sent nothing else for 10 s; a busy connection gets none. tungstenite
answers it by itself.
- **An unopened market can be subscribed:** its book is empty with `clock: 0`
and `market_id: ""`, and its changes flow from its first order after the
opening, with no new subscribe.
- **Time:** changes arrived 88 to 174 ms after their `ts_ms` (median 90),
from the server.
- **Listing:** `GET /markets?status=open&mve_filter=exclude&limit=1000`,
paged by `cursor`: 177,966 open markets on production, about 4 minutes
unauthenticated. `status=open` returns `active` markets, `unopened`
`initialized` ones, `closed` also `determined` ones. `min_created_ts`
(with no status: open and unopened alike), `status=closed&min_close_ts`
and `tickers=a,b` all work without a key. Markets open at their
`open_time` (none of the demo's 20,504 unopened ones was overdue), often
long after creation: of 15,000 open markets, 82% opened more than a minute
after, 19% more than 15 minutes.
- **Commands on an open subscription** (tried 2026-11-11): `skip_ticker_ack`
works on `update_subscription` too (the `ok` comes with an empty `msg`);
a subscription emptied by `delete_markets` stays and takes markets again,
with their books; subscribing a market already subscribed answers `ok` and
sends no book, so a fresh book takes `delete_markets` first; deleting an
unknown ticker answers `ok`.
- **A market's close sends its subscribers a fresh book**, unasked and
without the changes that lead to it (20-minute demo run, 2026-11-11: 54
markets closing at 20:00 UTC, each a newer, different book at 20:00:01 to
20:00:04). Taken as a book past ours that differs, it is logged as a loss;
the Snapshot is right.
- **A closed market answers like an unopened one**: an empty book, `clock:
0`, `market_id: ""`.
- **Handshakes are rate limited**: opening 110 connections at once, 54
handshakes were refused with 429; each got in on its retry a second later.
- **Production lists unopened markets past their `open_time`** (seen
2026-11-11: `initialized` 16 minutes after it); the demo had none.
- **The demo's size:** 246,365 markets open or opening within two hours
(2026-11-11, 19:29 UTC).
- **Maintenance (docs):** Thursdays 3:00 to 5:00 AM ET; connections drop.
## Clock accuracy bounds time accuracy
`received_at` comes from the local clock, so it carries the clock's error. On
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
+43 -21
View File
@@ -46,6 +46,9 @@ The interacting decisions it is made of:
chosen from its market's id; an orchestrator decides which connections
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
- **The exchange seam:** settled in sprint 06 — the pipeline is the shared
core; an exchange supplies its websocket layer and, inside the book
thread, its loss rule ([ADR 0004](adr/0004-exchange-seam.md)).
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
tension. Drop-on-full loses data (violates completeness); block adds latency
and fills socket buffers. This decision is the bet in miniature.
@@ -61,13 +64,16 @@ The output's value is completeness + time-accuracy, so whatever the topology, it
must guarantee:
1. **Receive time stamped at the socket edge**, before any queue or processing.
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
quantity 0 deletes it).
2. **A per-symbol book folded from the exchange's changes**, written as
absolute Deltas (the level's new quantity; 0 deletes it). Polymarket's
changes are already absolute; Kalshi's are relative and are added to the
level.
3. **Loss detection → resync** (a lost message means the book is silently
corrupt; replacing it with a fresh full book is the only safe response). How
a loss is detected depends on the exchange: a sequence gap where one exists;
on Polymarket, which has none, recomputing its book hash after every change
(settled in M2; the formula is in [notes.md](notes.md)).
a loss is detected depends on the exchange: on Polymarket, recomputing its
book hash after every change (settled in M2; the formula is in
[notes.md](notes.md)); on Kalshi, a skipped sequence number on a
connection, which reconnects it (settled in M6).
These are proven first, before redundancy or scale.
@@ -112,8 +118,9 @@ Their operational form, for writing code, is in the project
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
only.
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
takes a file path; the program's only argument is that path, default
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
takes a file path; the program takes the exchange (`polymarket`, `kalshi`
or `kalshi-demo`, required since sprint 06: one exchange per process), then
that path, default `/dev/stdout`, and logs go to stderr. No rotation: one output per run
(rotation can't apply to stdout; a consumer that wants files rotates them).
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
@@ -154,7 +161,8 @@ schema debate). As settled in sprints 04 and 05:
- The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting, and recycling one
connection at a time, about hourly. It never blocks on HTTP.
- The two **fetchers** only page Gamma on request and stream what they read.
- The two **fetchers** only page the exchange's market listing on request
and stream what they read.
- A **connection** reads, stamps the receive time, parses, and routes each
event to the book thread of its market. Every symbol is on two connections,
chosen independently. One silent for 30 s reconnects.
@@ -170,6 +178,14 @@ Settled: OS threads, shared-nothing, no mutexes
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
[sprint 04](../roadmap/04.%20redundant%20connections.md).
The pipeline is the same for every exchange. What an exchange supplies
([ADR 0004](adr/0004-exchange-seam.md)): the websocket layer of a connection
(request, encoding of commands, decoding of frames, keepalive), the fetchers,
and the rule a book thread applies to an event (duplicate, next, held, loss).
On Polymarket that rule is the hash chain above; on Kalshi a change is the
next one when its clock and rank pass the book's, and a held change doesn't
exist.
## Data model, first sketch
Supporting detail (mostly Claude's to settle), serving the pipeline above.
@@ -178,27 +194,33 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
- **Symbol** (from discovery): one order book, with its trading status and any
price/tick metadata the exchange gives. On Polymarket that is a token, not a
market: a market has two (Yes / No), each with its own, mirrored book.
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
produces so the core never sees exchange formats. Either a full snapshot or a
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
whatever the exchange offers for loss detection (a sequence number, a book
hash), and payload (side/price/quantity, or the full book).
market: a market has two (Yes / No), each with its own, mirrored book. On
Kalshi it is the market: its bids for Yes are the bids, its bids for No at
`q` the asks at `1 − q`.
- **Event** (settled in sprint 06, in place of the `NormalizedEvent` first
sketched): each exchange's own type. The core reads only the market it
belongs to and whether it is a full book; the exchange's rule reads the
rest (side, price, quantity, and what the exchange offers for loss
detection: a book hash, a sequence number, a clock) and folds it into the
symbol's book, setting a level's quantity (Polymarket) or moving it by a
signed amount (Kalshi).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
the last loss-detection marker seen and last exchange timestamp. It is fed
by the symbol's two connections, first one wins.
- **Connection** (one websocket): the set of symbols it carries. Live markets
are placed on connections by `conditionId`, both tokens together;
are placed on connections by market, all its symbols together;
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
live while active, not closed, with an order book and accepting orders).
live while active, not closed, with an order book and accepting orders),
Kalshi's its market listing (sprint 06: open, or opening within three
hours, tracked ahead so the book is there from its first order).
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
The debate lives in the **NormalizedEvent** shape (both exchanges must map
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
Redundancy was settled in sprint 04: no source is trusted, the first copy of
each change wins ([ADR 0003](adr/0003-first-one-wins.md)). On Kalshi the
first copy is recognised by clock and rank (sprint 06).
## Open questions
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
(Polymarket: settled in M2, by its book hash.)
None. (Loss detection on Kalshi was the last: settled in M6, by its
per-connection sequence number.)
+18 -5
View File
@@ -66,12 +66,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
quiet again) and no connection fell silent (sprint 05's Outcome).
6. **Extension 3 — generic core + Kalshi adapter** — *not started*
Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the
working Polymarket code, then add a Kalshi adapter so both exchanges run
through the same core. The abstraction is extracted from two concretes here,
not before. Requires a Kalshi API key.
6. **Extension 3 — generic core + Kalshi adapter** — *done (2026-11-11)*
The exchange-agnostic core extracted from the working Polymarket code,
and Kalshi added through it. Each exchange supplies its websocket layer
and its loss rule ([ADR 0004](adr/0004-exchange-seam.md)); on Kalshi,
first one wins by clock and rank.
Sprint: [../roadmap/06. generic core and kalshi.md](../roadmap/06.%20generic%20core%20and%20kalshi.md).
*Ends:* both Polymarket and Kalshi work through one generic core.
Result: 10-minute live runs of each, reconciled; Polymarket's losses no
higher than sprint 05's; Kalshi on its demo (about 245,000 markets on 110
connections), production not run for lack of a key (sprint 06's Outcome).
7. **Extension 4 — latency experiments + writeup** — *not started*
Experiment with faster connections (first-after-maintenance, connection
@@ -125,3 +129,12 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
- A Kalshi production run, once a production key exists.
- Retrying a failed rescan sooner than the next hourly one (`maybe`: Kalshi's
three-hour opening window covers one failure).
- One place for the two rules' resync bookkeeping (asking for a fresh book,
again on leave and at the minute), in the shell (`maybe`: it means
rewriting Polymarket's).
- Telling Kalshi's book at a market's close, and a connection's book merely
ahead of the other's stream, from a real loss (sprint 06 logs all three as
a loss; the Snapshot is right either way).