Archived
feat!: a generic core, and Kalshi through it
The orchestrator, the connection loop and the book thread's shell move to src/core/, generic over an Exchange and its loss rule (ADR 0004). Polymarket runs through it unchanged. Kalshi joins: a signed handshake (KALSHI_KEY_ID_FILE, KALSHI_PRIVATE_KEY_FILE), a reconnect on a skipped sequence number, first one wins by clock and rank, and its market listing with markets tracked from three hours before they open. Breaking: the exchange is now the first argument, `engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]`.
This commit is contained in:
33 files changed
+8150
-1846
No files matched your search
@@ -44,4 +44,5 @@ Trade-off: the routing step 0002 avoided is back, parsing moves into the
|
||||
connection thread, and the book thread's rules get subtler (held changes, a
|
||||
ceiling, a chain declared broken). The design leans on an exchange that lets
|
||||
the same change be recognised across connections: Polymarket's hash does;
|
||||
Kalshi's `clock` looks as if it would (to confirm in M6).
|
||||
on Kalshi a change's `clock` and its rank among the changes of that clock do
|
||||
(recorded in M6, [sprint 06](../../roadmap/06.%20generic%20core%20and%20kalshi.md)).
|
||||
@@ -0,0 +1,50 @@
|
||||
# 0004. The exchange seam: a codec and a loss rule per exchange
|
||||
|
||||
## Context
|
||||
|
||||
Extension 3 asks for a generic service where "the only thing we need to
|
||||
change is the encoding/decoding on the websocket layer". By M5 everything
|
||||
lived in `src/polymarket/`, and M6 adds Kalshi. Most of the code is the same
|
||||
for any exchange (the orchestrator, the connection loop, the book thread's
|
||||
minute Snapshots, quiet watch and moves, the writer) or plainly per exchange
|
||||
(URL, handshake, message formats, market listing). One part is neither: how
|
||||
a loss is detected and how a symbol's two connections are merged
|
||||
([0003](0003-first-one-wins.md)). Polymarket sends a hash of the whole book
|
||||
with every change, which both verifies our book and says where a change from
|
||||
either connection belongs. Kalshi has no hash: it numbers each connection's
|
||||
messages and gives each market a clock. These are different mechanisms, not
|
||||
two encodings of one.
|
||||
|
||||
## Decision
|
||||
|
||||
The seam has three parts. A **shared core** (`src/core/`) that names no
|
||||
exchange. An **`Exchange` trait** for the websocket layer: the request, the
|
||||
encoding of subscribe commands, the decoding of frames, the keepalive, and
|
||||
the market fetcher. And, inside the book thread, a **rule supplied by the
|
||||
exchange**: what a symbol's book does with an event from one of its
|
||||
connections (a duplicate, the next change, held, a loss, a resync to ask
|
||||
for), with its own per-symbol state.
|
||||
|
||||
The core understands of an event only the market it belongs to, which
|
||||
routes it to a book thread, and whether it is a full book, which ends a move.
|
||||
The event itself is the exchange's own type, handed to the exchange's rule
|
||||
unread: the rule folds it into the symbol's book, which is the core's and
|
||||
offers both ways a level changes (set to a quantity, moved by one).
|
||||
|
||||
## Why
|
||||
|
||||
Decode-only, as the brief's sentence reads, would put both loss rules in the
|
||||
core, picked by exchange: the "generic" core would know each exchange's
|
||||
quirks and a third exchange would mean editing it. A whole book thread per
|
||||
exchange would keep the rules apart but write the minute Snapshots, the
|
||||
quiet watch and the moves twice. With the rule behind the seam, the core is
|
||||
really exchange-agnostic and the shell is written once. What the two rules
|
||||
still share in shape (asking for a fresh book, and again when its connection
|
||||
leaves or the minute passes) stays in each: their bookkeeping differs, and
|
||||
Polymarket's was moved, not rewritten.
|
||||
|
||||
Trade-off: an exchange is more to write than a codec, and the seam is wider
|
||||
than the brief asks. The book thread's shell and the rule also meet at more
|
||||
points than a codec would (the minute's per-symbol questions, a connection
|
||||
going down), so the rule's interface is the part most likely to move when a
|
||||
third exchange arrives.
|
||||
+95
-4
@@ -4,15 +4,17 @@
|
||||
A delta carries the **new total quantity** at a price level, not a change to it.
|
||||
`quantity: 400` means "this level is now 400," not "+400". Overwrite the level;
|
||||
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
|
||||
available quantity".)
|
||||
available quantity".) That is the output, and Polymarket's wire. Kalshi's wire
|
||||
is relative (`delta_fp`: "+400"): the book adds it, and the Delta written
|
||||
still carries the level's new total.
|
||||
|
||||
## A missed message means a wrong book: resync it
|
||||
If a message is lost, the in-memory book is silently wrong and cannot be
|
||||
patched — throw it away and resync that symbol from a fresh full book. This is the
|
||||
"completeness" requirement made mechanical. *How* a loss is detected depends on
|
||||
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
|
||||
something else where it doesn't — Polymarket has no sequence number, but its
|
||||
book hash does the job (below).
|
||||
the exchange: a sequence-number gap where one exists (Kalshi: per connection,
|
||||
below), something else where it doesn't — Polymarket has no sequence number,
|
||||
but its book hash does the job (below).
|
||||
The challenge brief prescribes no mechanism; it only asks for complete data.
|
||||
|
||||
## Polymarket market channel (observed 2026-11-04)
|
||||
@@ -222,6 +224,95 @@ connection recycled every 60 s (sprint 05).
|
||||
20 s): a check reaching back 5 minutes misses some. 195,130 markets were
|
||||
live that evening.
|
||||
|
||||
## Kalshi's websocket (demo, observed 2026-11-11)
|
||||
|
||||
From Kalshi's docs and four recordings of the demo with a throwaway recorder
|
||||
(`../kalshi-spike/`, outside the repo). Production is untried where a number
|
||||
is the demo's.
|
||||
|
||||
- **Hosts:** production `wss://external-api-ws.kalshi.com/trade-api/ws/v2`
|
||||
and `https://external-api.kalshi.com/trade-api/v2`; demo
|
||||
`wss://external-api-ws.demo.kalshi.co/trade-api/ws/v2` and
|
||||
`https://external-api.demo.kalshi.co/trade-api/v2`. The older
|
||||
`api.elections.kalshi.com` and `demo-api.kalshi.co` still answer.
|
||||
- **Auth:** the websocket handshake carries `KALSHI-ACCESS-KEY` (the key id),
|
||||
`KALSHI-ACCESS-TIMESTAMP` (ms) and `KALSHI-ACCESS-SIGNATURE`: base64 of the
|
||||
Ed25519 signature of `<timestamp>GET/trade-api/ws/v2`. Without them: 401.
|
||||
The market listing needs no key. Keys are created per environment
|
||||
(Account → Profile → API Keys); the private key is a PKCS#8 PEM shown once.
|
||||
- **Subscribe:** `{"id":1,"cmd":"subscribe","params":{"channels":
|
||||
["orderbook_delta"],"market_tickers":[...]}}` is answered
|
||||
`{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}`,
|
||||
then one `orderbook_snapshot` per market. A connection has one such
|
||||
subscription: a second `subscribe`, or `{"cmd":"update_subscription",
|
||||
"params":{"sids":[1],"market_tickers":[...],"action":"add_markets"}}`, adds
|
||||
markets and sends their books; `delete_markets` stops a market's changes;
|
||||
`get_snapshot` sends books again (carrying the command's `id`). Each `ok`
|
||||
answer lists every market of the subscription unless `skip_ticker_ack:
|
||||
true` is sent. An empty ticker list is error 14.
|
||||
- **`orderbook_snapshot`:** `{"type","sid","seq","sending_ts_ms","msg":
|
||||
{"market_ticker","market_id","clock","yes_dollars_fp":[["0.4800","80.00"],
|
||||
...],"no_dollars_fp":[...]}}`. A side with no level is omitted. Bids only:
|
||||
YES bids and NO bids. No time of its own.
|
||||
- **`orderbook_delta`:** `msg` is `{"market_ticker","market_id","clock",
|
||||
"price_dollars":"0.9600","delta_fp":"-54.00","side":"yes"|"no","ts",
|
||||
"ts_ms"}`: a signed change of the level's quantity. Folding them gave the
|
||||
exchange's book every time (1,352 `get_snapshot` books equal to ours, no
|
||||
level below zero).
|
||||
- **`use_yes_price: true`** in the subscribe gives NO levels at `1 − q`, the
|
||||
YES ask they are (books and changes alike); the default, `false`, gives
|
||||
`q` and is announced to flip.
|
||||
- **`seq`** is contiguous per connection across all its markets, counting
|
||||
books, changes, `ok` and `error` (not `subscribed`): no gap in 343,000
|
||||
messages.
|
||||
- **`clock`** is per market, the same on every connection, never going back,
|
||||
and shared by 2 to 4 changes for 12% of its values (some identical in
|
||||
content). A book's clock is its last change's: none of 1,378 books was
|
||||
followed by a change of its own clock or an older one. Two connections
|
||||
deliver a market's changes in the same order (2,643 changes on 26 markets),
|
||||
the second copy 2 ms later at the median, 344 ms at worst.
|
||||
- **No limit met per connection:** 30,000 markets in one subscribe message,
|
||||
or in 30 messages 20 ms apart; every book within 1.7 s; ~1,560 changes and
|
||||
566 KB a second. The docs give 200 connections per user and unnumbered
|
||||
limits (errors 26 and 27).
|
||||
- **Keepalive:** Kalshi sends a Ping frame (`heartbeat`) to a connection it
|
||||
has sent nothing else for 10 s; a busy connection gets none. tungstenite
|
||||
answers it by itself.
|
||||
- **An unopened market can be subscribed:** its book is empty with `clock: 0`
|
||||
and `market_id: ""`, and its changes flow from its first order after the
|
||||
opening, with no new subscribe.
|
||||
- **Time:** changes arrived 88 to 174 ms after their `ts_ms` (median 90),
|
||||
from the server.
|
||||
- **Listing:** `GET /markets?status=open&mve_filter=exclude&limit=1000`,
|
||||
paged by `cursor`: 177,966 open markets on production, about 4 minutes
|
||||
unauthenticated. `status=open` returns `active` markets, `unopened`
|
||||
`initialized` ones, `closed` also `determined` ones. `min_created_ts`
|
||||
(with no status: open and unopened alike), `status=closed&min_close_ts`
|
||||
and `tickers=a,b` all work without a key. Markets open at their
|
||||
`open_time` (none of the demo's 20,504 unopened ones was overdue), often
|
||||
long after creation: of 15,000 open markets, 82% opened more than a minute
|
||||
after, 19% more than 15 minutes.
|
||||
- **Commands on an open subscription** (tried 2026-11-11): `skip_ticker_ack`
|
||||
works on `update_subscription` too (the `ok` comes with an empty `msg`);
|
||||
a subscription emptied by `delete_markets` stays and takes markets again,
|
||||
with their books; subscribing a market already subscribed answers `ok` and
|
||||
sends no book, so a fresh book takes `delete_markets` first; deleting an
|
||||
unknown ticker answers `ok`.
|
||||
- **A market's close sends its subscribers a fresh book**, unasked and
|
||||
without the changes that lead to it (20-minute demo run, 2026-11-11: 54
|
||||
markets closing at 20:00 UTC, each a newer, different book at 20:00:01 to
|
||||
20:00:04). Taken as a book past ours that differs, it is logged as a loss;
|
||||
the Snapshot is right.
|
||||
- **A closed market answers like an unopened one**: an empty book, `clock:
|
||||
0`, `market_id: ""`.
|
||||
- **Handshakes are rate limited**: opening 110 connections at once, 54
|
||||
handshakes were refused with 429; each got in on its retry a second later.
|
||||
- **Production lists unopened markets past their `open_time`** (seen
|
||||
2026-11-11: `initialized` 16 minutes after it); the demo had none.
|
||||
- **The demo's size:** 246,365 markets open or opening within two hours
|
||||
(2026-11-11, 19:29 UTC).
|
||||
- **Maintenance (docs):** Thursdays 3:00 to 5:00 AM ET; connections drop.
|
||||
|
||||
## Clock accuracy bounds time accuracy
|
||||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||||
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
|
||||
|
||||
+43
-21
@@ -46,6 +46,9 @@ The interacting decisions it is made of:
|
||||
chosen from its market's id; an orchestrator decides which connections
|
||||
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
|
||||
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
|
||||
- **The exchange seam:** settled in sprint 06 — the pipeline is the shared
|
||||
core; an exchange supplies its websocket layer and, inside the book
|
||||
thread, its loss rule ([ADR 0004](adr/0004-exchange-seam.md)).
|
||||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||||
and fills socket buffers. This decision is the bet in miniature.
|
||||
@@ -61,13 +64,16 @@ The output's value is completeness + time-accuracy, so whatever the topology, it
|
||||
must guarantee:
|
||||
|
||||
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
||||
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
||||
quantity 0 deletes it).
|
||||
2. **A per-symbol book folded from the exchange's changes**, written as
|
||||
absolute Deltas (the level's new quantity; 0 deletes it). Polymarket's
|
||||
changes are already absolute; Kalshi's are relative and are added to the
|
||||
level.
|
||||
3. **Loss detection → resync** (a lost message means the book is silently
|
||||
corrupt; replacing it with a fresh full book is the only safe response). How
|
||||
a loss is detected depends on the exchange: a sequence gap where one exists;
|
||||
on Polymarket, which has none, recomputing its book hash after every change
|
||||
(settled in M2; the formula is in [notes.md](notes.md)).
|
||||
a loss is detected depends on the exchange: on Polymarket, recomputing its
|
||||
book hash after every change (settled in M2; the formula is in
|
||||
[notes.md](notes.md)); on Kalshi, a skipped sequence number on a
|
||||
connection, which reconnects it (settled in M6).
|
||||
|
||||
These are proven first, before redundancy or scale.
|
||||
|
||||
@@ -112,8 +118,9 @@ Their operational form, for writing code, is in the project
|
||||
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
||||
only.
|
||||
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
|
||||
takes a file path; the program's only argument is that path, default
|
||||
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
||||
takes a file path; the program takes the exchange (`polymarket`, `kalshi`
|
||||
or `kalshi-demo`, required since sprint 06: one exchange per process), then
|
||||
that path, default `/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
||||
(rotation can't apply to stdout; a consumer that wants files rotates them).
|
||||
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
||||
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
||||
@@ -154,7 +161,8 @@ schema debate). As settled in sprints 04 and 05:
|
||||
- The **orchestrator** owns what is tracked (each market's two connections,
|
||||
the timers) and decides: placing, moving, splitting, and recycling one
|
||||
connection at a time, about hourly. It never blocks on HTTP.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- The two **fetchers** only page the exchange's market listing on request
|
||||
and stream what they read.
|
||||
- A **connection** reads, stamps the receive time, parses, and routes each
|
||||
event to the book thread of its market. Every symbol is on two connections,
|
||||
chosen independently. One silent for 30 s reconnects.
|
||||
@@ -170,6 +178,14 @@ Settled: OS threads, shared-nothing, no mutexes
|
||||
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
|
||||
[sprint 04](../roadmap/04.%20redundant%20connections.md).
|
||||
|
||||
The pipeline is the same for every exchange. What an exchange supplies
|
||||
([ADR 0004](adr/0004-exchange-seam.md)): the websocket layer of a connection
|
||||
(request, encoding of commands, decoding of frames, keepalive), the fetchers,
|
||||
and the rule a book thread applies to an event (duplicate, next, held, loss).
|
||||
On Polymarket that rule is the hash chain above; on Kalshi a change is the
|
||||
next one when its clock and rank pass the book's, and a held change doesn't
|
||||
exist.
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
||||
@@ -178,27 +194,33 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
|
||||
|
||||
- **Symbol** (from discovery): one order book, with its trading status and any
|
||||
price/tick metadata the exchange gives. On Polymarket that is a token, not a
|
||||
market: a market has two (Yes / No), each with its own, mirrored book.
|
||||
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
|
||||
produces so the core never sees exchange formats. Either a full snapshot or a
|
||||
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
|
||||
whatever the exchange offers for loss detection (a sequence number, a book
|
||||
hash), and payload (side/price/quantity, or the full book).
|
||||
market: a market has two (Yes / No), each with its own, mirrored book. On
|
||||
Kalshi it is the market: its bids for Yes are the bids, its bids for No at
|
||||
`q` the asks at `1 − q`.
|
||||
- **Event** (settled in sprint 06, in place of the `NormalizedEvent` first
|
||||
sketched): each exchange's own type. The core reads only the market it
|
||||
belongs to and whether it is a full book; the exchange's rule reads the
|
||||
rest (side, price, quantity, and what the exchange offers for loss
|
||||
detection: a book hash, a sequence number, a clock) and folds it into the
|
||||
symbol's book, setting a level's quantity (Polymarket) or moving it by a
|
||||
signed amount (Kalshi).
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
the last loss-detection marker seen and last exchange timestamp. It is fed
|
||||
by the symbol's two connections, first one wins.
|
||||
- **Connection** (one websocket): the set of symbols it carries. Live markets
|
||||
are placed on connections by `conditionId`, both tokens together;
|
||||
are placed on connections by market, all its symbols together;
|
||||
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
|
||||
live while active, not closed, with an order book and accepting orders).
|
||||
live while active, not closed, with an order book and accepting orders),
|
||||
Kalshi's its market listing (sprint 06: open, or opening within three
|
||||
hours, tracked ahead so the book is there from its first order).
|
||||
|
||||
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
||||
|
||||
The debate lives in the **NormalizedEvent** shape (both exchanges must map
|
||||
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
|
||||
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
|
||||
Redundancy was settled in sprint 04: no source is trusted, the first copy of
|
||||
each change wins ([ADR 0003](adr/0003-first-one-wins.md)). On Kalshi the
|
||||
first copy is recognised by clock and rank (sprint 06).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
None. (Loss detection on Kalshi was the last: settled in M6, by its
|
||||
per-connection sequence number.)
|
||||
+18
-5
@@ -66,12 +66,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
|
||||
quiet again) and no connection fell silent (sprint 05's Outcome).
|
||||
|
||||
6. **Extension 3 — generic core + Kalshi adapter** — *not started*
|
||||
Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the
|
||||
working Polymarket code, then add a Kalshi adapter so both exchanges run
|
||||
through the same core. The abstraction is extracted from two concretes here,
|
||||
not before. Requires a Kalshi API key.
|
||||
6. **Extension 3 — generic core + Kalshi adapter** — *done (2026-11-11)*
|
||||
The exchange-agnostic core extracted from the working Polymarket code,
|
||||
and Kalshi added through it. Each exchange supplies its websocket layer
|
||||
and its loss rule ([ADR 0004](adr/0004-exchange-seam.md)); on Kalshi,
|
||||
first one wins by clock and rank.
|
||||
Sprint: [../roadmap/06. generic core and kalshi.md](../roadmap/06.%20generic%20core%20and%20kalshi.md).
|
||||
*Ends:* both Polymarket and Kalshi work through one generic core.
|
||||
Result: 10-minute live runs of each, reconciled; Polymarket's losses no
|
||||
higher than sprint 05's; Kalshi on its demo (about 245,000 markets on 110
|
||||
connections), production not run for lack of a key (sprint 06's Outcome).
|
||||
|
||||
7. **Extension 4 — latency experiments + writeup** — *not started*
|
||||
Experiment with faster connections (first-after-maintenance, connection
|
||||
@@ -125,3 +129,12 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||||
minute: ~130 MB a minute at full coverage).
|
||||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|
||||
- A Kalshi production run, once a production key exists.
|
||||
- Retrying a failed rescan sooner than the next hourly one (`maybe`: Kalshi's
|
||||
three-hour opening window covers one failure).
|
||||
- One place for the two rules' resync bookkeeping (asking for a fresh book,
|
||||
again on leave and at the minute), in the shell (`maybe`: it means
|
||||
rewriting Polymarket's).
|
||||
- Telling Kalshi's book at a market's close, and a connection's book merely
|
||||
ahead of the other's stream, from a real loss (sprint 06 logs all three as
|
||||
a loss; the Snapshot is right either way).
|
||||
Reference in new issue
Block a user