feat!: a generic core, and Kalshi through it

The orchestrator, the connection loop and the book thread's shell move to
src/core/, generic over an Exchange and its loss rule (ADR 0004).
Polymarket runs through it unchanged. Kalshi joins: a signed handshake
(KALSHI_KEY_ID_FILE, KALSHI_PRIVATE_KEY_FILE), a reconnect on a skipped
sequence number, first one wins by clock and rank, and its market listing
with markets tracked from three hours before they open.

Breaking: the exchange is now the first argument,
`engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]`.
This commit is contained in:
pascal committed 2026-11-11 20:54:17 +00:00
1 parent 480ee3ad2d
commit 1a5bbcd602
33 files changed
+8150 -1846

No files matched your search

+43 -21
View File
@@ -46,6 +46,9 @@ The interacting decisions it is made of:
chosen from its market's id; an orchestrator decides which connections
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
- **The exchange seam:** settled in sprint 06 — the pipeline is the shared
core; an exchange supplies its websocket layer and, inside the book
thread, its loss rule ([ADR 0004](adr/0004-exchange-seam.md)).
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
tension. Drop-on-full loses data (violates completeness); block adds latency
and fills socket buffers. This decision is the bet in miniature.
@@ -61,13 +64,16 @@ The output's value is completeness + time-accuracy, so whatever the topology, it
must guarantee:
1. **Receive time stamped at the socket edge**, before any queue or processing.
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
quantity 0 deletes it).
2. **A per-symbol book folded from the exchange's changes**, written as
absolute Deltas (the level's new quantity; 0 deletes it). Polymarket's
changes are already absolute; Kalshi's are relative and are added to the
level.
3. **Loss detection → resync** (a lost message means the book is silently
corrupt; replacing it with a fresh full book is the only safe response). How
a loss is detected depends on the exchange: a sequence gap where one exists;
on Polymarket, which has none, recomputing its book hash after every change
(settled in M2; the formula is in [notes.md](notes.md)).
a loss is detected depends on the exchange: on Polymarket, recomputing its
book hash after every change (settled in M2; the formula is in
[notes.md](notes.md)); on Kalshi, a skipped sequence number on a
connection, which reconnects it (settled in M6).
These are proven first, before redundancy or scale.
@@ -112,8 +118,9 @@ Their operational form, for writing code, is in the project
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
only.
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
takes a file path; the program's only argument is that path, default
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
takes a file path; the program takes the exchange (`polymarket`, `kalshi`
or `kalshi-demo`, required since sprint 06: one exchange per process), then
that path, default `/dev/stdout`, and logs go to stderr. No rotation: one output per run
(rotation can't apply to stdout; a consumer that wants files rotates them).
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
@@ -154,7 +161,8 @@ schema debate). As settled in sprints 04 and 05:
- The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting, and recycling one
connection at a time, about hourly. It never blocks on HTTP.
- The two **fetchers** only page Gamma on request and stream what they read.
- The two **fetchers** only page the exchange's market listing on request
and stream what they read.
- A **connection** reads, stamps the receive time, parses, and routes each
event to the book thread of its market. Every symbol is on two connections,
chosen independently. One silent for 30 s reconnects.
@@ -170,6 +178,14 @@ Settled: OS threads, shared-nothing, no mutexes
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
[sprint 04](../roadmap/04.%20redundant%20connections.md).
The pipeline is the same for every exchange. What an exchange supplies
([ADR 0004](adr/0004-exchange-seam.md)): the websocket layer of a connection
(request, encoding of commands, decoding of frames, keepalive), the fetchers,
and the rule a book thread applies to an event (duplicate, next, held, loss).
On Polymarket that rule is the hash chain above; on Kalshi a change is the
next one when its clock and rank pass the book's, and a held change doesn't
exist.
## Data model, first sketch
Supporting detail (mostly Claude's to settle), serving the pipeline above.
@@ -178,27 +194,33 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
- **Symbol** (from discovery): one order book, with its trading status and any
price/tick metadata the exchange gives. On Polymarket that is a token, not a
market: a market has two (Yes / No), each with its own, mirrored book.
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
produces so the core never sees exchange formats. Either a full snapshot or a
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
whatever the exchange offers for loss detection (a sequence number, a book
hash), and payload (side/price/quantity, or the full book).
market: a market has two (Yes / No), each with its own, mirrored book. On
Kalshi it is the market: its bids for Yes are the bids, its bids for No at
`q` the asks at `1 − q`.
- **Event** (settled in sprint 06, in place of the `NormalizedEvent` first
sketched): each exchange's own type. The core reads only the market it
belongs to and whether it is a full book; the exchange's rule reads the
rest (side, price, quantity, and what the exchange offers for loss
detection: a book hash, a sequence number, a clock) and folds it into the
symbol's book, setting a level's quantity (Polymarket) or moving it by a
signed amount (Kalshi).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
the last loss-detection marker seen and last exchange timestamp. It is fed
by the symbol's two connections, first one wins.
- **Connection** (one websocket): the set of symbols it carries. Live markets
are placed on connections by `conditionId`, both tokens together;
are placed on connections by market, all its symbols together;
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
live while active, not closed, with an order book and accepting orders).
live while active, not closed, with an order book and accepting orders),
Kalshi's its market listing (sprint 06: open, or opening within three
hours, tracked ahead so the book is there from its first order).
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
The debate lives in the **NormalizedEvent** shape (both exchanges must map
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
Redundancy was settled in sprint 04: no source is trusted, the first copy of
each change wins ([ADR 0003](adr/0003-first-one-wins.md)). On Kalshi the
first copy is recognised by clock and rank (sprint 06).
## Open questions
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
(Polymarket: settled in M2, by its book hash.)
None. (Loss detection on Kalshi was the last: settled in M6, by its
per-connection sequence number.)