Archived
feat!: a generic core, and Kalshi through it
The orchestrator, the connection loop and the book thread's shell move to src/core/, generic over an Exchange and its loss rule (ADR 0004). Polymarket runs through it unchanged. Kalshi joins: a signed handshake (KALSHI_KEY_ID_FILE, KALSHI_PRIVATE_KEY_FILE), a reconnect on a skipped sequence number, first one wins by clock and rank, and its market listing with markets tracked from three hours before they open. Breaking: the exchange is now the first argument, `engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]`.
This commit is contained in:
33 files changed
+8150
-1846
No files matched your search
+43
-21
@@ -46,6 +46,9 @@ The interacting decisions it is made of:
|
||||
chosen from its market's id; an orchestrator decides which connections
|
||||
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
|
||||
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
|
||||
- **The exchange seam:** settled in sprint 06 — the pipeline is the shared
|
||||
core; an exchange supplies its websocket layer and, inside the book
|
||||
thread, its loss rule ([ADR 0004](adr/0004-exchange-seam.md)).
|
||||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||||
and fills socket buffers. This decision is the bet in miniature.
|
||||
@@ -61,13 +64,16 @@ The output's value is completeness + time-accuracy, so whatever the topology, it
|
||||
must guarantee:
|
||||
|
||||
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
||||
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
||||
quantity 0 deletes it).
|
||||
2. **A per-symbol book folded from the exchange's changes**, written as
|
||||
absolute Deltas (the level's new quantity; 0 deletes it). Polymarket's
|
||||
changes are already absolute; Kalshi's are relative and are added to the
|
||||
level.
|
||||
3. **Loss detection → resync** (a lost message means the book is silently
|
||||
corrupt; replacing it with a fresh full book is the only safe response). How
|
||||
a loss is detected depends on the exchange: a sequence gap where one exists;
|
||||
on Polymarket, which has none, recomputing its book hash after every change
|
||||
(settled in M2; the formula is in [notes.md](notes.md)).
|
||||
a loss is detected depends on the exchange: on Polymarket, recomputing its
|
||||
book hash after every change (settled in M2; the formula is in
|
||||
[notes.md](notes.md)); on Kalshi, a skipped sequence number on a
|
||||
connection, which reconnects it (settled in M6).
|
||||
|
||||
These are proven first, before redundancy or scale.
|
||||
|
||||
@@ -112,8 +118,9 @@ Their operational form, for writing code, is in the project
|
||||
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
||||
only.
|
||||
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
|
||||
takes a file path; the program's only argument is that path, default
|
||||
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
||||
takes a file path; the program takes the exchange (`polymarket`, `kalshi`
|
||||
or `kalshi-demo`, required since sprint 06: one exchange per process), then
|
||||
that path, default `/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
||||
(rotation can't apply to stdout; a consumer that wants files rotates them).
|
||||
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
||||
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
||||
@@ -154,7 +161,8 @@ schema debate). As settled in sprints 04 and 05:
|
||||
- The **orchestrator** owns what is tracked (each market's two connections,
|
||||
the timers) and decides: placing, moving, splitting, and recycling one
|
||||
connection at a time, about hourly. It never blocks on HTTP.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- The two **fetchers** only page the exchange's market listing on request
|
||||
and stream what they read.
|
||||
- A **connection** reads, stamps the receive time, parses, and routes each
|
||||
event to the book thread of its market. Every symbol is on two connections,
|
||||
chosen independently. One silent for 30 s reconnects.
|
||||
@@ -170,6 +178,14 @@ Settled: OS threads, shared-nothing, no mutexes
|
||||
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
|
||||
[sprint 04](../roadmap/04.%20redundant%20connections.md).
|
||||
|
||||
The pipeline is the same for every exchange. What an exchange supplies
|
||||
([ADR 0004](adr/0004-exchange-seam.md)): the websocket layer of a connection
|
||||
(request, encoding of commands, decoding of frames, keepalive), the fetchers,
|
||||
and the rule a book thread applies to an event (duplicate, next, held, loss).
|
||||
On Polymarket that rule is the hash chain above; on Kalshi a change is the
|
||||
next one when its clock and rank pass the book's, and a held change doesn't
|
||||
exist.
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
||||
@@ -178,27 +194,33 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
|
||||
|
||||
- **Symbol** (from discovery): one order book, with its trading status and any
|
||||
price/tick metadata the exchange gives. On Polymarket that is a token, not a
|
||||
market: a market has two (Yes / No), each with its own, mirrored book.
|
||||
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
|
||||
produces so the core never sees exchange formats. Either a full snapshot or a
|
||||
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
|
||||
whatever the exchange offers for loss detection (a sequence number, a book
|
||||
hash), and payload (side/price/quantity, or the full book).
|
||||
market: a market has two (Yes / No), each with its own, mirrored book. On
|
||||
Kalshi it is the market: its bids for Yes are the bids, its bids for No at
|
||||
`q` the asks at `1 − q`.
|
||||
- **Event** (settled in sprint 06, in place of the `NormalizedEvent` first
|
||||
sketched): each exchange's own type. The core reads only the market it
|
||||
belongs to and whether it is a full book; the exchange's rule reads the
|
||||
rest (side, price, quantity, and what the exchange offers for loss
|
||||
detection: a book hash, a sequence number, a clock) and folds it into the
|
||||
symbol's book, setting a level's quantity (Polymarket) or moving it by a
|
||||
signed amount (Kalshi).
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
the last loss-detection marker seen and last exchange timestamp. It is fed
|
||||
by the symbol's two connections, first one wins.
|
||||
- **Connection** (one websocket): the set of symbols it carries. Live markets
|
||||
are placed on connections by `conditionId`, both tokens together;
|
||||
are placed on connections by market, all its symbols together;
|
||||
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
|
||||
live while active, not closed, with an order book and accepting orders).
|
||||
live while active, not closed, with an order book and accepting orders),
|
||||
Kalshi's its market listing (sprint 06: open, or opening within three
|
||||
hours, tracked ahead so the book is there from its first order).
|
||||
|
||||
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
||||
|
||||
The debate lives in the **NormalizedEvent** shape (both exchanges must map
|
||||
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
|
||||
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
|
||||
Redundancy was settled in sprint 04: no source is trusted, the first copy of
|
||||
each change wins ([ADR 0003](adr/0003-first-one-wins.md)). On Kalshi the
|
||||
first copy is recognised by clock and rank (sprint 06).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
None. (Loss detection on Kalshi was the last: settled in M6, by its
|
||||
per-connection sequence number.)
|
||||
Reference in new issue
Block a user