Archived
The orchestrator, the connection loop and the book thread's shell move to src/core/, generic over an Exchange and its loss rule (ADR 0004). Polymarket runs through it unchanged. Kalshi joins: a signed handshake (KALSHI_KEY_ID_FILE, KALSHI_PRIVATE_KEY_FILE), a reconnect on a skipped sequence number, first one wins by clock and rank, and its market listing with markets tracked from three hours before they open. Breaking: the exchange is now the first argument, `engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]`.
470 lines
27 KiB
Markdown
470 lines
27 KiB
Markdown
# 06. Generic core and Kalshi
|
||
|
||
**Status: done (2026-11-11)**
|
||
|
||
The collector does the hard things on one exchange. This sprint pulls the
|
||
exchange-agnostic core out of `src/polymarket/`, then adds Kalshi beside
|
||
Polymarket so both run through that core: Extension 3 of the brief, and the
|
||
exchange the brief names. The brief asks that only "the encoding/decoding on
|
||
the websocket layer" change per exchange; that holds for everything except
|
||
how a loss is detected and how a symbol's two connections are merged, which
|
||
differ in kind between the two, so each exchange also supplies that rule.
|
||
Design: [ADR 0004](../docs/adr/0004-exchange-seam.md),
|
||
[../docs/project.md](../docs/project.md) (pipeline, unchanged),
|
||
[ADR 0003](../docs/adr/0003-first-one-wins.md),
|
||
[../GLOSSARY.md](../GLOSSARY.md) (market, symbol).
|
||
|
||
## Kalshi, measured (2026-11-11)
|
||
|
||
From docs.kalshi.com, the public listing, and four recordings of the demo's
|
||
websocket with a throwaway recorder (`../kalshi-spike/`, outside the repo;
|
||
up to six connections, 200 to 240 s). The details go to `docs/notes.md`.
|
||
Production may differ where a number is the demo's.
|
||
|
||
- **The websocket needs a signed handshake; the listing doesn't.**
|
||
`GET /markets` answers 200 unauthenticated on production and the demo; an
|
||
unauthenticated websocket handshake gets 401 on both. Signed with `ring`
|
||
from the key files, the demo answered 101.
|
||
- **Size:** 177,966 open markets without combos (`mve_filter=exclude`) on
|
||
production, one book each. At most 200 connections per user (docs).
|
||
- **One book per market, bids only:** YES bids and NO bids. Prices are
|
||
dollar strings with 4 decimals, quantities strings with 2. With
|
||
`use_yes_price: true` a NO bid at `q` comes at `1 − q`, the YES ask it
|
||
is; the default (`false`, announced to flip) gives `q`.
|
||
- **A change is relative:** `delta_fp` is a signed change of a level's
|
||
quantity. Folding them gave the exchange's book every time: 1,352 books
|
||
asked for mid-stream (`get_snapshot`) all equalled ours, no level below
|
||
zero.
|
||
- **`seq` is contiguous per connection** across all its markets, counting
|
||
books, changes and the `ok` / `error` answers to commands (`subscribed`
|
||
has none): no gap in 343,000 messages on one connection.
|
||
- **`clock` is per market and the same on every connection, but several
|
||
changes share one**: 12% of clock values carried 2 to 4 changes, some
|
||
identical in content. Two connections on the same 26 changing markets
|
||
delivered the same changes in the same order (2,643 and 2,644, the
|
||
difference at the edges). A book's clock is that of the last change it
|
||
holds: none of 1,378 books was followed by a change of its own clock or
|
||
an older one. So a change is named by its clock and its rank among the
|
||
changes of that clock.
|
||
- **No limit met per connection:** 30,000 markets subscribed on one
|
||
connection, in one message or in 30 messages 20 ms apart, all books within
|
||
1.7 s, ~1,560 changes and 566 KB a second, no error. 52 `get_snapshot`
|
||
commands 3 s apart, none refused.
|
||
- **Adding markets** (a second `subscribe` or `add_markets`) sends their
|
||
books; `delete_markets` stops a market's changes. Each answer lists every
|
||
market of the subscription unless `skip_ticker_ack` is sent.
|
||
- **Kalshi pings only a connection it has sent nothing to for 10 s** (a
|
||
Ping frame, `heartbeat`): an idle connection got one every 10 s, a busy
|
||
one none.
|
||
- **A book has no time of its own**, only the frame's `sending_ts_ms`; a
|
||
change has `ts_ms`. Changes arrived 88 to 174 ms after it (median 90).
|
||
- **An unopened market can be subscribed:** it answers an empty book
|
||
(`clock: 0`, `market_id: ""`), and its changes flow from its first order
|
||
after the opening, with no new subscribe (two markets subscribed 107 s
|
||
ahead; a connection subscribing after the opening saw the same).
|
||
- **Markets are created ahead of their opening**, and the listing filters on
|
||
creation time, not opening time. Of the first 15,000 open markets listed
|
||
on production (not a random sample), 82% opened more than a minute after
|
||
creation and 19% more than 15 minutes after; at least 40,000 were
|
||
unopened, 755 of them opening within the hour. In 10 minutes, 352 markets
|
||
were created (45 still unopened) and 25 closed. None of the demo's 20,504
|
||
unopened markets was past its `open_time`: they open on time.
|
||
- **The demo** has its own hosts and keys and the same protocol; its books
|
||
are thin (of its first 60,000 open markets, 540 had bids on both sides,
|
||
by the research probe), but its bots keep ~1,500 changes a second going.
|
||
|
||
## The seam
|
||
|
||
Three parts ([ADR 0004](../docs/adr/0004-exchange-seam.md)):
|
||
|
||
```
|
||
src/core/ orchestrator, connection, book thread, the Exchange trait
|
||
src/polymarket/ messages, Gamma, the hash rule
|
||
src/kalshi/ messages, signing, the market listing, the sequence and clock rule
|
||
```
|
||
|
||
**Shared (`src/core/`)**, moved from `src/polymarket/` as it is:
|
||
|
||
- the orchestrator (`orchestrator.rs`): tracking, placing, balancing,
|
||
splitting, recycling;
|
||
- the connection loop (`socket.rs`): connect with a timeout, read, stamp,
|
||
route, reconnect, silent after 30 s, recycle, clean close;
|
||
- the book thread's shell (from `feed.rs`): markets and symbols, connections
|
||
up and down, the minute's Snapshots, load reports, the quiet watch, moves,
|
||
the minute line;
|
||
- `main.rs`'s writer, `book.rs`, `reconcile`, `cause.rs` (already neutral:
|
||
`cause::parse` looks for `Snapshot of` anywhere in a line).
|
||
|
||
**Per exchange, one trait (`Exchange`)**, chosen at compile time (the
|
||
orchestrator, connections and book threads are generic over it; `main`
|
||
picks by name):
|
||
|
||
- the websocket request (URL and headers: Kalshi signs it) and the messages
|
||
sent on connect;
|
||
- encoding `Subscribe`, `Unsubscribe` and `Resubscribe` into frames, with
|
||
the connection's own state (Kalshi's subscription id);
|
||
- decoding a frame into events, each with the market that routes it, and
|
||
saying when the connection itself lost something (Kalshi's `seq`);
|
||
- the keepalive we send, if any (Polymarket's `PING` text; none on Kalshi,
|
||
whose Ping frames tungstenite answers by itself);
|
||
- the fetcher: Gamma's `Request` / `Finding` interface
|
||
(`gamma.rs:26-47`) is kept as the core's, and each exchange implements it.
|
||
|
||
Not per exchange after all (see Decisions): the close that means "too slow"
|
||
(1013, websocket's own "try again later"), and the per-connection numbers.
|
||
|
||
**Per exchange, inside the book thread (its rule):** what a symbol's book
|
||
does with an event from one of its connections: a duplicate, the next change,
|
||
held, a loss, a resync to ask for. The core hands it the symbol's book, which
|
||
connections are up, and a place to put effects; the rule owns its own
|
||
per-symbol state.
|
||
|
||
**What the core understands of an event**: the market it belongs to, which
|
||
routes it, and whether it is a full book, which ends a move. The event is
|
||
the exchange's own type, handed to its rule unread, loss markers (hash,
|
||
`seq`, `clock`) included. The rule folds it into the symbol's book, the
|
||
core's: a level is set to a quantity (Polymarket) or moved by a signed
|
||
amount (Kalshi), and a Delta always carries the absolute quantity.
|
||
|
||
**A market** is `id`, its symbols, and the exchange's own details
|
||
(Polymarket: `min_order_size`, `neg_risk`; Kalshi: none). Its book thread
|
||
comes from a hash of the id (`std`'s `DefaultHasher`; it parsed the
|
||
conditionId's hex digits, which a ticker doesn't have).
|
||
|
||
## Extracting the core
|
||
|
||
A move, not a rewrite: Polymarket's rules, constants, log lines and tests
|
||
stay as they are, and every existing test stays green at each step.
|
||
|
||
The cut doesn't follow today's functions. In `feed.rs`, `Symbol` holds both
|
||
halves (shell: `book`, `connections`, `watches`, `quiet_on`, `last_written`;
|
||
rule: `state`, `recent_changes`, `held`, the hash inputs), and three
|
||
functions interleave them:
|
||
|
||
- `tick` (`feed.rs:826`): the shell's minute loop (loads, quiet judging,
|
||
Snapshots) calls the rule per symbol in an order that matters (count old
|
||
copies, "may this symbol be judged quiet", "does it have a book", ask
|
||
again for a fresh book).
|
||
- `on_book` (`feed.rs:1046`): ending a move and spotting a reconnect's
|
||
opening book are the shell's; judging the book is the rule's, and it
|
||
writes Snapshots and cause lines.
|
||
- `on_price_change` (`feed.rs:1248`): counting the change for the quiet
|
||
watch is the shell's, the rest the rule's.
|
||
|
||
So `Symbol` becomes the shell's fields plus the rule's state, and the rule
|
||
gets named hooks: an event, a connection down, a subscription leaving, the
|
||
ceilings' timer, and the minute's questions above. The minute line keeps its
|
||
text: the shell prints its counts, the rule adds its own.
|
||
|
||
The tests move with the code they exercise. Polymarket's 43 `feed.rs` tests
|
||
drive the shell through hash-correct frames, so they stay Polymarket's and
|
||
keep covering the shell; Kalshi's replay tests cover it a second time. No
|
||
stand-in rule is written for the shell alone.
|
||
|
||
## Kalshi
|
||
|
||
**Symbol.** The market, named by its ticker. YES bids are bids; a NO bid at
|
||
`q` is an ask at `1 − q`, exact in fixed point.
|
||
|
||
**Key.** `KALSHI_KEY_ID_FILE` and `KALSHI_PRIVATE_KEY_FILE` name two files
|
||
(on the server, in `/run/secrets`). Read once at startup; a missing or
|
||
unreadable one stops the run with the variable's name. Ed25519 only (Kalshi's
|
||
default). Each connect signs `<ms since epoch>GET/trade-api/ws/v2` and sends
|
||
`KALSHI-ACCESS-KEY`, `KALSHI-ACCESS-TIMESTAMP`, `KALSHI-ACCESS-SIGNATURE`
|
||
(base64). The key is never logged. The listing is read unauthenticated.
|
||
|
||
**Command line.** `engineering-challenge <polymarket|kalshi|kalshi-demo>
|
||
[OUTPUT]`, the exchange required; one exchange per process. `kalshi` and
|
||
`kalshi-demo` differ only by their hosts.
|
||
|
||
**A skipped `seq`.** The connection thread checks it (a book thread sees only
|
||
its own markets' share). On a gap the connection closes and reconnects, like
|
||
a silent one: every book it carries comes fresh, and each symbol's other
|
||
connection covers meanwhile.
|
||
|
||
**Merging two connections.** First one wins, by clock and rank. A symbol's
|
||
book stands at a position: the clock of its last change and how many changes
|
||
of that clock it holds. Each connection has its own position for the symbol,
|
||
counted the same way from the book it delivered. A change is applied and
|
||
written when its connection's position passes the book's, and is a duplicate
|
||
otherwise. A connection counts for a symbol only once it has delivered that
|
||
symbol's book since it came up; before that its changes are dropped. Nothing
|
||
is held: two connections deliver a market's changes in the same order, so
|
||
the first copy of each is always the next one.
|
||
|
||
**A fresh book** (on subscribe, reconnect or resubscribe) for a symbol that
|
||
has one: older than ours, it only sets its connection's position. At or past
|
||
our clock and equal to ours, the same. At or past our clock and different,
|
||
it replaces ours and is written as a Snapshot, with its cause line: the
|
||
other connection is behind or lost something, and we can't tell which.
|
||
|
||
**A change ahead of the book.** After a resync's answer the book can stand
|
||
behind what another connection has delivered. That connection's next change
|
||
would skip the changes between, so a change is taken only when its
|
||
connection's position before it was at or behind the book: the stream that
|
||
answered carries the book forward, and the other counts again once the book
|
||
reaches it. (From the review: without it, the changes between were dropped
|
||
as duplicates, with no loss logged.)
|
||
|
||
**A book that can't be right.** A level driven below zero by a change means
|
||
our book is wrong: the symbol is resynced (`Resubscribe` on one connection)
|
||
and the fresh book written as a Snapshot, with its cause line.
|
||
|
||
**Quiet, silent, recycle, split, balance.** The core's, unchanged. Silent is
|
||
30 s without a frame, Kalshi's Pings included (it pings a connection it has
|
||
nothing else to send to). `Resubscribe` is `delete_markets` then a
|
||
`subscribe`, which adds to the connection's one subscription as
|
||
`add_markets` would and needs no subscription id. Nothing on Kalshi answers
|
||
to Polymarket's 1013, so nothing splits. Errors 10 and 25 (a channel error,
|
||
a subscription buffer overflow: terminal, by Kalshi's docs) reconnect the
|
||
connection like a skipped `seq`; other errors are logged. At the minute, a
|
||
symbol no connection delivers (no book yet, or none after a reconnect) is
|
||
asked for again, and gets no Snapshot until one delivers it.
|
||
|
||
**Numbers.** Polymarket's: 5,000 symbols and 300 KB/s per connection, which
|
||
puts production's 178,000 markets, twice, on about 80 connections, under the
|
||
cap of 200. Subscribes go in messages of 500 tickers with `use_yes_price:
|
||
true` and `skip_ticker_ack: true`. A Snapshot's exchange time is its
|
||
frame's `sending_ts_ms`.
|
||
|
||
**Discovery.** The fetcher scans open and unopened markets at startup and
|
||
hourly (`status=open` / `status=unopened`, `mve_filter=exclude`, pages of
|
||
1,000). Each minute it asks for the markets created since the last check
|
||
(`min_created_ts`, which returns open and unopened ones alike: tried) and
|
||
those closed since (`status=closed&min_close_ts`: tried). A market is
|
||
tracked while open, or unopened with an `open_time` at most three hours
|
||
ahead: its book is subscribed, empty, before its first order. Three hours is
|
||
three hourly rescans, so every market is reached by a scan before it opens
|
||
even when one scan fails, and the fetcher keeps no memory of opening times.
|
||
A scan pages the unopened markets first: a market that opens meanwhile is
|
||
then on one list at least (open first, it could be on neither and be
|
||
removed). The core sees the same findings as from Gamma.
|
||
|
||
## Decisions
|
||
|
||
- **The seam has three parts; each exchange supplies its loss rule**
|
||
([ADR 0004](../docs/adr/0004-exchange-seam.md)). *Rejected:* decode only,
|
||
as the brief's sentence reads (the core would hold both loss rules and a
|
||
third exchange would mean editing it); a whole book thread per exchange
|
||
(minute Snapshots, the quiet watch and moves written twice).
|
||
- **A Kalshi symbol is the market, asks derived from NO bids.** The output
|
||
reads like any order book. *Rejected:* two bid-only symbols per market
|
||
(the reader must know Kalshi's rule to find a spread).
|
||
- **The core reads only an event's market and whether it is a book; the
|
||
rule folds it into the core's book** (supersedes "the core's event is a
|
||
book or one change, set or moved": with each exchange supplying its rule,
|
||
a normalized event in the core would have no reader, the shell doing
|
||
nothing with a change but hand it on). *Rejected:* that normalized type,
|
||
both rules converting into it (more code for the same behaviour); a common
|
||
marker field (a hash and a sequence number have nothing in common).
|
||
- **A skipped `seq` reconnects the connection.** Both ways fetch every book
|
||
again; this one reuses sprint 05's path, and the other connection covers
|
||
the extra second. *Rejected:* `get_snapshot` for each market on the open
|
||
connection (a second recovery path, under an unpublished command limit).
|
||
- **First one wins by clock and rank.** The recording decided: the clock
|
||
alone can't name a change (12% of clock values carry several), the rank
|
||
within it can, since both connections deliver the same order. Simulated on
|
||
two recorded connections, one with a 5 s hole on five markets: 2,271
|
||
changes applied, 2,212 duplicates, 1,434 books compared at our clock, none
|
||
different. *Rejected:* the clock alone (drops the second change of a
|
||
clock); follow one connection and switch (the fallback, not needed: it
|
||
keeps the slower receive time and a Snapshot per switch).
|
||
- **A fresh book at or past our clock that differs replaces ours, with a
|
||
Snapshot.** The state is right at once. *Rejected:* holding the new
|
||
connection's changes until the other catches up (Polymarket's held
|
||
changes again, for a few ms after a reconnect).
|
||
- **A level below zero resyncs the symbol.** With relative changes it is the
|
||
one sign of a wrong book that `seq` doesn't give.
|
||
- **The rule was recorded before the sprint was agreed.** A rule taken from
|
||
the docs alone is what went wrong on Polymarket (CLAUDE.md, "Live data
|
||
beats the docs"); here the docs' "clock" was not enough by itself.
|
||
- **`use_yes_price: true`, sent explicitly.** It is the announced default,
|
||
and a NO bid then arrives as the ask it is. *Rejected:* leaving the
|
||
default (it is announced to flip, which would mirror every ask silently).
|
||
- **Polymarket's per-connection numbers on Kalshi.** No limit was met at
|
||
30,000 markets on the demo, so the cap of 200 connections doesn't bind;
|
||
one set of numbers is less to explain. Production's limits are untried.
|
||
- **The exchange is a required first argument; one exchange per process.**
|
||
The output has no exchange field, and one exchange failing doesn't stop
|
||
the other. *Rejected:* a default (Kalshi is the brief's, Polymarket the
|
||
one needing no key); both in one process.
|
||
- **The key comes as two file paths in the environment, Ed25519 only, signed
|
||
with `ring`.** Paths keep the secret out of the shell. `ring` is already
|
||
our TLS crypto; tried on a throwaway key: an OpenSSL Ed25519 PEM parsed
|
||
(`rustls-pki-types`, `Ed25519KeyPair::from_pkcs8_maybe_unchecked`) and
|
||
`openssl` verified the signature. `ring`, `rustls-pki-types` and `base64`
|
||
are already in `Cargo.lock`. *Rejected:* `ed25519-dalek` (a new
|
||
dependency for the same job); RSA keys too (a second signing path nobody
|
||
here uses).
|
||
- **An unopened market is tracked from three hours before its opening**
|
||
(supersedes "the fetcher remembers each opening time and reports the
|
||
market a minute before it", which superseded "ask for it by ticker once
|
||
its opening time has passed"). The two fetchers are separate threads and
|
||
the rescan's would have to hand the opening times to the minute check's;
|
||
a window over the rescan interval needs no memory at all, for a few
|
||
thousand empty books more (production: 755 markets opening within an
|
||
hour). Three hours rather than two, from the review: a failed rescan is
|
||
retried only an hour later, and two hours left no margin for one. *Rejected:* Polymarket's newest-created check alone (19% of
|
||
markets open more than 15 minutes after creation, so they would wait for
|
||
the hourly scan); subscribing every unopened market (tens of thousands of
|
||
empty books); the `market_lifecycle_v2` channel (no event at opening, by
|
||
the docs).
|
||
- **The 1013 close and the per-connection numbers stay in the core**
|
||
(supersedes "which close means too slow" and "its numbers" in the
|
||
`Exchange` trait). 1013 is the websocket's own "try again later", which
|
||
Kalshi never sent; Kalshi uses Polymarket's numbers (decided above). A
|
||
trait item each would be an abstraction with one value.
|
||
- **The extraction changes no Polymarket behaviour.** A change inside a move
|
||
of 3,300 lines can't be reviewed. One exception, with no effect on the
|
||
output: a market's book thread comes from a hash of its id.
|
||
- **No stand-in rule to test the shell alone.** Two real rules exercise it.
|
||
*Rejected:* a trivial test rule (a third implementation to maintain).
|
||
|
||
## Tests
|
||
|
||
- Every existing test, moved with its code, green and unchanged in what it
|
||
asserts.
|
||
- Kalshi replay: the one- and two-connection recordings folded into books;
|
||
the output reconciles; a NO bid becomes the right ask; relative changes
|
||
give the recorded books.
|
||
- Each Kalshi rule (duplicate by clock and rank, several changes of one
|
||
clock all applied, a connection not counted before its book, a fresh book
|
||
older / equal / different, a level below zero) has a test that fails with
|
||
the rule removed: checked by removing it.
|
||
- Signing: a fixed test key and string give a known signature; a missing
|
||
file, an unreadable PEM and an RSA key each stop with a clear error.
|
||
- The connection against a local server: the three headers arrive, the
|
||
server's Ping gets its Pong, a skipped `seq` makes it reconnect and tells
|
||
the book threads it went down.
|
||
- The fetcher on recorded listing pages, served by a local listing:
|
||
unopened then open scanned; an unopened market tracked from three hours
|
||
before its opening time and not before; created and closed ones reported;
|
||
a closed one not tracked.
|
||
- Live, each reconciled against its log: Polymarket 10 minutes; Kalshi 10
|
||
minutes on the demo, and on production if a key exists.
|
||
|
||
## Not in this sprint
|
||
|
||
- Combo (MVE) markets.
|
||
- The `market_lifecycle_v2` channel.
|
||
- Conflated updates (`update_interval_ms`).
|
||
- Thursday's maintenance (3:00 to 5:00 AM ET) beyond the ordinary reconnect.
|
||
- `get_snapshot`, RSA keys, authenticated REST.
|
||
- Any change to Polymarket's rules.
|
||
- Latency work (M7); the README (M8).
|
||
|
||
## Done when
|
||
|
||
- `engineering-challenge polymarket` and `engineering-challenge kalshi-demo`
|
||
run through the same `src/core/`; `src/core/` names no exchange.
|
||
- Polymarket, 10 minutes live: reconciled with every mismatch explained, and
|
||
losses per million changes no higher than sprint 05's (26.5 and 28.4).
|
||
- Kalshi demo, 10 minutes live: reconciled with every mismatch explained;
|
||
markets opened and closed during the run were added and removed; SIGINT
|
||
closes the connections and the output.
|
||
- Kalshi production: the same run if a key exists, "not run" otherwise.
|
||
- `docs/notes.md` has Kalshi's protocol facts as measured.
|
||
- `scripts/ci.sh` is green.
|
||
|
||
## Outcome
|
||
|
||
Both exchanges run through one core. `src/core/` holds the orchestrator, the
|
||
connection loop and the book thread's shell, generic over an `Exchange`
|
||
(request, session, fetcher) and its `Rule`; it names no exchange.
|
||
Polymarket moved behind it with its rules, constants and log lines as they
|
||
were: its 79 tests moved with it and pass unchanged. Kalshi joined with a
|
||
signed handshake, the `seq` check in the connection, first one wins by clock
|
||
and rank, and a fetcher on its listing. `engineering-challenge` now takes
|
||
the exchange first.
|
||
|
||
**Live runs**, each reconciled with every mismatch explained by its log
|
||
line, no orphan Delta:
|
||
|
||
| Run | Markets, connections | Changes written | Mismatches | Peak memory, threads |
|
||
|---|---|---|---|---|
|
||
| Polymarket, 10 min, before the review's fixes | 200,052 on 179 | 18,971,851 | 461 losses (24.3 / M), 2,839 gaps | 1.93 GB, 201 |
|
||
| Kalshi demo, 10 min, before | 246,349 on 110 | 2.4 M | 16, all "loss" | 0.67 GB, 131 |
|
||
| Kalshi demo, 21 min across 20:00 UTC, before | 245,383 on 110 | 5.6 M | 63, all "loss" | 0.68 GB, 131 |
|
||
| Kalshi demo, 10 min, after | 244,903 on 109 | 2,470,013 | 30, all "loss" | 0.66 GB, 130 |
|
||
| Polymarket, 5 min, after | 201,124 on 180 | 4,842,531 | 130 losses (26.8 / M), 2,014 gaps | 1.60 GB, 201 |
|
||
| Kalshi production | not run: no production key | | | |
|
||
|
||
Polymarket's losses are no higher than sprint 05's (26.5 and 28.4 per
|
||
million). After the review's fixes it changed only by renames and the shared
|
||
retry helper, so its rerun was 5 minutes, not 10. On Kalshi every change
|
||
came twice (2,464,467 duplicates for 2,470,013 written), none below zero,
|
||
none ahead of the book, no skipped `seq` in any run. The 21-minute run
|
||
removed 3,273 markets that closed (3,271 at 20:00, checked against the
|
||
listing) and added 1,382; SIGINT closed the output each time.
|
||
|
||
**Kalshi's "losses" are not known to be losses.** They are fresh books at or
|
||
past ours that differ, the case we agreed to write as a Snapshot without
|
||
being able to tell why. 54 of the 63 in the long run came at 20:00:01 to
|
||
20:00:04 for markets closing at 20:00: Kalshi sends a cleared book at the
|
||
close, without the changes that lead to it. The rest came at startup or
|
||
with a recycle, when one connection's book runs ahead of the other's
|
||
stream. The Snapshots are right; telling the three apart is in the roadmap's
|
||
features.
|
||
|
||
**Fixed at the close**, from the code review, each with a test that failed
|
||
first:
|
||
|
||
- After a resync's answer older than what the other connection had
|
||
delivered, the changes between were dropped as duplicates with no loss
|
||
logged: a change ahead of the book now waits for it.
|
||
- A market opening during a rescan could be on neither list and be removed
|
||
for up to an hour: unopened markets are paged first.
|
||
- A key file flattened onto one line made the error print the key's bytes
|
||
(the PEM reader's error quotes its input): the error names the file only.
|
||
- A symbol whose book didn't come back after both connections reconnected
|
||
was never asked for again: it is, at the minute.
|
||
- A first book asked from one connection and delivered by the other was
|
||
logged as a loss.
|
||
|
||
After them, each of Kalshi's 21 rules was removed in turn and a test failed
|
||
every time.
|
||
|
||
**Diverged from the plan**, each in Decisions or its section: an unopened
|
||
market is tracked from three hours ahead with nothing remembered (not "a
|
||
minute before", which needed the two fetcher threads to share opening
|
||
times); the core reads only an event's market and whether it is a book, the
|
||
rule folding it into the core's book (not a normalized event in the core;
|
||
agreed at the close); the 1013 close and the per-connection numbers stayed
|
||
in the core; `Resubscribe` uses `subscribe`, not `add_markets`; errors 10
|
||
and 25 reconnect. One Polymarket log text changed: a refused handshake reads
|
||
"handshake refused: <status>".
|
||
|
||
**Left out**, and where it went: Kalshi production (no key; the command is
|
||
`engineering-challenge kalshi` with production's two files); telling a
|
||
close's book and a book ahead from a real loss, retrying a failed rescan
|
||
sooner than the next hour, and sharing the two rules' resync bookkeeping
|
||
(Possible features). Not covered by a test: the orchestrator's loop, as in
|
||
sprints 04 and 05; `Key::from_env`; the 429s of a hundred handshakes at once
|
||
(seen live, each retried); the "ahead of the book" rule live (0 in 10
|
||
minutes: it needs a resync, and none was needed).
|
||
|
||
### Retrospective
|
||
|
||
- **Went well:** recording the demo before agreeing the sprint overturned
|
||
the rule on paper (the clock alone can't name a change) and answered the
|
||
three open points in an hour. Moving Polymarket's rule by script, with
|
||
every cut asserted, kept 3,300 lines reviewable: the reviewer found no
|
||
behaviour change. The reviewer's adversarial sequences on two connections
|
||
found what the tests didn't.
|
||
- **Went wrong:** three bugs in Kalshi's rule passed my own tests and the
|
||
mutation check, which proves the rules written, not the ones missing: I
|
||
never asked what happens when the book goes backwards. I changed an agreed
|
||
point (the minute-before opening) while coding and reported it after,
|
||
instead of asking first; and I didn't notice the core's event had drifted
|
||
from the agreed one until the review. A "flaky" test cost a detour: my
|
||
mutation script restored files with an older timestamp and cargo kept the
|
||
mutant's build. The live runs came before the review, so its fixes meant
|
||
running them again.
|
||
- **Tested enough?** The rules are, by removal. The orchestrator's loop
|
||
still isn't, production Kalshi not at all, and the fetchers only against a
|
||
local listing with one page per status.
|
||
- **Next time:** for each piece of state a rule keeps, ask what makes it go
|
||
backwards or stay empty, and write that test. Ask before diverging from an
|
||
agreed point, even a small one. Review before the live runs that close a
|
||
sprint.
|