feat(polymarket)!: two connections per market, first one wins

Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
This commit is contained in:
pascal committed 2026-11-11 15:18:48 +00:00
1 parent 8114a51dc2
commit 02747b882f
24 files changed
+8984 -1214

No files matched your search

+24 -15
View File
@@ -12,17 +12,22 @@ outside this repo.
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
agreed plan. Read the current sprint file before writing code.
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
0002: lanes and the orchestrator).
0002: the orchestrator, its lanes superseded; 0003: two connections per
symbol, first one wins).
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
loss detection, message shapes). Read before touching exchange parsing.
- `GLOSSARY.md`: the terms (symbol, resync, loss, subscription gap, lane,
load, Snapshot, Delta).
- `src/polymarket/orchestrator.rs`: what is tracked, the lanes and their
balancing; `src/polymarket/gamma.rs`: the fetchers (discovery);
`src/polymarket/feed.rs`: a lane's book thread and its loss rules (the
subtle part); `src/polymarket/socket.rs`: a lane's connection;
`src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay.
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
Snapshot, Delta).
- `src/polymarket/orchestrator.rs`: what is tracked, which connections carry
each market, and their balancing; `src/polymarket/gamma.rs`: the fetchers
(discovery); `src/polymarket/socket.rs`: a connection (read, stamp, decode,
route); `src/polymarket/wire.rs`: the exchange's messages;
`src/polymarket/feed.rs`: a book thread and its loss rules (the subtle
part); `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feed, on one and two
connections, that the tests replay.
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
output stays readable by the template's `AlignedSbeReader`. Our code is the
rest of `src/`.
@@ -57,17 +62,21 @@ and let the user settle it. Claude owns the data model.
A 403 is not DNS (see `docs/notes.md`, REST from scripts).
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
- **The collector runs until SIGINT, and only SIGINT flushes its output.**
Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten.
Bound a run with GNU `timeout -s INT`, then check the output:
- **Only a clean stop flushes the output**: SIGINT, SIGTERM or SIGHUP. SIGKILL
or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
`timeout -s INT`, then check the output against the log:
`timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log;
./target/release/reconcile out.sbe`. `reconcile` counts subscription gaps
as mismatches too: compare its list with the log's "loss", "subscription
gap" and "first book after a move" lines.
./target/release/reconcile out.sbe run.log`. With the log, `reconcile`
pairs each mismatch with its cause line and fails only on one left
unexplained.
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
"capacity scan"` matches the waiting shell's own command line and never
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
2026-11-11): a starved collector falls behind and gets closed with 1013.
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
alternate the builds in short rounds rather than running them back to back.
- **Live data beats the docs**: Polymarket's market channel has rules its
docs don't state (`docs/notes.md`); check a protocol assumption on a
recording of the real feed (`tests/fixtures/record.py`) before relying on it.