feat(polymarket): track every live market over load-balanced lanes

An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
pascal committed 2026-11-11 06:28:25 +00:00
1 parent adb7fee8d8
commit 8114a51dc2
18 files changed
+2826 -320

No files matched your search

+25 -18
View File
@@ -11,13 +11,17 @@ outside this repo.
reasons behind the design principles. Read before any design discussion.
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
agreed plan. Read the current sprint file before writing code.
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing).
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
0002: lanes and the orchestrator).
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
loss detection, message shapes). Read before touching exchange parsing.
- `GLOSSARY.md`: the terms (symbol, resync, loss, resync gap, Snapshot, Delta).
- `src/polymarket/feed.rs`: the book thread and its loss rules (the subtle
part); `src/polymarket/socket.rs`: the socket thread; `src/book.rs`: the
book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
- `GLOSSARY.md`: the terms (symbol, resync, loss, subscription gap, lane,
load, Snapshot, Delta).
- `src/polymarket/orchestrator.rs`: what is tracked, the lanes and their
balancing; `src/polymarket/gamma.rs`: the fetchers (discovery);
`src/polymarket/feed.rs`: a lane's book thread and its loss rules (the
subtle part); `src/polymarket/socket.rs`: a lane's connection;
`src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay.
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
output stays readable by the template's `AlignedSbeReader`. Our code is the
@@ -45,22 +49,25 @@ and let the user settle it. Claude owns the data model.
## Environment traps
- **Polymarket and Kalshi are DNS-blocked on the dev network**: the local
resolver answers `195.186.4.210` / `2001:4d98:4:b10c::1` (a Swisscom block
page). The user pins the hosts with `sudo ../host.sh on` (workspace root,
outside this repo; `off` removes them). A timeout or the block address again
means the CDN addresses rotated: ask the user to rerun it. A host not yet in
`host.sh`'s `HOSTS` list fails the same way: add it there and ask the user to
rerun it. A 403 is not DNS (see `docs/notes.md`, REST from scripts).
- **DNS: the exchanges resolve normally on the server** (a remote-claude
workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver
answers `195.186.4.210` / `2001:4d98:4:b10c::1`, a Swisscom block page);
there the user pins the hosts with `sudo ../host.sh on` (workspace root,
outside this repo; `off` removes them), rerun when the CDN addresses rotate.
A 403 is not DNS (see `docs/notes.md`, REST from scripts).
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
- **The collector runs until SIGINT, and only SIGINT flushes its output.**
macOS has no `timeout`, and a `perl alarm` kill (SIGALRM) leaves the last
64 MiB chunk unwritten. Bound a run with SIGINT, then check the output:
`./target/release/engineering-challenge out.sbe 2> run.log & pid=$!;
sleep 600; kill -INT $pid; wait $pid; ./target/release/reconcile out.sbe`.
`reconcile` counts resync gaps as mismatches too: compare its list with the
log's "loss" and "resync gap" lines.
Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten.
Bound a run with GNU `timeout -s INT`, then check the output:
`timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log;
./target/release/reconcile out.sbe`. `reconcile` counts subscription gaps
as mismatches too: compare its list with the log's "loss", "subscription
gap" and "first book after a move" lines.
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
"capacity scan"` matches the waiting shell's own command line and never
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
- **Live data beats the docs**: Polymarket's market channel has rules its
docs don't state (`docs/notes.md`); check a protocol assumption on a
recording of the real feed (`tests/fixtures/record.py`) before relying on it.