Archived
feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
This commit is contained in:
24 files changed
+8984
-1214
No files matched your search
@@ -12,17 +12,22 @@ outside this repo.
|
||||
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
|
||||
agreed plan. Read the current sprint file before writing code.
|
||||
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
|
||||
0002: lanes and the orchestrator).
|
||||
0002: the orchestrator, its lanes superseded; 0003: two connections per
|
||||
symbol, first one wins).
|
||||
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
|
||||
loss detection, message shapes). Read before touching exchange parsing.
|
||||
- `GLOSSARY.md`: the terms (symbol, resync, loss, subscription gap, lane,
|
||||
load, Snapshot, Delta).
|
||||
- `src/polymarket/orchestrator.rs`: what is tracked, the lanes and their
|
||||
balancing; `src/polymarket/gamma.rs`: the fetchers (discovery);
|
||||
`src/polymarket/feed.rs`: a lane's book thread and its loss rules (the
|
||||
subtle part); `src/polymarket/socket.rs`: a lane's connection;
|
||||
`src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
|
||||
check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay.
|
||||
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
|
||||
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
|
||||
Snapshot, Delta).
|
||||
- `src/polymarket/orchestrator.rs`: what is tracked, which connections carry
|
||||
each market, and their balancing; `src/polymarket/gamma.rs`: the fetchers
|
||||
(discovery); `src/polymarket/socket.rs`: a connection (read, stamp, decode,
|
||||
route); `src/polymarket/wire.rs`: the exchange's messages;
|
||||
`src/polymarket/feed.rs`: a book thread and its loss rules (the subtle
|
||||
part); `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
|
||||
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
|
||||
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feed, on one and two
|
||||
connections, that the tests replay.
|
||||
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
|
||||
output stays readable by the template's `AlignedSbeReader`. Our code is the
|
||||
rest of `src/`.
|
||||
@@ -57,17 +62,21 @@ and let the user settle it. Claude owns the data model.
|
||||
A 403 is not DNS (see `docs/notes.md`, REST from scripts).
|
||||
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
|
||||
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
|
||||
- **The collector runs until SIGINT, and only SIGINT flushes its output.**
|
||||
Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten.
|
||||
Bound a run with GNU `timeout -s INT`, then check the output:
|
||||
- **Only a clean stop flushes the output**: SIGINT, SIGTERM or SIGHUP. SIGKILL
|
||||
or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
|
||||
`timeout -s INT`, then check the output against the log:
|
||||
`timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log;
|
||||
./target/release/reconcile out.sbe`. `reconcile` counts subscription gaps
|
||||
as mismatches too: compare its list with the log's "loss", "subscription
|
||||
gap" and "first book after a move" lines.
|
||||
./target/release/reconcile out.sbe run.log`. With the log, `reconcile`
|
||||
pairs each mismatch with its cause line and fails only on one left
|
||||
unexplained.
|
||||
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
|
||||
"capacity scan"` matches the waiting shell's own command line and never
|
||||
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
|
||||
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
|
||||
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
|
||||
2026-11-11): a starved collector falls behind and gets closed with 1013.
|
||||
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
|
||||
alternate the builds in short rounds rather than running them back to back.
|
||||
- **Live data beats the docs**: Polymarket's market channel has rules its
|
||||
docs don't state (`docs/notes.md`); check a protocol assumption on a
|
||||
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
|
||||
|
||||
Reference in new issue
Block a user