feat(polymarket)!: two connections per market, first one wins

Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
This commit is contained in:
pascal committed 2026-11-11 15:18:48 +00:00
1 parent 8114a51dc2
commit 02747b882f
24 files changed
+8984 -1214

No files matched your search

+10 -8
View File
@@ -1,7 +1,7 @@
//! The collector (roadmap/03): an orchestrator keeps the tracked markets equal
//! to Gamma's live ones and spreads them over lanes, each a connection (read,
//! stamp) and a book thread (parse, fold, check, resync), all writing through
//! one writer (SBE).
//! The collector (roadmap/04): an orchestrator keeps the tracked markets equal
//! to Gamma's live ones and places them on connections (read, stamp, decode,
//! route), which feed the book threads (fold, check, resync), all writing
//! through one writer (SBE).
//! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs
//! go to stderr.
@@ -43,9 +43,10 @@ fn main() -> anyhow::Result<()> {
writer_stop.store(true, Ordering::Relaxed);
written
});
// SIGINT stops the orchestrator and the connections; each book thread
// stops when its connection does, and the writer flushes once they all
// have, even after a panic upstream (the run still fails then).
// SIGINT (and SIGTERM, SIGHUP) stops the orchestrator and the
// connections; the book threads stop once every connection has, and the
// writer flushes once they all have, even after a panic upstream (the run
// still fails then).
let failed = polymarket::orchestrator::run(outputs_tx, &stop);
let written = writer
.join()
@@ -60,7 +61,8 @@ fn main() -> anyhow::Result<()> {
/// thread is gone, then flushes the last chunk.
// ponytail: the chunk being filled lives only in memory until it is full, so
// a hard crash loses it; flushing more often pads every flush to 64 MiB.
// Taken up in M4.
// Kept as a ceiling in M4 (roadmap/04): writing to a file of our own and
// copying it into chunks would lift it.
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
let mut writer = AlignedSbeWriter::create(path, CHUNK)?;
for output in outputs {