feat(polymarket): track every live market over load-balanced lanes

An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
pascal committed 2026-11-11 06:28:25 +00:00
1 parent adb7fee8d8
commit 8114a51dc2
18 files changed
+2826 -320

No files matched your search

+20 -27
View File
@@ -1,5 +1,7 @@
//! The collector: three threads joined by channels (roadmap/02):
//! socket (read, stamp) → book (parse, fold, check, resync) → writer (SBE).
//! The collector (roadmap/03): an orchestrator keeps the tracked markets equal
//! to Gamma's live ones and spreads them over lanes, each a connection (read,
//! stamp) and a book thread (parse, fold, check, resync), all writing through
//! one writer (SBE).
//! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs
//! go to stderr.
@@ -20,54 +22,45 @@ macro_rules! log {
mod polymarket;
use engineering_challenge::book::Output;
use polymarket::feed::Feed;
/// `AlignedSbeReader::open` reads nothing else.
const CHUNK: usize = 64 << 20;
const MARKETS: u32 = 20;
fn main() -> anyhow::Result<()> {
let output = std::env::args()
.nth(1)
.unwrap_or_else(|| "/dev/stdout".to_string());
let markets = polymarket::top_markets(MARKETS)?;
let feed = Feed::new(&markets);
let tokens = feed.tokens();
let stop = Arc::new(AtomicBool::new(false));
let on_sigint = Arc::clone(&stop);
ctrlc::set_handler(move || on_sigint.store(true, Ordering::Relaxed))?;
let (frames_tx, frames_rx) = unbounded();
let (resyncs_tx, resyncs_rx) = unbounded();
let (outputs_tx, outputs_rx) = unbounded();
// Each stage stops when the one before it does, so SIGINT only reaches the
// socket thread; the writer flushes whatever came before.
let socket =
thread::spawn(move || polymarket::socket::run(&tokens, &frames_tx, &resyncs_rx, &stop));
let book =
thread::spawn(move || polymarket::feed::run(feed, frames_rx, outputs_tx, resyncs_tx));
let writer = thread::spawn(move || write(&output, outputs_rx));
// Every thread is joined, so the writer flushes even after a panic
// upstream; the run still fails then.
let socket_panicked = socket.join().is_err();
let book_panicked = book.join().is_err();
let writer_stop = Arc::clone(&stop);
let writer = thread::spawn(move || {
let written = write(&output, outputs_rx);
// A writer that failed stops the rest: nothing it reads is kept.
writer_stop.store(true, Ordering::Relaxed);
written
});
// SIGINT stops the orchestrator and the connections; each book thread
// stops when its connection does, and the writer flushes once they all
// have, even after a panic upstream (the run still fails then).
let failed = polymarket::orchestrator::run(outputs_tx, &stop);
let written = writer
.join()
.map_err(|_| anyhow::anyhow!("the writer thread panicked"))?;
log!("output closed");
written?;
anyhow::ensure!(!socket_panicked, "the socket thread panicked");
anyhow::ensure!(!book_panicked, "the book thread panicked");
anyhow::ensure!(failed == 0, "{failed} threads failed");
Ok(())
}
/// The writer thread: writes outputs in arrival order until the book thread
/// is gone, then flushes the last chunk.
/// The writer thread: writes outputs in arrival order until every book
/// thread is gone, then flushes the last chunk.
// ponytail: the chunk being filled lives only in memory until it is full, so
// a hard crash loses it (hours of data); flushing more often pads every flush
// to 64 MiB. Taken up in M4.
// a hard crash loses it; flushing more often pads every flush to 64 MiB.
// Taken up in M4.
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
let mut writer = AlignedSbeWriter::create(path, CHUNK)?;
for output in outputs {