Archived
feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
This commit is contained in:
24 files changed
+8984
-1214
No files matched your search
+10
-8
@@ -1,7 +1,7 @@
|
||||
//! The collector (roadmap/03): an orchestrator keeps the tracked markets equal
|
||||
//! to Gamma's live ones and spreads them over lanes, each a connection (read,
|
||||
//! stamp) and a book thread (parse, fold, check, resync), all writing through
|
||||
//! one writer (SBE).
|
||||
//! The collector (roadmap/04): an orchestrator keeps the tracked markets equal
|
||||
//! to Gamma's live ones and places them on connections (read, stamp, decode,
|
||||
//! route), which feed the book threads (fold, check, resync), all writing
|
||||
//! through one writer (SBE).
|
||||
//! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs
|
||||
//! go to stderr.
|
||||
|
||||
@@ -43,9 +43,10 @@ fn main() -> anyhow::Result<()> {
|
||||
writer_stop.store(true, Ordering::Relaxed);
|
||||
written
|
||||
});
|
||||
// SIGINT stops the orchestrator and the connections; each book thread
|
||||
// stops when its connection does, and the writer flushes once they all
|
||||
// have, even after a panic upstream (the run still fails then).
|
||||
// SIGINT (and SIGTERM, SIGHUP) stops the orchestrator and the
|
||||
// connections; the book threads stop once every connection has, and the
|
||||
// writer flushes once they all have, even after a panic upstream (the run
|
||||
// still fails then).
|
||||
let failed = polymarket::orchestrator::run(outputs_tx, &stop);
|
||||
let written = writer
|
||||
.join()
|
||||
@@ -60,7 +61,8 @@ fn main() -> anyhow::Result<()> {
|
||||
/// thread is gone, then flushes the last chunk.
|
||||
// ponytail: the chunk being filled lives only in memory until it is full, so
|
||||
// a hard crash loses it; flushing more often pads every flush to 64 MiB.
|
||||
// Taken up in M4.
|
||||
// Kept as a ceiling in M4 (roadmap/04): writing to a file of our own and
|
||||
// copying it into chunks would lift it.
|
||||
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
|
||||
let mut writer = AlignedSbeWriter::create(path, CHUNK)?;
|
||||
for output in outputs {
|
||||
|
||||
Reference in new issue
Block a user