Collects the order books of Polymarket's 20 busiest markets (40 symbols) into the template's SBE format: a Delta per level change, plus Snapshots on subscribe, every minute, and whenever a fresh book corrects ours. Every change is checked against Polymarket's own book hash; a failed check resyncs the symbol on the open socket, and the log counts losses apart from the few ms our own resync hides. Runs until SIGINT, which closes the socket and flushes the output; reconnects when the server drops it. A `reconcile` binary checks an output file against itself.
2.8 KiB
Evaluation
How we judge the finished software. This is a post-completion evaluation, not a metric we develop against. Correctness (soundness/completeness tests), benchmarks, and profiling drive development; this is the "does the data have financial value" story told afterward.
The idea: a trading simulator on top of the collector
Write a separate program that consumes our collector's output as its source of truth, makes trading decisions, runs for a while, and reports P&L — a tangible demonstration of the "faster/more complete makes more money" claim.
The trap, and how to design around it
A trading sim measures the strategy, not the collector. Absolute P&L is mostly "was the bot any good," which is noise for our purposes.
Fix: make it differential — same strategy, two feeds. Run the identical bot on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L difference. That delta is attributable to data quality, which is what the challenge claims matters.
The two claims split apart:
- Completeness is cleanly measurable on recorded data. Treat our output as ground truth, make a lossy copy (drop a % of messages, skip some resyncs), run the same strategy on both, compare P&L. The gap = cost of dropped data. No live trading needed.
- Latency is NOT measurable from recorded data. Collection latency is baked into a recording; a backtest can't show "we'd have been a few ms faster." A dollar figure on latency needs a live/forward A/B (two collectors, one slower, same strategy in real time, or injected delay). Much bigger apparatus.
The fill model is the hard part. A naive "see best ask → buy → book profit" bot overstates P&L massively: it ignores queue position, the level vanishing before the order arrives, latency to the exchange, and market impact. A credible fill model for Kalshi/Polymarket microstructure is arguably bigger than the challenge itself. Scope it carefully or not at all.
Cheaper metrics that target the claims directly
Prefer these; they can't be gamed by strategy luck:
- Time-accuracy: distribution of
received_at − exchange_timestamp, and separately our internal added latency (edge-stamp → output byte). - Completeness: reconciliation error rate (snapshot↔delta divergence); count of missed deletes / stale levels under induced loss.
- Redundancy value (once ext1 exists): messages the backup socket caught that the primary missed — literal "data loss avoided."
Recommendation
Keep the trading-sim as the narrative. For an actual P&L demo, build only the differential completeness version (full vs lossy, recorded data, same strategy) — the one variant that's both doable and meaningful. Treat the full live-latency-P&L rig as out of scope. Revisit at the polish milestone.