This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/evaluation.md
T
pascal 02747b882f feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
2026-11-11 15:18:48 +00:00

2.8 KiB
Raw Blame History

Evaluation

How we judge the finished software. This is a post-completion evaluation, not a metric we develop against. Correctness (soundness/completeness tests), benchmarks, and profiling drive development; this is the "does the data have financial value" story told afterward.

The idea: a trading simulator on top of the collector

Write a separate program that consumes our collector's output as its source of truth, makes trading decisions, runs for a while, and reports P&L — a tangible demonstration of the "faster/more complete makes more money" claim.

The trap, and how to design around it

A trading sim measures the strategy, not the collector. Absolute P&L is mostly "was the bot any good," which is noise for our purposes.

Fix: make it differential — same strategy, two feeds. Run the identical bot on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L difference. That delta is attributable to data quality, which is what the challenge claims matters.

The two claims split apart:

  • Completeness is cleanly measurable on recorded data. Treat our output as ground truth, make a lossy copy (drop a % of messages, skip some resyncs), run the same strategy on both, compare P&L. The gap = cost of dropped data. No live trading needed.
  • Latency is NOT measurable from recorded data. Collection latency is baked into a recording; a backtest can't show "we'd have been a few ms faster." A dollar figure on latency needs a live/forward A/B (two collectors, one slower, same strategy in real time, or injected delay). Much bigger apparatus.

The fill model is the hard part. A naive "see best ask → buy → book profit" bot overstates P&L massively: it ignores queue position, the level vanishing before the order arrives, latency to the exchange, and market impact. A credible fill model for Kalshi/Polymarket microstructure is arguably bigger than the challenge itself. Scope it carefully or not at all.

Cheaper metrics that target the claims directly

Prefer these; they can't be gamed by strategy luck:

  • Time-accuracy: distribution of received_at − exchange_timestamp, and separately our internal added latency (edge-stamp → output byte).
  • Completeness: reconciliation error rate (snapshot↔delta divergence); count of missed deletes / stale levels under induced loss.
  • Redundancy value: changes one connection missed and the other delivered (rescued) — literal "data loss avoided."

Recommendation

Keep the trading-sim as the narrative. For an actual P&L demo, build only the differential completeness version (full vs lossy, recorded data, same strategy) — the one variant that's both doable and meaningful. Treat the full live-latency-P&L rig as out of scope. Revisit at the polish milestone.