This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/evaluation.md
T
pascal 02747b882f feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
2026-11-11 15:18:48 +00:00

58 lines
2.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Evaluation
How we judge the finished software. This is a **post-completion evaluation**, not
a metric we develop against. Correctness (soundness/completeness tests),
benchmarks, and profiling drive development; this is the "does the data have
financial value" story told afterward.
## The idea: a trading simulator on top of the collector
Write a *separate* program that consumes our collector's output as its source of
truth, makes trading decisions, runs for a while, and reports P&L — a tangible
demonstration of the "faster/more complete makes more money" claim.
## The trap, and how to design around it
A trading sim measures the **strategy**, not the collector. Absolute P&L is
mostly "was the bot any good," which is noise for our purposes.
**Fix: make it differential — same strategy, two feeds.** Run the identical bot
on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L
*difference*. That delta is attributable to data quality, which is what the
challenge claims matters.
The two claims split apart:
- **Completeness is cleanly measurable on recorded data.** Treat our output as
ground truth, make a lossy copy (drop a % of messages, skip some
resyncs), run the same strategy on both, compare P&L. The gap = cost of
dropped data. No live trading needed.
- **Latency is NOT measurable from recorded data.** Collection latency is baked
into a recording; a backtest can't show "we'd have been a few ms faster." A
dollar figure on latency needs a live/forward A/B (two collectors, one slower,
same strategy in real time, or injected delay). Much bigger apparatus.
**The fill model is the hard part.** A naive "see best ask → buy → book profit"
bot overstates P&L massively: it ignores queue position, the level vanishing
before the order arrives, latency to the exchange, and market impact. A credible
fill model for Kalshi/Polymarket microstructure is arguably bigger than the
challenge itself. Scope it carefully or not at all.
## Cheaper metrics that target the claims directly
Prefer these; they can't be gamed by strategy luck:
- **Time-accuracy:** distribution of `received_at − exchange_timestamp`, and
separately our *internal* added latency (edge-stamp → output byte).
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
of missed deletes / stale levels under induced loss.
- **Redundancy value:** changes one connection missed and the other delivered
(rescued) — literal "data loss avoided."
## Recommendation
Keep the trading-sim as the narrative. For an actual P&L demo, build only the
**differential completeness** version (full vs lossy, recorded data, same
strategy) — the one variant that's both doable and meaningful. Treat the full
live-latency-P&L rig as out of scope. Revisit at the polish milestone.