docs: add project definition, roadmap, and evaluation plan

Order-book collector for prediction-market data (Polymarket, then Kalshi). Central bet is the concurrency pipeline; correctness/completeness first, speed later. Roadmap covers base challenge plus all four extensions.
This commit is contained in:
pascal committed 2026-11-04 13:36:36 +01:00
1 parent 97360aefac
commit adcf205f0b
4 files changed
+337

No files matched your search

+57
View File
@@ -0,0 +1,57 @@
# Evaluation
How we judge the finished software. This is a **post-completion evaluation**, not
a metric we develop against. Correctness (soundness/completeness tests),
benchmarks, and profiling drive development; this is the "does the data have
financial value" story told afterward.
## The idea: a trading simulator on top of the collector
Write a *separate* program that consumes our collector's output as its source of
truth, makes trading decisions, runs for a while, and reports P&L — a tangible
demonstration of the "faster/more complete makes more money" claim.
## The trap, and how to design around it
A trading sim measures the **strategy**, not the collector. Absolute P&L is
mostly "was the bot any good," which is noise for our purposes.
**Fix: make it differential — same strategy, two feeds.** Run the identical bot
on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L
*difference*. That delta is attributable to data quality, which is what the
challenge claims matters.
The two claims split apart:
- **Completeness is cleanly measurable on recorded data.** Treat our output as
ground truth, make a lossy copy (drop a % of messages, skip some
re-snapshots), run the same strategy on both, compare P&L. The gap = cost of
dropped data. No live trading needed.
- **Latency is NOT measurable from recorded data.** Collection latency is baked
into a recording; a backtest can't show "we'd have been a few ms faster." A
dollar figure on latency needs a live/forward A/B (two collectors, one slower,
same strategy in real time, or injected delay). Much bigger apparatus.
**The fill model is the hard part.** A naive "see best ask → buy → book profit"
bot overstates P&L massively: it ignores queue position, the level vanishing
before the order arrives, latency to the exchange, and market impact. A credible
fill model for Kalshi/Polymarket microstructure is arguably bigger than the
challenge itself. Scope it carefully or not at all.
## Cheaper metrics that target the claims directly
Prefer these; they can't be gamed by strategy luck:
- **Time-accuracy:** distribution of `received_at − exchange_timestamp`, and
separately our *internal* added latency (edge-stamp → output byte).
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
of missed deletes / stale levels under induced loss.
- **Redundancy value (once ext1 exists):** messages the backup socket caught that
the primary missed — literal "data loss avoided."
## Recommendation
Keep the trading-sim as the narrative. For an actual P&L demo, build only the
**differential completeness** version (full vs lossy, recorded data, same
strategy) — the one variant that's both doable and meaningful. Treat the full
live-latency-P&L rig as out of scope. Revisit at the polish milestone.