feat!: a generic core, and Kalshi through it

The orchestrator, the connection loop and the book thread's shell move to
src/core/, generic over an Exchange and its loss rule (ADR 0004).
Polymarket runs through it unchanged. Kalshi joins: a signed handshake
(KALSHI_KEY_ID_FILE, KALSHI_PRIVATE_KEY_FILE), a reconnect on a skipped
sequence number, first one wins by clock and rank, and its market listing
with markets tracked from three hours before they open.

Breaking: the exchange is now the first argument,
`engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]`.
This commit is contained in:
pascal committed 2026-11-11 20:54:17 +00:00
1 parent 480ee3ad2d
commit 1a5bbcd602
33 files changed
+8150 -1846

No files matched your search

+25
View File
@@ -77,6 +77,19 @@ impl Book {
previous.unwrap_or(0)
}
/// Moves a level's quantity by `delta`, which can be negative, and returns
/// the sum. A level can't hold less than nothing: a negative sum removes
/// it, and tells the caller the book was wrong.
pub fn add(&mut self, side: Side, price: i64, delta: i64) -> i64 {
let levels = match side {
Side::Buy => &self.bids,
Side::Sell => &self.asks,
};
let quantity = levels.get(&price).copied().unwrap_or(0) + delta;
self.set(side, price, quantity.max(0));
quantity
}
/// Both sides in ascending price order.
pub fn snapshot(
&self,
@@ -157,6 +170,18 @@ mod tests {
assert_eq!(format_fixed(parse_fixed("0.50").unwrap()), "0.5");
}
#[test]
fn adding_moves_a_level_and_never_leaves_it_below_zero() {
let mut book = Book::default();
assert_eq!(book.add(Side::Sell, 600_000, 5), 5);
assert_eq!(book.add(Side::Sell, 600_000, -2), 3);
assert_eq!(book.asks, BTreeMap::from([(600_000, 3)]));
assert_eq!(book.add(Side::Sell, 600_000, -3), 0);
assert!(book.asks.is_empty());
assert_eq!(book.add(Side::Buy, 400_000, -1), -1);
assert!(book.bids.is_empty());
}
#[test]
fn a_zero_quantity_removes_the_level() {
let mut book = Book::default();
+772
View File
@@ -0,0 +1,772 @@
//! A book thread: owns the books of its markets' symbols and folds into them
//! what their connections deliver, by the exchange's [`Rule`] (ADR 0004). The
//! shell here is the same for every exchange: which markets and symbols it
//! holds and on which connections, which connections are up, moves between
//! connections, each minute's Snapshots and load report, and the watch for a
//! subscription gone quiet on one connection (roadmap/05). What an event does
//! to a book, how a loss is found and when to resync is the rule's.
use std::collections::{BTreeMap, BTreeSet, HashMap};
use std::time::Duration;
use chrono::{DateTime, DurationRound, TimeDelta, Utc};
use crossbeam::channel::{Receiver, Sender, never, select};
use engineering_challenge::Side;
use engineering_challenge::book::{Book, Output};
use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain;
use super::socket::Command;
use super::{Arrival, Event, Market, Received};
/// A market measured for less than this waits for the next load report: over
/// a few seconds, its opening books alone would read as a burst.
pub(crate) const MIN_MEASURED: TimeDelta = TimeDelta::seconds(30);
/// How often a book thread whose rule asks for it wakes for the rule's timer.
const WAKE_EVERY: Duration = Duration::from_millis(10);
/// A connection that delivered none of a symbol's changes in a minute while
/// another delivered at least this many is quiet for it (sprint 05: crossed
/// once in 21 live minutes; at 1, copies landing in the next minute would
/// count).
const QUIET_AGAINST: u32 = 3;
/// What the book thread does with an event or a command: write output, ask a
/// connection (by number) for something, or report to the orchestrator.
#[derive(Debug)]
pub enum Effect {
Write(Output),
Send(usize, Command),
Report(Report),
}
/// What a book thread tells the orchestrator.
#[derive(Debug, PartialEq, Eq)]
pub enum Report {
Loads(Loads),
/// The market stayed quiet on the connection after being resubscribed
/// there: its subscription should move.
Quiet {
market: String,
connection: usize,
},
}
/// Each market's load (market id → bytes per second on one connection),
/// every market of the book thread measured for at least [`MIN_MEASURED`].
pub type Loads = HashMap<String, u64>;
/// What the orchestrator tells a book thread. A market's commands all go to
/// its one book thread, in order.
#[derive(Debug)]
pub enum BookCommand<M> {
/// A connection just opened, and how to reach it: sent to every book
/// thread before any command that names it.
Connected(usize, Sender<Command>),
/// Track the market, on these connections.
Add(M, Vec<usize>),
/// The market's subscription on `from` goes to `to`.
Move {
market: M,
from: usize,
to: usize,
},
Remove(M),
}
/// What an exchange's book thread does with its events: the part of a book
/// thread that differs between exchanges, with a state of its own per symbol
/// ([`Rule::Symbol`]) and for the thread. The shell calls it at each step
/// that concerns it, and it acts through the shell's symbols and the effects
/// it pushes.
pub trait Rule: Send + 'static {
type Market: Market;
type Event: Event;
/// Its state for one symbol.
type Symbol: Send;
/// A symbol of `market` starts being tracked.
fn symbol(&self, market: &Self::Market) -> Self::Symbol;
/// An event from `connection`, received at `received_at`.
fn on_event(
&mut self,
shell: &mut Shell<Self::Symbol>,
connection: usize,
received_at: DateTime<Utc>,
event: Self::Event,
effects: &mut Vec<Effect>,
);
/// `connection` stopped carrying the symbol: its subscription there left
/// (the shell has taken the connection off the symbol's).
fn on_leave(
&mut self,
up: &[bool],
name: &str,
symbol: &mut Symbol<Self::Symbol>,
connection: usize,
effects: &mut Vec<Effect>,
);
/// `connection` went down (the shell has marked it so).
fn on_down(
&mut self,
shell: &mut Shell<Self::Symbol>,
connection: usize,
effects: &mut Vec<Effect>,
);
/// After every event, and while none comes.
fn on_timer(
&mut self,
shell: &mut Shell<Self::Symbol>,
now: DateTime<Utc>,
effects: &mut Vec<Effect>,
);
/// Whether the book thread must wake soon for [`Rule::on_timer`].
fn wakes_soon(&self) -> bool;
/// The symbol is no longer tracked.
fn on_removed(&mut self, name: &str);
/// Whether the symbol has a book to write in a Snapshot.
fn has_book(symbol: &Self::Symbol) -> bool;
/// Whether the symbol may be judged quiet this minute: its book is in
/// order and no fresh one is on its way.
fn may_be_quiet(symbol: &Self::Symbol) -> bool;
/// At the minute, after the symbol's Snapshot.
fn on_minute(
&mut self,
up: &[bool],
name: &str,
symbol: &mut Symbol<Self::Symbol>,
now: DateTime<Utc>,
effects: &mut Vec<Effect>,
);
/// Logs the minute line, the shell's figures in it.
fn log_minute(&mut self, minute: &Minute);
}
/// The shell's figures for the minute line.
pub struct Minute {
pub number: usize,
pub symbols: usize,
/// Subscriptions found quiet, each resubscribed, since the start.
pub quiet: u64,
/// Markets quiet again after their resubscribe, reported for a move.
pub quiet_reported: u64,
/// Changes received this minute, from all connections.
pub received: u64,
/// The most events waiting for the book thread since the last line: how
/// far behind it fell.
pub peak_waiting: usize,
}
/// A symbol's subscription on one of its connections, as watched for going
/// quiet.
#[derive(Debug, Clone, Copy)]
pub struct Watch {
pub connection: usize,
/// Its changes received from this connection this minute: every one,
/// whatever the rule then made of it.
pub received: u32,
/// When it was last subscribed there: a minute it didn't cover whole is
/// not judged.
pub since: DateTime<Utc>,
/// When the connection last delivered a complete book of it: its first
/// since the connection came up is the reconnect's.
pub book_at: DateTime<Utc>,
}
impl Watch {
fn new(connection: usize, since: DateTime<Utc>) -> Self {
Self {
connection,
received: 0,
since,
book_at: DateTime::UNIX_EPOCH,
}
}
}
/// A symbol: its book, where it is subscribed, and the rule's state for it.
pub struct Symbol<S> {
pub book: Book,
/// Of the last change applied, or of the book that replaced ours: the
/// time a Snapshot's book is the exchange's as of.
pub exchange_timestamp: DateTime<Utc>,
/// Its market's id.
pub market: String,
/// The connections it is subscribed on (three while a move overlaps).
pub connections: Vec<usize>,
/// One per connection in `connections`.
pub watches: Vec<Watch>,
/// The connection it was quiet on last minute, and resubscribed.
pub quiet_on: Option<usize>,
/// The receive time of its last Delta or Snapshot: a later Delta is
/// stamped no earlier, so replaying the output by time keeps its order.
pub last_written: DateTime<Utc>,
pub rule: S,
}
impl<S> Symbol<S> {
fn new(market: &str, connections: &[usize], now: DateTime<Utc>, rule: S) -> Self {
Self {
book: Book::default(),
exchange_timestamp: DateTime::UNIX_EPOCH,
market: market.to_string(),
connections: connections.to_vec(),
watches: connections
.iter()
.map(|&connection| Watch::new(connection, now))
.collect(),
quiet_on: None,
last_written: DateTime::UNIX_EPOCH,
rule,
}
}
pub fn watch_mut(&mut self, connection: usize) -> Option<&mut Watch> {
self.watches
.iter_mut()
.find(|watch| watch.connection == connection)
}
/// One of its connections that is up, other than `except`.
pub fn other_up(&self, up: &[bool], except: usize) -> Option<usize> {
self.connections
.iter()
.copied()
.find(|&connection| connection != except && is_up(up, connection))
}
/// The Delta setting a level of its book to `quantity`, received at
/// `received_at`: stamped no earlier than what it last wrote.
pub fn delta(
&mut self,
name: &str,
side: Side,
price: i64,
quantity: i64,
received_at: DateTime<Utc>,
exchange_timestamp: DateTime<Utc>,
) -> Output {
self.last_written = self.last_written.max(received_at);
Output::Delta(PriceLevelUpdateDomain {
backtest_timestamp: self.last_written,
exchange_timestamp,
price_level: price,
quantity,
side,
symbol: name.to_string(),
})
}
/// The Snapshot of its book, received at `received_at`.
pub fn snapshot(&mut self, name: &str, received_at: DateTime<Utc>) -> Output {
self.last_written = self.last_written.max(received_at);
self.book
.snapshot(name, received_at, self.exchange_timestamp)
}
/// The connection quiet for it over the minute begun at `started`: one
/// that delivered none of its changes while another delivered at least
/// [`QUIET_AGAINST`], both up and subscribed for the whole minute.
fn quiet(&self, started: DateTime<Utc>, up_since: &[Option<DateTime<Utc>>]) -> Option<usize> {
let judged: Vec<&Watch> = self
.watches
.iter()
.filter(|watch| {
watch.since <= started
&& up_since
.get(watch.connection)
.copied()
.flatten()
.is_some_and(|up| up <= started)
})
.collect();
let busiest = judged.iter().map(|watch| watch.received).max()?;
(busiest >= QUIET_AGAINST)
.then(|| judged.iter().find(|watch| watch.received == 0))
.flatten()
.map(|watch| watch.connection)
}
}
pub fn is_up(up: &[bool], connection: usize) -> bool {
up.get(connection).copied().unwrap_or(false)
}
/// A market's frame bytes since `since`, from all of its connections.
struct Measure {
bytes: u64,
since: DateTime<Utc>,
}
impl Measure {
/// Bytes per second since `since` on one of `connections`, then starts
/// again from `now`; none before [`MIN_MEASURED`], the bytes kept for the
/// next report.
fn take(&mut self, now: DateTime<Utc>, connections: usize) -> Option<u64> {
let measured = now - self.since;
if measured < MIN_MEASURED {
return None;
}
let load =
self.bytes * 1_000 / measured.num_milliseconds() as u64 / connections.max(1) as u64;
*self = Measure {
bytes: 0,
since: now,
};
Some(load)
}
}
/// A market of the book thread.
pub struct Tracked {
/// Its symbols' names.
pub(crate) names: Vec<String>,
measure: Measure,
/// A move under way: the connection left once the new one, the second
/// here, delivers a book of the market. Until then both carry it, so the
/// move costs no change.
leaving: Option<(usize, usize)>,
}
/// What every exchange's book thread holds: its symbols, its markets, its
/// connections' state and the minute's.
pub struct Shell<S> {
/// The exchange's name, for the logs.
pub exchange: &'static str,
/// Its book thread's number, for the logs.
pub number: usize,
pub symbols: BTreeMap<String, Symbol<S>>,
/// By market id.
pub markets: HashMap<String, Tracked>,
/// By connection number: whether it is up.
pub up: Vec<bool>,
/// By connection number: since when it is up, if it is.
pub up_since: Vec<Option<DateTime<Utc>>>,
/// When the minute being counted began: its quiet subscriptions are
/// judged at its end. None before the first.
minute_started: Option<DateTime<Utc>>,
/// Changes received this minute, from all connections: the rule counts
/// them, as it reads them.
pub received_this_minute: u64,
/// The receive time of the last event applied: the time a Snapshot's book
/// is exact as of.
pub last_received_at: Option<DateTime<Utc>>,
/// The next wall-clock minute: its Snapshots, load report and log line.
next_minute_at: Option<DateTime<Utc>>,
/// The most events waiting for the book thread since the last minute log.
pub peak_waiting: usize,
pub quiet: u64,
pub quiet_reported: u64,
}
impl<S> Shell<S> {
/// A change of `name` from `connection`, counted for the quiet watch:
/// every one of a tracked symbol, before the rule makes anything of it.
pub fn count_received(&mut self, name: &str, connection: usize) {
let Some(symbol) = self.symbols.get_mut(name) else {
return;
};
if let Some(watch) = symbol.watch_mut(connection) {
watch.received += 1;
}
self.received_this_minute += 1;
}
fn set_up(&mut self, connection: usize, up: bool, at: DateTime<Utc>) {
if self.up.len() <= connection {
self.up.resize(connection + 1, false);
self.up_since.resize(connection + 1, None);
}
if up && !self.up[connection] {
self.up_since[connection] = Some(at);
} else if !up {
self.up_since[connection] = None;
}
self.up[connection] = up;
}
}
pub struct Feed<R: Rule> {
pub shell: Shell<R::Symbol>,
pub rule: R,
}
impl<R: Rule> Feed<R> {
pub fn new(exchange: &'static str, number: usize, rule: R) -> Self {
Self {
shell: Shell {
exchange,
number,
symbols: BTreeMap::new(),
markets: HashMap::new(),
up: Vec::new(),
up_since: Vec::new(),
minute_started: None,
received_this_minute: 0,
last_received_at: None,
next_minute_at: None,
peak_waiting: 0,
quiet: 0,
quiet_reported: 0,
},
rule,
}
}
/// Tracks a market's symbols, then asks its connections to subscribe
/// them: a book can't arrive for a symbol not yet known. Its load is
/// measured from `now`.
pub fn add_market(
&mut self,
market: &R::Market,
connections: &[usize],
now: DateTime<Utc>,
) -> Vec<Effect> {
let names = market.symbols().to_vec();
for name in &names {
let rule = self.rule.symbol(market);
self.shell.symbols.insert(
name.clone(),
Symbol::new(market.id(), connections, now, rule),
);
}
let tracked = Tracked {
names: names.clone(),
measure: Measure {
bytes: 0,
since: now,
},
leaving: None,
};
self.shell.markets.insert(market.id().to_string(), tracked);
connections
.iter()
.map(|&connection| Effect::Send(connection, Command::Subscribe(names.clone())))
.collect()
}
/// Moves a market's subscription from one connection to another: it
/// subscribes on `to` now, and leaves `from` once `to` delivers a book of
/// the market. The book stays as it is: no Snapshot, no subscription gap.
pub fn move_market(
&mut self,
market: &R::Market,
from: usize,
to: usize,
now: DateTime<Utc>,
) -> Vec<Effect> {
let Some(tracked) = self.shell.markets.get_mut(market.id()) else {
return Vec::new();
};
// A move still under way when the next comes: it ends now.
let mut effects = match tracked.leaving.take() {
Some((left, _)) => self.leave(market.id(), left),
None => Vec::new(),
};
if let Some(tracked) = self.shell.markets.get_mut(market.id()) {
tracked.leaving = Some((from, to));
}
for name in market.symbols() {
if let Some(symbol) = self.shell.symbols.get_mut(name)
&& !symbol.connections.contains(&to)
{
symbol.connections.push(to);
symbol.watches.push(Watch::new(to, now));
}
}
effects.push(Effect::Send(
to,
Command::Subscribe(market.symbols().to_vec()),
));
effects
}
/// The market's symbols stop being carried by `connection`: it is
/// unsubscribed there, and the rule told.
fn leave(&mut self, market: &str, connection: usize) -> Vec<Effect> {
let Some(tracked) = self.shell.markets.get(market) else {
return Vec::new();
};
let names = tracked.names.clone();
let mut effects = vec![Effect::Send(
connection,
Command::Unsubscribe(names.clone()),
)];
let Shell { symbols, up, .. } = &mut self.shell;
for name in &names {
if let Some(symbol) = symbols.get_mut(name) {
symbol.connections.retain(|&on| on != connection);
symbol
.watches
.retain(|watch| watch.connection != connection);
self.rule
.on_leave(up, name, symbol, connection, &mut effects);
}
}
effects
}
/// Forgets a market's symbols and unsubscribes them; nothing is written
/// for them. Their events still on the way are ignored, as any unknown
/// symbol's.
pub fn remove_market(&mut self, market: &R::Market) -> Vec<Effect> {
let mut connections = Vec::new();
for name in market.symbols() {
if let Some(symbol) = self.shell.symbols.remove(name) {
connections = symbol.connections;
}
self.rule.on_removed(name);
}
self.shell.markets.remove(market.id());
connections
.into_iter()
.map(|connection| {
Effect::Send(connection, Command::Unsubscribe(market.symbols().to_vec()))
})
.collect()
}
/// What a connection sent: an event, or itself coming up or going down.
pub fn on_received(&mut self, received: Received<R::Event>) -> Vec<Effect> {
match received {
Received::Event(arrival) => self.on_event(arrival),
Received::Up(connection) => {
// Its first events come after this marker; any other
// connection's before it were received earlier.
let at = self.shell.last_received_at.unwrap_or(DateTime::UNIX_EPOCH);
self.shell.set_up(connection, true, at);
Vec::new()
}
Received::Down(connection) => {
self.shell.set_up(connection, false, DateTime::UNIX_EPOCH);
let mut effects = Vec::new();
self.rule.on_down(&mut self.shell, connection, &mut effects);
effects
}
}
}
pub fn on_event(&mut self, arrival: Arrival<R::Event>) -> Vec<Effect> {
let (received_at, connection) = (arrival.received_at, arrival.connection);
// A connection that delivers is up, its marker possibly still behind.
if !is_up(&self.shell.up, connection) {
self.shell.set_up(connection, true, received_at);
}
// Minute Snapshots due before this event hold the books as they were.
let mut effects = self.tick(received_at);
self.shell.last_received_at = Some(received_at);
let market = arrival.event.market().map(str::to_string);
if let Some(tracked) = market
.as_ref()
.and_then(|market| self.shell.markets.get_mut(market))
{
tracked.measure.bytes += arrival.bytes;
// A move ends once the new connection delivers a book of the
// market.
let ends = tracked
.leaving
.filter(|&(_, to)| to == connection && arrival.event.is_book());
if let (Some((from, _)), Some(market)) = (ends, &market) {
tracked.leaving = None;
effects.extend(self.leave(market, from));
}
}
self.rule.on_event(
&mut self.shell,
connection,
received_at,
arrival.event,
&mut effects,
);
self.rule
.on_timer(&mut self.shell, received_at, &mut effects);
effects
}
/// While no event comes: the minute, and the rule's timer.
pub fn idle(&mut self, now: DateTime<Utc>) -> Vec<Effect> {
let mut effects = self.tick(now);
self.rule.on_timer(&mut self.shell, now, &mut effects);
effects
}
/// Once `now` reaches the next wall-clock minute: reports the loads,
/// emits the minute Snapshots, resubscribes (or reports) the
/// subscriptions quiet over the minute that ends, and lets the rule act
/// on each symbol.
pub fn tick(&mut self, now: DateTime<Utc>) -> Vec<Effect> {
let shell = &mut self.shell;
let next = *shell.next_minute_at.get_or_insert_with(|| next_minute(now));
if now < next {
return Vec::new();
}
shell.next_minute_at = Some(next_minute(now));
let started = shell.minute_started.replace(now);
let mut loads = Loads::new();
for (market, tracked) in &mut shell.markets {
let connections = tracked
.names
.first()
.and_then(|name| shell.symbols.get(name))
.map_or(1, |symbol| symbol.connections.len());
if let Some(load) = tracked.measure.take(now, connections) {
loads.insert(market.clone(), load);
}
}
let mut effects = vec![Effect::Report(Report::Loads(loads))];
let Some(received_at) = shell.last_received_at else {
return effects;
};
self.rule.log_minute(&Minute {
number: shell.number,
symbols: shell.symbols.len(),
quiet: shell.quiet,
quiet_reported: shell.quiet_reported,
received: shell.received_this_minute,
peak_waiting: shell.peak_waiting,
});
shell.peak_waiting = 0;
shell.received_this_minute = 0;
let mut reported = BTreeSet::new();
let exchange = shell.exchange;
for (name, symbol) in &mut shell.symbols {
let quiet = started
.filter(|_| R::may_be_quiet(&symbol.rule))
.and_then(|started| symbol.quiet(started, &shell.up_since));
for watch in &mut symbol.watches {
watch.received = 0;
}
match quiet {
// Resubscribed already, and still quiet: it should move.
Some(connection) if symbol.quiet_on == Some(connection) => {
symbol.quiet_on = None;
if reported.insert((symbol.market.clone(), connection)) {
shell.quiet_reported += 1;
effects.push(Effect::Report(Report::Quiet {
market: symbol.market.clone(),
connection,
}));
}
}
// The other connection carries it meanwhile: the book that
// comes back is ignored, no Snapshot, no subscription gap.
Some(connection) => {
log!("{exchange}: {name}: quiet on connection {connection}, resubscribed");
shell.quiet += 1;
symbol.quiet_on = Some(connection);
if let Some(watch) = symbol.watch_mut(connection) {
watch.since = now;
}
effects.push(Effect::Send(connection, Command::Resubscribe(name.clone())));
}
None => symbol.quiet_on = None,
}
if R::has_book(&symbol.rule) {
// A change held across the minute is written after it.
effects.push(Effect::Write(symbol.snapshot(name, received_at)));
}
self.rule
.on_minute(&shell.up, name, symbol, now, &mut effects);
}
effects
}
}
/// A book thread: runs until every connection and the orchestrator have
/// dropped its channels, then drops its senders so the writer thread finishes
/// once every book thread has.
// ponytail: unbounded channels never block a socket read nor drop an event;
// the ceiling is memory if a consumer stays behind. The minute log's peak of
// events waiting makes that visible; bounded channels are a constructor change
// if it ever matters.
pub fn run<R: Rule>(
feed: Feed<R>,
events: Receiver<Received<R::Event>>,
commands: Receiver<BookCommand<R::Market>>,
outputs: Sender<Output>,
reports: Sender<Report>,
) {
serve(feed, Vec::new(), events, commands, outputs, reports)
}
/// [`run`], knowing `connections` already (by number); the orchestrator
/// announces the others as it opens them.
pub(crate) fn serve<R: Rule>(
mut feed: Feed<R>,
mut connections: Vec<Sender<Command>>,
events: Receiver<Received<R::Event>>,
commands: Receiver<BookCommand<R::Market>>,
outputs: Sender<Output>,
reports: Sender<Report>,
) {
let no_commands = never();
let mut orchestrator_gone = false;
let exchange = feed.shell.exchange;
loop {
let mut wake = (next_minute(Utc::now()) - Utc::now())
.to_std()
.unwrap_or(Duration::ZERO);
if feed.rule.wakes_soon() {
wake = wake.min(WAKE_EVERY);
}
let commands = if orchestrator_gone {
&no_commands
} else {
&commands
};
let effects = select! {
recv(events) -> received => match received {
Ok(received) => {
feed.shell.peak_waiting = feed.shell.peak_waiting.max(events.len());
feed.on_received(received)
}
Err(_) => return,
},
recv(commands) -> command => match command {
Ok(BookCommand::Connected(number, connection)) => {
assert_eq!(number, connections.len(), "connections open in order");
connections.push(connection);
Vec::new()
}
Ok(BookCommand::Add(market, on)) => feed.add_market(&market, &on, Utc::now()),
Ok(BookCommand::Move { market, from, to }) => {
feed.move_market(&market, from, to, Utc::now())
}
Ok(BookCommand::Remove(market)) => feed.remove_market(&market),
// It is stopping: the connections' end still decides ours.
Err(_) => {
orchestrator_gone = true;
Vec::new()
}
},
default(wake) => feed.idle(Utc::now()),
};
for effect in effects {
match effect {
Effect::Write(output) => {
if outputs.send(output).is_err() {
return log!("{exchange}: the writer thread is gone, stopping");
}
}
// The connection may be gone already, while stopping, and the
// orchestrator too.
Effect::Send(connection, command) => _ = connections[connection].send(command),
Effect::Report(report) => _ = reports.send(report),
}
}
}
}
pub fn next_minute(now: DateTime<Utc>) -> DateTime<Utc> {
now.duration_trunc(TimeDelta::minutes(1))
.expect("a minute divides any UTC time")
+ TimeDelta::minutes(1)
}
+187
View File
@@ -0,0 +1,187 @@
//! The exchange-agnostic collector (ADR 0004): the orchestrator, the
//! connections and the book threads, generic over an [`Exchange`], which
//! supplies the websocket layer, the fetchers and the rule a book thread
//! applies to each event.
pub mod feed;
pub mod orchestrator;
pub mod socket;
use std::fmt::Debug;
use std::hash::{DefaultHasher, Hash, Hasher};
use std::thread;
use std::time::Duration;
use chrono::{DateTime, Utc};
use crossbeam::channel::{Receiver, Sender};
use feed::Rule;
/// How many book threads own the books. One handled ~30,000 frames a second
/// in sprint 02.
pub const BOOK_THREADS: usize = 16;
/// The book thread that owns a market's books, from a hash of its id.
/// Computed, so the orchestrator and every connection agree without a shared
/// table.
pub fn book_thread(market: &str) -> usize {
let mut hasher = DefaultHasher::new();
market.hash(&mut hasher);
(hasher.finish() % BOOK_THREADS as u64) as usize
}
/// An exchange, as the collector uses it.
pub trait Exchange: Send + Sync + 'static {
type Market: Market;
type Event: Event;
type Session: Session<Event = Self::Event>;
type Rule: Rule<Market = Self::Market, Event = Self::Event>;
/// What its log lines start with.
const NAME: &'static str;
/// The websocket's handshake request, made again for every connect.
fn request(&self) -> anyhow::Result<tungstenite::handshake::client::Request>;
/// A connection's state on the exchange's side, fresh for every connect.
fn session(&self) -> Self::Session;
/// A book thread's rule.
fn rule(&self) -> Self::Rule;
/// A fetcher: answers each request with findings until either channel is
/// gone. Two run, so a full scan never holds up the minute check.
fn fetch(&self, requests: Receiver<Request>, findings: Sender<Finding<Self::Market>>);
}
/// A live market: its id and its symbols, plus what its exchange needs.
pub trait Market: Clone + Debug + Send + 'static {
/// The id it is tracked and routed by.
fn id(&self) -> &str;
/// Its symbols, each with a book.
fn symbols(&self) -> &[String];
}
/// An exchange's event, as a connection routes it.
pub trait Event: Debug + Send + 'static {
/// The market it belongs to, which decides its book thread; none for an
/// event no book needs.
fn market(&self) -> Option<&str>;
/// Whether it is a full book: a market's first from a connection ends a
/// move to it.
fn is_book(&self) -> bool;
}
/// One connection's side of the exchange: the frames that carry what a book
/// thread asks for, and the events of each frame received.
pub trait Session: Send {
type Event;
/// Sent on connect, before any subscribe.
fn opening(&mut self) -> Vec<String>;
/// The frames that subscribe `symbols` on an open connection.
fn subscribe(&mut self, symbols: &[&String]) -> Vec<String>;
fn unsubscribe(&mut self, symbols: &[&String]) -> Vec<String>;
/// The text sent every few seconds to keep the connection open, if the
/// exchange wants one.
fn keepalive(&self) -> Option<&'static str>;
/// A frame's events.
fn decode(&mut self, text: &str) -> Result<Vec<Self::Event>, Undecoded>;
}
/// Why a frame gave no events.
#[derive(Debug)]
pub enum Undecoded {
/// Not a message we read: it is logged and dropped.
Unreadable(String),
/// The connection itself lost a message: it reconnects.
Lost(String),
}
/// A fetcher's request, tried again after `wait` when it fails: `tries` times
/// in all.
pub fn retried<T>(
tries: u32,
wait: Duration,
mut fetch: impl FnMut() -> anyhow::Result<T>,
) -> anyhow::Result<T> {
let mut tried = 1;
loop {
match fetch() {
Err(_) if tried < tries => {
tried += 1;
thread::sleep(wait);
}
result => return result,
}
}
}
/// What the orchestrator asks a fetcher for.
pub enum Request {
/// Every live market.
Rescan,
/// The live markets created and the markets closed since then.
Recent { since: DateTime<Utc> },
}
/// What a fetcher reports.
#[derive(Debug)]
pub enum Finding<M> {
/// A rescan began: once it is done, a market known now and not seen in it
/// is gone.
ScanStarted,
/// A page of live markets.
ScanPage(Vec<M>),
ScanDone,
/// The scan gave up: what it reported says nothing of the rest.
ScanFailed,
/// Live markets created since the request's time.
Created(Vec<M>),
/// The ids of markets closed since then.
Closed(Vec<String>),
RecentDone,
RecentFailed,
}
/// What a connection sends a book thread.
// Nearly every message is an event: boxing it to shrink the two rare markers
// would cost an allocation per event for nothing.
#[allow(clippy::large_enum_variant)]
#[derive(Debug)]
pub enum Received<E> {
Event(Arrival<E>),
/// The connection came up: its subscriptions are being sent.
Up(usize),
/// The connection went down: nothing more comes from it until it is up.
Down(usize),
}
/// One event of a frame on its way to its market's book thread.
#[derive(Debug)]
pub struct Arrival<E> {
/// The frame's receive time, stamped right after the socket read.
pub received_at: DateTime<Utc>,
/// The connection that delivered it.
pub connection: usize,
/// Its share of the frame's bytes: a frame is one market's change, or
/// books of several, each taking its share.
pub bytes: u64,
pub event: E,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_market_always_gets_the_same_book_thread_and_markets_spread() {
let threads: std::collections::BTreeSet<usize> = (0..40)
.map(|market| book_thread(&format!("KXMARKET-{market}")))
.collect();
assert!(threads.iter().all(|&thread| thread < BOOK_THREADS));
// 40 markets over 16 threads: several threads, not one.
assert!(threads.len() > 5, "{threads:?}");
assert_eq!(book_thread("0xabc"), book_thread(&String::from("0xabc")));
}
}
@@ -1,7 +1,7 @@
//! The orchestrator: the one owner of what is tracked. It keeps the tracked
//! markets equal to Gamma's live ones, decides which connections carry each
//! market and owns every timer, recycling connections among them; the two
//! fetchers only fetch (ADR 0002, ADR 0003, roadmap/04, roadmap/05).
//! markets equal to the exchange's live ones, decides which connections carry
//! each market and owns every timer, recycling connections among them; the
//! two fetchers only fetch (ADR 0002, ADR 0003, roadmap/04, roadmap/05).
use std::cmp::Reverse;
use std::collections::{HashMap, HashSet, VecDeque};
@@ -15,17 +15,15 @@ use crossbeam::channel::{Sender, select, unbounded};
use engineering_challenge::book::Output;
use super::feed::{self, BookCommand, Feed, Loads, Report};
use super::gamma::{self, Finding, Request};
use super::socket::{Command, Notice};
use super::wire::Received;
use super::{BOOK_THREADS, Market, book_thread, socket};
use super::socket::{self, Command, Notice};
use super::{BOOK_THREADS, Exchange, Finding, Market, Received, Request, book_thread};
/// How many connections carry each market, first one wins (ADR 0003). 1 for
/// a run to compare with.
const SUBSCRIPTIONS: usize = 2;
/// The most tokens a connection holds: a reconnect replays every one's book
/// The most symbols a connection holds: a reconnect replays every one's book
/// at once.
const CONNECTION_TOKENS: usize = 5_000;
const CONNECTION_SYMBOLS: usize = 5_000;
/// What the startup deal fills a connection to, leaving room for new markets.
const STARTUP_FILL: usize = 4_500;
/// A connection's byte budget, under the lowest rate at which one was seen to
@@ -46,10 +44,10 @@ const MOVED_STAYS: Duration = Duration::from_secs(10 * 60);
const SPLIT_WITHIN: Duration = Duration::from_secs(10 * 60);
const RECENT_EVERY: Duration = Duration::from_secs(60);
const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60);
/// Gamma's indexing delay: each recent check reaches this far before the
/// last successful one. New markets were listed up to 5 minutes after their
/// `createdAt` (docs/notes.md); known markets are skipped, so the margin costs
/// only the pages it reads again.
/// The listing's indexing delay: each recent check reaches this far before
/// the last successful one. New markets were listed up to 5 minutes after
/// their creation (docs/notes.md); known markets are skipped, so the margin
/// costs only the pages it reads again.
const RECENT_MARGIN: TimeDelta = TimeDelta::minutes(15);
/// Each connection is recycled about this often, one at a time, to shed what
/// accumulates on the exchange's side of it.
@@ -60,7 +58,12 @@ const WAKE_EVERY: Duration = Duration::from_secs(1);
/// Runs until `stop` is set, then waits for every connection and book thread.
/// Returns how many threads failed: those that panicked, and one if a fetcher
/// died.
pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
pub fn run<E: Exchange>(
exchange: Arc<E>,
outputs: Sender<Output>,
stop: &Arc<AtomicBool>,
) -> usize {
let name = E::NAME;
let (findings_tx, findings) = unbounded();
let (rescan, rescan_requests) = unbounded();
let (recent, recent_requests) = unbounded();
@@ -69,14 +72,14 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
let fetchers: Vec<JoinHandle<()>> = [rescan_requests, recent_requests]
.into_iter()
.map(|requests| {
let findings = findings_tx.clone();
thread::spawn(move || gamma::run(requests, findings))
let (findings, exchange) = (findings_tx.clone(), Arc::clone(&exchange));
thread::spawn(move || exchange.fetch(requests, findings))
})
.collect();
drop(findings_tx);
let (reports_tx, reports) = unbounded::<Report>();
let (notices_tx, notices) = unbounded::<Notice>();
let mut threads = Threads::start(&outputs, &reports_tx, &notices_tx, stop);
let mut threads = Threads::start(exchange, &outputs, &reports_tx, &notices_tx, stop);
drop(notices_tx);
// The book threads hold the only senders the writer waits for.
drop(outputs);
@@ -89,10 +92,10 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
// Until the first scan is done: the live markets it found, waiting to be
// dealt at random. Dealing page by page would keep markets created
// together on the same connections.
let mut first_scan: Option<Vec<Market>> = Some(Vec::new());
let mut first_scan: Option<Vec<E::Market>> = Some(Vec::new());
// The startup deal's commands still to send, and whatever came behind
// them: a market's commands keep their order.
let mut queue: VecDeque<BookCommand> = VecDeque::new();
let mut queue: VecDeque<BookCommand<E::Market>> = VecDeque::new();
let mut next_deal = Instant::now();
let (mut added, mut removed, mut moved, mut splits) = (0, 0, 0, 0);
let (mut recycled, mut quiet_moves, mut silent) = (0, 0, 0);
@@ -110,7 +113,7 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
// of a request, whose answer would never come: nothing would keep the
// set current.
if fetchers.iter().any(JoinHandle::is_finished) {
log!("polymarket: orchestrator: a fetcher died, stopping");
log!("{name}: orchestrator: a fetcher died, stopping");
stop.store(true, Ordering::Relaxed);
fetcher_died = true;
break;
@@ -138,7 +141,7 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
commands.extend(moves);
let busiest = tracker.connections.iter().map(|one| one.load).max();
log!(
"polymarket: orchestrator: {} markets on {} connections; {added} added, {removed} \
"{name}: orchestrator: {} markets on {} connections; {added} added, {removed} \
removed, {moved} moved ({quiet_moves} quiet), {splits} splits, {recycled} recycled, \
{silent} silent this minute; busiest connection at {} bytes/s",
tracker.markets.len(),
@@ -217,7 +220,7 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
Ok(Finding::Closed(ids)) => {
// Closed while the first scan runs: never dealt.
if let Some(waiting) = &mut first_scan {
waiting.retain(|market| !ids.contains(&market.condition_id));
waiting.retain(|market| !ids.iter().any(|id| id == market.id()));
}
let gone: Vec<_> = ids.iter().filter_map(|id| tracker.remove(id)).collect();
removed += gone.len();
@@ -237,7 +240,7 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
Ok(Report::Quiet { market, connection }) => {
if let Some(command) = tracker.quiet(&market, connection, Instant::now()) {
log!(
"polymarket: orchestrator: {market} still quiet on connection \
"{name}: orchestrator: {market} still quiet on connection \
{connection} after a resubscribe, moved"
);
quiet_moves += 1;
@@ -255,7 +258,7 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
let moves = tracker.closed(connection, Instant::now());
if !moves.is_empty() {
log!(
"polymarket: orchestrator: connection {connection} closed with 1013 \
"{name}: orchestrator: connection {connection} closed with 1013 \
twice, split: {} markets moved off",
moves.len()
);
@@ -285,11 +288,12 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
/// The threads the orchestrator feeds: the book threads, there from the
/// start, and the connections it opens.
struct Threads {
book_commands: Vec<Sender<BookCommand>>,
struct Threads<E: Exchange> {
exchange: Arc<E>,
book_commands: Vec<Sender<BookCommand<E::Market>>>,
/// Every book thread's events channel, by number: each connection gets a
/// clone of them all.
events: Vec<Sender<Received>>,
events: Vec<Sender<Received<E::Event>>>,
notices: Sender<Notice>,
/// Each connection's commands, by number, for its recycles.
connection_commands: Vec<Sender<Command>>,
@@ -298,14 +302,16 @@ struct Threads {
stop: Arc<AtomicBool>,
}
impl Threads {
impl<E: Exchange> Threads<E> {
fn start(
exchange: Arc<E>,
outputs: &Sender<Output>,
reports: &Sender<Report>,
notices: &Sender<Notice>,
stop: &Arc<AtomicBool>,
) -> Self {
let mut threads = Self {
exchange,
book_commands: Vec::new(),
events: Vec::new(),
notices: notices.clone(),
@@ -318,10 +324,11 @@ impl Threads {
let (commands, commands_rx) = unbounded();
let (events, events_rx) = unbounded();
let (outputs, reports) = (outputs.clone(), reports.clone());
let feed = Feed::new(E::NAME, number, threads.exchange.rule());
threads.book_commands.push(commands);
threads.events.push(events);
threads.book_threads.push(thread::spawn(move || {
feed::run(Feed::new(number), events_rx, commands_rx, outputs, reports)
feed::run(feed, events_rx, commands_rx, outputs, reports)
}));
}
threads
@@ -334,10 +341,10 @@ impl Threads {
let number = self.connections.len();
let (commands, commands_rx) = unbounded();
let (events, stop) = (self.events.clone(), Arc::clone(&self.stop));
let notices = self.notices.clone();
let (notices, exchange) = (self.notices.clone(), Arc::clone(&self.exchange));
self.connections.push(thread::spawn(move || {
socket::run(
socket::MARKET_CHANNEL_URL,
&*exchange,
number,
&events,
&commands_rx,
@@ -359,14 +366,14 @@ impl Threads {
/// To the market's book thread. One that died keeps its markets unserved,
/// and its connections stop the run.
fn send(&self, command: BookCommand) {
fn send(&self, command: BookCommand<E::Market>) {
let market = match &command {
BookCommand::Add(market, _)
| BookCommand::Move { market, .. }
| BookCommand::Remove(market) => market,
BookCommand::Connected(..) => unreachable!("sent by open() to every book thread"),
};
_ = self.book_commands[book_thread(&market.condition_id)].send(command);
_ = self.book_commands[book_thread(market.id())].send(command);
}
/// Waits for the connections, which stop on the flag, then for the book
@@ -388,8 +395,8 @@ impl Threads {
}
/// A tracked market, where it is and what it weighs.
struct Tracked {
market: Market,
struct Tracked<M> {
market: M,
/// The connections that carry it, all different.
connections: Vec<usize>,
/// Bytes per second on one connection, over its book thread's last
@@ -398,11 +405,11 @@ struct Tracked {
moved_at: Option<Instant>,
}
/// A connection's tokens and load, against its two limits, its 1013 closes,
/// A connection's symbols and load, against its two limits, its 1013 closes,
/// and whether and since when it is up.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
struct ConnectionState {
tokens: usize,
symbols: usize,
/// The sum of its markets' loads, bytes per second.
load: u64,
/// When it was last closed with 1013.
@@ -415,10 +422,10 @@ struct ConnectionState {
/// Which connections carry each market and what it weighs, and which markets
/// a rescan in progress hasn't seen yet.
struct Tracker {
struct Tracker<M> {
subscriptions: usize,
/// By `conditionId`.
markets: HashMap<String, Tracked>,
/// By market id.
markets: HashMap<String, Tracked<M>>,
/// By connection number.
connections: Vec<ConnectionState>,
/// While a rescan runs: the markets known at its start, not seen since.
@@ -430,7 +437,7 @@ struct Tracker {
recycling: Option<usize>,
}
impl Tracker {
impl<M: Market> Tracker<M> {
fn new(subscriptions: usize, seed: u64) -> Self {
Self {
subscriptions,
@@ -447,14 +454,14 @@ impl Tracker {
/// connections, each subscription dealt in an order of its own, so a
/// market's connections hold different neighbours and no connection gets
/// the markets created together.
fn deal(&mut self, markets: Vec<Market>) -> Vec<BookCommand> {
fn deal(&mut self, markets: Vec<M>) -> Vec<BookCommand<M>> {
let mut known = HashSet::new();
let markets: Vec<Market> = markets
let markets: Vec<M> = markets
.into_iter()
.filter(|market| known.insert(market.condition_id.clone()))
.filter(|market| known.insert(market.id().to_string()))
.collect();
let tokens: usize = markets.iter().map(|market| market.tokens.len()).sum();
let count = (self.subscriptions * tokens)
let symbols: usize = markets.iter().map(|market| market.symbols().len()).sum();
let count = (self.subscriptions * symbols)
.div_ceil(STARTUP_FILL)
.max(self.subscriptions);
self.connections = vec![ConnectionState::default(); count];
@@ -468,7 +475,7 @@ impl Tracker {
connection = (connection + 1) % count;
}
placed[market].push(connection);
self.connections[connection].tokens += markets[market].tokens.len();
self.connections[connection].symbols += markets[market].symbols().len();
}
}
markets
@@ -480,43 +487,43 @@ impl Tracker {
/// A market it doesn't know goes on different random connections with
/// room: its Add.
fn add(&mut self, market: Market) -> Option<BookCommand> {
fn add(&mut self, market: M) -> Option<BookCommand<M>> {
if let Some(unseen) = &mut self.unseen {
unseen.remove(&market.condition_id);
unseen.remove(market.id());
}
if self.markets.contains_key(&market.condition_id) {
if self.markets.contains_key(market.id()) {
return None;
}
let mut connections = Vec::new();
for _ in 0..self.subscriptions {
let connection = self.pick(market.tokens.len(), 0, &connections);
self.connections[connection].tokens += market.tokens.len();
let connection = self.pick(market.symbols().len(), 0, &connections);
self.connections[connection].symbols += market.symbols().len();
connections.push(connection);
}
Some(self.track(market, connections))
}
fn track(&mut self, market: Market, connections: Vec<usize>) -> BookCommand {
fn track(&mut self, market: M, connections: Vec<usize>) -> BookCommand<M> {
let tracked = Tracked {
market: market.clone(),
connections: connections.clone(),
load: 0,
moved_at: None,
};
self.markets.insert(market.condition_id.clone(), tracked);
self.markets.insert(market.id().to_string(), tracked);
BookCommand::Add(market, connections)
}
/// A random connection with room for `tokens` more and `load` more
/// A random connection with room for `symbols` more and `load` more
/// bytes/s, none of `except`; a new connection if none has. Random, since
/// the least loaded one would take every market of a batch created
/// together: the cluster a load average can't see.
fn pick(&mut self, tokens: usize, load: u64, except: &[usize]) -> usize {
fn pick(&mut self, symbols: usize, load: u64, except: &[usize]) -> usize {
let with_room: Vec<usize> = (0..self.connections.len())
.filter(|number| {
let connection = &self.connections[*number];
!except.contains(number)
&& connection.tokens + tokens <= CONNECTION_TOKENS
&& connection.symbols + symbols <= CONNECTION_SYMBOLS
&& connection.load + load <= CONNECTION_BYTES_PER_SECOND
})
.collect();
@@ -527,11 +534,11 @@ impl Tracker {
with_room[self.random.below(with_room.len())]
}
fn remove(&mut self, condition_id: &str) -> Option<BookCommand> {
let tracked = self.markets.remove(condition_id)?;
fn remove(&mut self, id: &str) -> Option<BookCommand<M>> {
let tracked = self.markets.remove(id)?;
for &connection in &tracked.connections {
let connection = &mut self.connections[connection];
connection.tokens -= tracked.market.tokens.len();
connection.symbols -= tracked.market.symbols().len();
connection.load -= tracked.load;
}
Some(BookCommand::Remove(tracked.market))
@@ -540,8 +547,8 @@ impl Tracker {
/// A book thread's loads per market, each weighing on every connection
/// that carries the market.
fn report(&mut self, loads: &Loads) {
for (condition_id, &load) in loads {
let Some(tracked) = self.markets.get_mut(condition_id) else {
for (id, &load) in loads {
let Some(tracked) = self.markets.get_mut(id) else {
continue;
};
for &connection in &tracked.connections {
@@ -556,7 +563,7 @@ impl Tracker {
/// byte budget, busiest connection first, each to a random connection with
/// room that doesn't carry the market, until it is under: at most
/// [`MOVES_PER_MINUTE`], and none moved in the last [`MOVED_STAYS`].
fn rebalance(&mut self, now: Instant) -> Vec<BookCommand> {
fn rebalance(&mut self, now: Instant) -> Vec<BookCommand<M>> {
let mut moves = Vec::new();
let mut over: Vec<usize> = (0..self.connections.len())
.filter(|&number| self.connections[number].load > CONNECTION_BYTES_PER_SECOND)
@@ -579,12 +586,12 @@ impl Tracker {
.collect();
busiest.sort_by_key(|&(_, load)| Reverse(load));
for (id, load) in busiest {
let size = self.markets[&id].market.tokens.len();
let size = self.markets[&id].market.symbols().len();
// Alone on its connection, a market would only take its load
// along; one with no load would lighten nothing.
if self.connections[from].load <= CONNECTION_BYTES_PER_SECOND
|| moves.len() == MOVES_PER_MINUTE
|| self.connections[from].tokens == size
|| self.connections[from].symbols == size
|| load == 0
{
break;
@@ -597,14 +604,14 @@ impl Tracker {
/// Moves the market's subscription on `from` to a random connection with
/// room that doesn't carry it yet: its Move.
fn move_off(&mut self, id: &str, from: usize, now: Instant) -> BookCommand {
fn move_off(&mut self, id: &str, from: usize, now: Instant) -> BookCommand<M> {
let tracked = &self.markets[id];
let (size, load) = (tracked.market.tokens.len(), tracked.load);
let (size, load) = (tracked.market.symbols().len(), tracked.load);
let carrying = tracked.connections.clone();
let to = self.pick(size, load, &carrying);
self.connections[from].tokens -= size;
self.connections[from].symbols -= size;
self.connections[from].load -= load;
self.connections[to].tokens += size;
self.connections[to].symbols += size;
self.connections[to].load += load;
let tracked = self.markets.get_mut(id).expect("tracked");
for connection in &mut tracked.connections {
@@ -623,7 +630,7 @@ impl Tracker {
/// A connection closed with 1013: on its second close within
/// [`SPLIT_WITHIN`], every second of its markets, in load order, moves off
/// at once, beyond rebalancing's limits.
fn closed(&mut self, connection: usize, now: Instant) -> Vec<BookCommand> {
fn closed(&mut self, connection: usize, now: Instant) -> Vec<BookCommand<M>> {
let Some(closed) = self.connections.get_mut(connection) else {
return Vec::new();
};
@@ -654,7 +661,7 @@ impl Tracker {
/// A market the book thread found quiet on `connection` again after a
/// resubscribe: its subscription there moves, beyond rebalancing's
/// limits. None if it no longer is there.
fn quiet(&mut self, id: &str, connection: usize, now: Instant) -> Option<BookCommand> {
fn quiet(&mut self, id: &str, connection: usize, now: Instant) -> Option<BookCommand<M>> {
let tracked = self.markets.get(id)?;
// A market moved lately stays, as for rebalancing: its last move may
// still be under way, and a second one would end it early.
@@ -724,7 +731,7 @@ impl Tracker {
/// Removes the markets known at the scan's start and not seen in it:
/// those added since were never at risk.
fn scan_done(&mut self) -> Vec<BookCommand> {
fn scan_done(&mut self) -> Vec<BookCommand<M>> {
let unseen = self.unseen.take().unwrap_or_default();
unseen.iter().filter_map(|id| self.remove(id)).collect()
}
@@ -762,12 +769,30 @@ impl Random {
mod tests {
use super::*;
fn market(id: usize) -> Market {
Market {
condition_id: format!("0x{id}"),
tokens: vec![format!("{id}a"), format!("{id}b")],
min_order_size: "5".to_string(),
neg_risk: false,
/// A market of two symbols.
#[derive(Debug, Clone)]
struct TestMarket {
id: String,
symbols: Vec<String>,
}
impl Market for TestMarket {
fn id(&self) -> &str {
&self.id
}
fn symbols(&self) -> &[String] {
&self.symbols
}
}
type Tracker = super::Tracker<TestMarket>;
type BookCommand = super::BookCommand<TestMarket>;
fn market(id: usize) -> TestMarket {
TestMarket {
id: format!("0x{id}"),
symbols: vec![format!("{id}a"), format!("{id}b")],
}
}
@@ -776,8 +801,8 @@ mod tests {
Tracker::new(2, 7)
}
fn tokens(tracker: &Tracker) -> Vec<usize> {
tracker.connections.iter().map(|one| one.tokens).collect()
fn symbols(tracker: &Tracker) -> Vec<usize> {
tracker.connections.iter().map(|one| one.symbols).collect()
}
fn connections(tracker: &Tracker, id: usize) -> Vec<usize> {
@@ -798,9 +823,7 @@ mod tests {
commands
.iter()
.map(|command| match command {
BookCommand::Move { market, from, to } => {
(market.condition_id.as_str(), *from, *to)
}
BookCommand::Move { market, from, to } => (market.id.as_str(), *from, *to),
other => panic!("not a move: {other:?}"),
})
.collect()
@@ -809,21 +832,21 @@ mod tests {
#[test]
fn the_startup_deal_is_even_and_gives_each_market_different_connections() {
let mut tracker = tracker();
// 10,000 markets, 20,000 tokens, twice: 9 connections of 4,500.
// 10,000 markets, 20,000 symbols, twice: 9 connections of 4,500.
let adds = tracker.deal((0..10_000).map(market).collect());
assert_eq!(adds.len(), 10_000);
assert_eq!(tracker.connections.len(), 9);
let tokens = tokens(&tracker);
assert_eq!(tokens.iter().sum::<usize>(), 40_000);
assert!(tokens.iter().all(|&held| held <= CONNECTION_TOKENS));
let symbols = symbols(&tracker);
assert_eq!(symbols.iter().sum::<usize>(), 40_000);
assert!(symbols.iter().all(|&held| held <= CONNECTION_SYMBOLS));
// Even, but for the markets whose second connection had to be the next.
assert!(tokens.iter().max().unwrap() - tokens.iter().min().unwrap() <= 100);
assert!(symbols.iter().max().unwrap() - symbols.iter().min().unwrap() <= 100);
for command in &adds {
let BookCommand::Add(market, on) = command else {
panic!("not an add: {command:?}");
};
assert_eq!(on.len(), 2);
assert_ne!(on[0], on[1], "{}", market.condition_id);
assert_ne!(on[0], on[1], "{}", market.id);
}
// Markets created together (neighbours in the list) are spread: the
// first 100 don't share one pair of connections.
@@ -836,7 +859,7 @@ mod tests {
let mut tracker = tracker();
let adds = tracker.deal(vec![market(0), market(1), market(0)]);
assert_eq!(adds.len(), 2);
assert_eq!(tokens(&tracker).iter().sum::<usize>(), 8);
assert_eq!(symbols(&tracker).iter().sum::<usize>(), 8);
}
#[test]
@@ -845,22 +868,22 @@ mod tests {
// Nothing open: its two connections open.
tracker.add(market(0)).unwrap();
assert_eq!(connections(&tracker, 0).len(), 2);
assert_eq!(tokens(&tracker), [2, 2]);
assert_eq!(symbols(&tracker), [2, 2]);
// Both fill together; then a third and a fourth open.
for id in 1..CONNECTION_TOKENS / 2 {
for id in 1..CONNECTION_SYMBOLS / 2 {
tracker.add(market(id)).unwrap();
}
assert_eq!(tokens(&tracker), [CONNECTION_TOKENS, CONNECTION_TOKENS]);
assert_eq!(symbols(&tracker), [CONNECTION_SYMBOLS, CONNECTION_SYMBOLS]);
tracker.add(market(9_999)).unwrap();
assert_eq!(
tokens(&tracker),
[CONNECTION_TOKENS, CONNECTION_TOKENS, 2, 2]
symbols(&tracker),
[CONNECTION_SYMBOLS, CONNECTION_SYMBOLS, 2, 2]
);
// Room on one connection only: the other subscription opens a new one.
tracker.remove("0x0").unwrap();
tracker.connections[1].tokens = CONNECTION_TOKENS;
tracker.connections[2].tokens = CONNECTION_TOKENS;
tracker.connections[3].tokens = CONNECTION_TOKENS;
tracker.connections[1].symbols = CONNECTION_SYMBOLS;
tracker.connections[2].symbols = CONNECTION_SYMBOLS;
tracker.connections[3].symbols = CONNECTION_SYMBOLS;
tracker.add(market(10_000)).unwrap();
let mut on = connections(&tracker, 10_000);
on.sort();
@@ -883,7 +906,7 @@ mod tests {
let mut tracker = tracker();
assert!(tracker.add(market(1)).is_some());
assert!(tracker.add(market(1)).is_none());
assert_eq!(tokens(&tracker), [2, 2]);
assert_eq!(symbols(&tracker), [2, 2]);
}
#[test]
@@ -900,7 +923,7 @@ mod tests {
// A market gone takes its load along.
tracker.remove("0x1");
assert_eq!(tracker.connections[1].load, 100);
assert_eq!(tokens(&tracker), [2, 2]);
assert_eq!(symbols(&tracker), [2, 2]);
}
#[test]
@@ -932,7 +955,7 @@ mod tests {
);
assert_eq!(moved.len(), 1 + 5);
assert!(tracker.connections.iter().all(|one| one.load <= budget));
assert_eq!(tokens(&tracker).iter().sum::<usize>(), 40);
assert_eq!(symbols(&tracker).iter().sum::<usize>(), 40);
// Under budget now: nothing more moves.
assert!(tracker.rebalance(now).is_empty());
}
@@ -968,13 +991,13 @@ mod tests {
tracker.connections.push(ConnectionState::default());
// One connection so far: put markets 2 and 3 on another.
for id in [2, 3] {
tracker.connections[0].tokens = CONNECTION_TOKENS;
tracker.connections[2].tokens = CONNECTION_TOKENS;
tracker.connections[0].symbols = CONNECTION_SYMBOLS;
tracker.connections[2].symbols = CONNECTION_SYMBOLS;
tracker.add(market(id));
}
tracker.connections[0].tokens = 4;
tracker.connections[2].tokens = 0;
assert_eq!(tokens(&tracker), [4, 4, 0]);
tracker.connections[0].symbols = 4;
tracker.connections[2].symbols = 0;
assert_eq!(symbols(&tracker), [4, 4, 0]);
let budget = CONNECTION_BYTES_PER_SECOND;
report(&mut tracker, &[(0, budget), (1, budget / 10)]);
report(&mut tracker, &[(2, budget / 2), (3, budget)]);
@@ -1014,7 +1037,7 @@ mod tests {
assert!(moves(&second).iter().all(|&(_, from, _)| from == 0));
// Ten minutes on, connection 0 holds one market: it is left alone.
assert!(tracker.rebalance(now + MOVED_STAYS).is_empty());
assert_eq!(tracker.connections[0].tokens, 2);
assert_eq!(tracker.connections[0].symbols, 2);
}
/// Over budget on its own, a market would only carry its load from
@@ -1037,7 +1060,7 @@ mod tests {
let now = Instant::now();
// Market 0 moves to a new connection, which then turns busy.
assert_eq!(moves(&tracker.rebalance(now)), [("0x0", 0, 1)]);
tracker.connections[0].tokens = CONNECTION_TOKENS;
tracker.connections[0].symbols = CONNECTION_SYMBOLS;
tracker.add(market(2));
assert_eq!(connections(&tracker, 2), [1]);
report(&mut tracker, &[(0, budget * 7 / 10), (2, budget * 6 / 10)]);
@@ -1127,7 +1150,7 @@ mod tests {
.iter()
.all(|&(_, from, to)| from == 0 && to != 0)
);
assert_eq!(tracker.connections[0].tokens, 10);
assert_eq!(tracker.connections[0].symbols, 10);
// The split connection is left alone for ten minutes.
let later = now + Duration::from_secs(6 * 60);
assert!(tracker.closed(0, later).is_empty());
@@ -1174,12 +1197,12 @@ mod tests {
assert!(matches!(
removed.as_slice(),
[BookCommand::Remove(market)] if market.condition_id == "0x2"
[BookCommand::Remove(market)] if market.id == "0x2"
));
let mut tracked: Vec<_> = tracker.markets.keys().cloned().collect();
tracked.sort();
assert_eq!(tracked, ["0x0", "0x9"]);
assert_eq!(tokens(&tracker), [4, 4]);
assert_eq!(symbols(&tracker), [4, 4]);
}
#[test]
+397
View File
@@ -0,0 +1,397 @@
//! A connection: reads the exchange's websocket, stamps each frame's receive
//! time, decodes it and sends each event to the book thread of its market;
//! and sends the exchange what the book threads ask for (subscriptions,
//! resyncs). It reconnects when it drops, falls silent, loses a message, or is
//! recycled.
use std::collections::BTreeSet;
use std::io::ErrorKind;
use std::net::{TcpStream, ToSocketAddrs};
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread;
use std::time::{Duration, Instant};
use chrono::{DateTime, Utc};
use crossbeam::channel::{Receiver, Sender};
use tungstenite::protocol::frame::coding::CloseCode;
use tungstenite::stream::MaybeTlsStream;
use tungstenite::{Message, WebSocket};
use super::{Arrival, Event, Exchange, Received, Session, Undecoded, book_thread};
/// For the TCP connect and the TLS + websocket handshake, and for every send.
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
/// How often the loop wakes when no frame comes: tungstenite has no `split`,
/// so the reads and the writes below (keepalive, commands, close) share this
/// thread.
const READ_TIMEOUT: Duration = Duration::from_millis(200);
const PING_EVERY: Duration = Duration::from_secs(10);
const RECONNECT_AFTER: Duration = Duration::from_secs(1);
const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
/// Nothing received this long, not even a keepalive's answer or a ping: the
/// connection is silent, and reconnects.
pub const SILENT_AFTER: Duration = Duration::from_secs(30);
/// What a book thread asks a connection for.
#[derive(Debug, PartialEq, Eq)]
pub enum Command {
Subscribe(Vec<String>),
Unsubscribe(Vec<String>),
/// Unsubscribe and subscribe again: the server then sends a fresh book.
/// A resync asks for that book; a quiet subscription only for the new
/// subscription.
Resubscribe(String),
/// Close and reconnect, subscribing every symbol again.
Recycle,
}
/// What a connection tells the orchestrator.
#[derive(Debug, PartialEq, Eq)]
pub enum Notice {
/// Connected, its symbols subscribed.
Up(usize),
Down(usize),
/// Closed by the server with 1013: it couldn't send to us fast enough.
TooSlow(usize),
/// Closed by us: nothing came for [`SILENT_AFTER`].
Silent(usize),
}
type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
/// Runs until `stop` is set or a book thread is gone, reconnecting (and
/// subscribing its symbols again) whenever the connection drops, falls silent
/// for `silent_after`, loses a message or is recycled, and telling every book
/// thread and the orchestrator (`notices`) when it comes up and goes down.
/// `book_threads` holds every book thread's channel, by number.
pub fn run<E: Exchange>(
exchange: &E,
number: usize,
book_threads: &[Sender<Received<E::Event>>],
commands: &Receiver<Command>,
notices: &Sender<Notice>,
silent_after: Duration,
stop: &AtomicBool,
) {
let name = E::NAME;
let tell = |received: fn(usize) -> Received<E::Event>, notice: fn(usize) -> Notice| {
for book_thread in book_threads {
_ = book_thread.send(received(number));
}
_ = notices.send(notice(number));
};
let mut symbols = BTreeSet::new();
while !stop.load(Ordering::Relaxed) {
// Asked for while disconnected: the subscribe on connect covers them,
// and its books are fresh, so resyncs have nothing left to do.
let mut session = exchange.session();
for command in commands.try_iter() {
_ = apply(&mut session, &mut symbols, &command);
}
match connect(exchange, &mut session, &symbols) {
Ok(mut socket) => {
log!(
"{name}: connection {number}: connected, {} symbols",
symbols.len()
);
tell(Received::Up, Notice::Up);
let ended = read(
&mut socket,
&mut session,
number,
&mut symbols,
book_threads,
commands,
notices,
silent_after,
stop,
name,
);
if !matches!(ended, Ok(Ended::Stopped)) {
tell(Received::Down, Notice::Down);
}
match ended {
Ok(Ended::Stopped) => {
return close(&mut socket, &mut session, number, book_threads, name);
}
Ok(Ended::Recycled) => log!("{name}: connection {number}: recycled"),
Ok(Ended::BookThreadGone) => {
// Only a panic ends a book thread before us: its
// markets would go unserved, so the run stops.
log!("{name}: connection {number}: a book thread is gone, stopping");
return stop.store(true, Ordering::Relaxed);
}
Err(error) => {
log!("{name}: connection {number}: connection lost: {error}")
}
}
}
Err(error) => log!("{name}: connection {number}: connect failed: {error}"),
}
thread::sleep(RECONNECT_AFTER);
}
}
enum Ended {
Stopped,
BookThreadGone,
Recycled,
}
/// Updates the symbol set, and returns the frames that carry `command` out
/// on an open connection: none for a resync of a symbol not in the set.
#[must_use]
pub(crate) fn apply<S: Session>(
session: &mut S,
symbols: &mut BTreeSet<String>,
command: &Command,
) -> Vec<String> {
match command {
Command::Subscribe(added) => {
symbols.extend(added.iter().cloned());
session.subscribe(&added.iter().collect::<Vec<_>>())
}
Command::Unsubscribe(removed) => {
removed.iter().for_each(|symbol| _ = symbols.remove(symbol));
session.unsubscribe(&removed.iter().collect::<Vec<_>>())
}
// Subscribing a symbol already subscribed sends no fresh book:
// unsubscribing first makes the server send one.
Command::Resubscribe(symbol) if symbols.contains(symbol) => {
[session.unsubscribe(&[symbol]), session.subscribe(&[symbol])].concat()
}
Command::Resubscribe(_) => Vec::new(),
// The read loop acts on it; disconnected, the reconnect does.
Command::Recycle => Vec::new(),
}
}
fn connect<E: Exchange>(
exchange: &E,
session: &mut E::Session,
symbols: &BTreeSet<String>,
) -> anyhow::Result<Socket> {
let request = exchange.request()?;
let uri = request.uri().clone();
let host = uri
.host()
.ok_or_else(|| anyhow::anyhow!("{uri} has no host"))?;
// `wss://` on TLS's port; the tests name theirs.
let port = uri.port_u16().unwrap_or(443);
// Timed, unlike `tungstenite::connect`, so a stalled connect or handshake
// can't keep SIGINT waiting. The name lookup itself has no timeout.
let address = (host, port)
.to_socket_addrs()?
.next()
.ok_or_else(|| anyhow::anyhow!("{host} has no address"))?;
let stream = TcpStream::connect_timeout(&address, CONNECT_TIMEOUT)?;
stream.set_read_timeout(Some(CONNECT_TIMEOUT))?;
stream.set_write_timeout(Some(CONNECT_TIMEOUT))?;
let (mut socket, _response) = tungstenite::client_tls(request, stream).map_err(|error| {
// A refusal's status says why: 401 for a key the exchange refuses.
match error {
tungstenite::HandshakeError::Failure(tungstenite::Error::Http(response)) => {
anyhow::anyhow!("handshake refused: {}", response.status())
}
error => anyhow::anyhow!("handshake failed: {error}"),
}
})?;
let stream = match socket.get_ref() {
MaybeTlsStream::Plain(stream) => stream,
MaybeTlsStream::Rustls(stream) => stream.get_ref(),
_ => anyhow::bail!("unexpected TLS stream"),
};
stream.set_read_timeout(Some(READ_TIMEOUT))?;
for message in session.opening() {
socket.send(Message::text(message))?;
}
if !symbols.is_empty() {
for message in session.subscribe(&symbols.iter().collect::<Vec<_>>()) {
socket.send(Message::text(message))?;
}
}
Ok(socket)
}
/// What [`route`] made of a frame.
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum Routed {
/// Its events, if any, went to their book threads.
Sent,
BookThreadGone,
/// The connection lost a message: it must reconnect.
Lost(String),
}
/// Decodes a frame and sends each event to the book thread of its market.
pub(crate) fn route<S: Session>(
session: &mut S,
received_at: DateTime<Utc>,
text: &str,
number: usize,
book_threads: &[Sender<Received<S::Event>>],
name: &str,
) -> Routed
where
S::Event: Event,
{
let events = match session.decode(text) {
Ok(events) => events,
Err(Undecoded::Unreadable(error)) => {
log!("{name}: unreadable frame ({error}): {}", truncate(text));
return Routed::Sent;
}
Err(Undecoded::Lost(why)) => return Routed::Lost(why),
};
let bytes = (text.len() / events.len().max(1)) as u64;
for event in events {
// An event of no market (none we ask for) has no book thread.
let Some(thread) = event.market().map(book_thread) else {
continue;
};
let arrival = Arrival {
received_at,
connection: number,
bytes,
event,
};
if book_threads[thread].send(Received::Event(arrival)).is_err() {
return Routed::BookThreadGone;
}
}
Routed::Sent
}
fn truncate(text: &str) -> &str {
text.char_indices()
.nth(200)
.map_or(text, |(end, _)| &text[..end])
}
// The connection's whole context, each a different part of it.
#[allow(clippy::too_many_arguments)]
fn read<S: Session>(
socket: &mut Socket,
session: &mut S,
number: usize,
symbols: &mut BTreeSet<String>,
book_threads: &[Sender<Received<S::Event>>],
commands: &Receiver<Command>,
notices: &Sender<Notice>,
silent_after: Duration,
stop: &AtomicBool,
name: &str,
) -> anyhow::Result<Ended>
where
S::Event: Event,
{
let mut last_ping = Instant::now();
let mut last_heard = Instant::now();
loop {
let message = socket.read();
// Stamp as soon as the frame is off the wire, before any processing.
let received_at = Utc::now();
if message.is_ok() {
last_heard = Instant::now();
}
match message {
Ok(Message::Text(text)) => {
match route(
session,
received_at,
text.as_str(),
number,
book_threads,
name,
) {
Routed::Sent => {}
Routed::BookThreadGone => return Ok(Ended::BookThreadGone),
Routed::Lost(why) => anyhow::bail!("lost a message: {why}"),
}
}
Ok(Message::Close(frame)) => {
// 1013: the server couldn't send to us fast enough. The
// orchestrator splits a connection closed so twice.
if frame
.as_ref()
.is_some_and(|frame| frame.code == CloseCode::Again)
{
_ = notices.send(Notice::TooSlow(number));
}
anyhow::bail!("closed by the server: {frame:?}")
}
Ok(_) => {}
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
Err(error) => return Err(error.into()),
}
if stop.load(Ordering::Relaxed) {
return Ok(Ended::Stopped);
}
if last_heard.elapsed() >= silent_after {
_ = notices.send(Notice::Silent(number));
anyhow::bail!("silent for {} s", silent_after.as_secs_f64());
}
if last_ping.elapsed() >= PING_EVERY {
if let Some(keepalive) = session.keepalive() {
socket.send(Message::text(keepalive))?;
}
last_ping = Instant::now();
}
for command in commands.try_iter() {
if command == Command::Recycle {
// Not waited on: whatever still comes would arrive from a
// connection already down for the book threads. The commands
// behind it are applied before the reconnect.
_ = socket.close(None);
_ = socket.flush();
return Ok(Ended::Recycled);
}
for message in apply(session, symbols, &command) {
socket.send(Message::text(message))?;
}
}
}
}
/// Closes the connection, forwarding what still arrives until the server
/// confirms or [`CLOSE_TIMEOUT`] passes.
fn close<S: Session>(
socket: &mut Socket,
session: &mut S,
number: usize,
book_threads: &[Sender<Received<S::Event>>],
name: &str,
) where
S::Event: Event,
{
if let Err(error) = socket.close(None) {
return log!("{name}: connection {number}: close failed: {error}");
}
let deadline = Instant::now() + CLOSE_TIMEOUT;
while Instant::now() < deadline {
let message = socket.read();
let received_at = Utc::now();
match message {
Ok(Message::Text(text)) => {
_ = route(
session,
received_at,
text.as_str(),
number,
book_threads,
name,
)
}
Ok(_) => {}
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
// ConnectionClosed: the close handshake is done.
Err(_) => break,
}
}
}
/// A read timeout shows up as `WouldBlock` on macOS and `TimedOut` elsewhere;
/// tungstenite keeps a partly read frame, so the next read resumes it.
fn timed_out(error: &std::io::Error) -> bool {
matches!(error.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut)
}
+823
View File
@@ -0,0 +1,823 @@
//! Kalshi's rule for a book thread: first one wins by clock and rank
//! (roadmap/06). A market's changes carry its clock, the same on every
//! connection, and come in the same order on each; several can share a
//! clock, so a change is named by its clock and its rank among the changes of
//! that clock. The book stands at the position of the last change it holds;
//! each connection's stream at the position of the last change it delivered,
//! counted from the book it last delivered. A change passing the book's
//! position is the next one; any other is a duplicate. Nothing is held.
//! A connection that lost a message reconnects (its `seq`, in
//! `src/kalshi/mod.rs`); the book then comes fresh.
use std::collections::BTreeMap;
use chrono::{DateTime, Utc};
use engineering_challenge::book::Book;
use engineering_challenge::cause::{self, Cause};
use super::Market;
use super::wire::{BookMessage, ChangeMessage, Event};
use crate::core::feed::{self, Effect, Minute, Rule, Shell, is_up};
use crate::core::socket::Command;
/// A clock, and a rank among the changes of that clock.
type Position = (u64, u32);
/// A book's rank: it holds every change of its clock.
const BOOK: u32 = u32::MAX;
/// A symbol's state for the rule.
#[derive(Debug, Default)]
pub struct Clocked {
/// Where the book stands; none before its first book.
at: Option<Position>,
/// By connection: where its stream stands, since the book it last
/// delivered. A connection with none has delivered no book since it came
/// up or subscribed: its changes have nowhere to go.
on: BTreeMap<usize, Position>,
/// The connection a fresh book was asked from: its answer replaces ours.
asked: Option<usize>,
}
type Symbol = feed::Symbol<Clocked>;
/// What the minute line counts, since the start.
#[derive(Debug, Default)]
struct Counts {
written: u64,
duplicates: u64,
/// Changes from a connection that had delivered no book of the symbol.
unplaced: u64,
/// Fresh books after a symbol's first.
books: u64,
/// Of them, books at or past ours that differed: written as a Snapshot.
losses: u64,
/// Changes from a connection past the book, after the book went back to
/// an answer: dropped until the book reaches it.
ahead: u64,
/// Changes that took a level below zero: our book was wrong.
below_zero: u64,
resyncs: u64,
}
/// Kalshi's rule, for one book thread.
#[derive(Default)]
pub struct Clock {
counts: Counts,
}
impl Clock {
fn on_book(
&mut self,
shell: &mut Shell<Clocked>,
received_at: DateTime<Utc>,
connection: usize,
message: BookMessage,
effects: &mut Vec<Effect>,
) {
let ticker = message.ticker;
let Some(symbol) = shell.symbols.get_mut(&ticker) else {
return;
};
if !symbol.connections.contains(&connection) {
return;
}
let position = (message.clock, BOOK);
symbol.rule.on.insert(connection, position);
let book = Book {
bids: message.bids.into_iter().collect(),
asks: message.asks.into_iter().collect(),
};
let asked = symbol.rule.asked == Some(connection);
let cause = match symbol.rule.at {
None => None,
Some(at) => {
self.counts.books += 1;
// Behind ours, unasked: it only says where its connection is.
if position < at && !asked {
return;
}
if book == symbol.book {
symbol.rule.at = Some(at.max(position));
if asked {
symbol.rule.asked = None;
}
return;
}
Some(at)
}
};
// The first book, an answer, or a book at or past ours that differs:
// ours is replaced. Which connection missed what can't be told: the
// other may only be behind.
symbol.book = book;
symbol.exchange_timestamp = message.sent_at;
symbol.rule.at = Some(position);
// A first book answers whichever connection it was asked from.
if asked || cause.is_none() {
symbol.rule.asked = None;
}
effects.push(Effect::Write(symbol.snapshot(&ticker, received_at)));
if let Some(ours) = cause {
self.counts.losses += 1;
let detail = format!(" (its clock {}, ours {})", message.clock, ours.0);
log!(
"kalshi: {}",
cause::line(&ticker, received_at, Cause::Loss, &detail)
);
}
}
fn on_change(
&mut self,
shell: &mut Shell<Clocked>,
received_at: DateTime<Utc>,
connection: usize,
change: ChangeMessage,
effects: &mut Vec<Effect>,
) {
let ticker = change.ticker;
shell.count_received(&ticker, connection);
let Some(symbol) = shell.symbols.get_mut(&ticker) else {
return;
};
let Some(on) = symbol.rule.on.get_mut(&connection) else {
self.counts.unplaced += 1;
return;
};
let before = *on;
*on = match before {
(clock, rank) if clock == change.clock => (clock, rank.saturating_add(1)),
_ => (change.clock, 1),
};
let position = *on;
match symbol.rule.at {
Some(at) if position <= at => {
self.counts.duplicates += 1;
return;
}
// Its stream was already past the book, which went back to an
// answer: the changes between are still to come from the
// connection that answered, and this one would skip them.
Some(at) if before > at => {
self.counts.ahead += 1;
return;
}
_ => {}
}
symbol.rule.at = Some(position);
let quantity = symbol.book.add(change.side, change.price, change.delta);
let quantity = if quantity < 0 {
// A level can't hold less than nothing: ours was wrong.
self.counts.below_zero += 1;
log!(
"kalshi: {ticker}: a change takes a level below zero, resync on connection \
{connection}"
);
if symbol.rule.asked.is_none() {
Self::ask_fresh_book(&ticker, symbol, connection, effects);
self.counts.resyncs += 1;
}
0
} else {
quantity
};
symbol.exchange_timestamp = change.at;
self.counts.written += 1;
effects.push(Effect::Write(symbol.delta(
&ticker,
change.side,
change.price,
quantity,
received_at,
change.at,
)));
}
/// Asks `connection` for a fresh book: unsubscribe and subscribe again.
fn ask_fresh_book(
ticker: &str,
symbol: &mut Symbol,
connection: usize,
effects: &mut Vec<Effect>,
) {
symbol.rule.asked = Some(connection);
effects.push(Effect::Send(
connection,
Command::Resubscribe(ticker.to_string()),
));
}
}
impl Rule for Clock {
type Market = Market;
type Event = Event;
type Symbol = Clocked;
fn symbol(&self, _market: &Market) -> Clocked {
Clocked::default()
}
fn on_event(
&mut self,
shell: &mut Shell<Clocked>,
connection: usize,
received_at: DateTime<Utc>,
event: Event,
effects: &mut Vec<Effect>,
) {
match event {
Event::Book(book) => self.on_book(shell, received_at, connection, book, effects),
Event::Change(change) => {
self.on_change(shell, received_at, connection, change, effects)
}
}
}
/// Its stream there counts no more; a fresh book asked from it is asked
/// from another.
fn on_leave(
&mut self,
up: &[bool],
ticker: &str,
symbol: &mut Symbol,
connection: usize,
effects: &mut Vec<Effect>,
) {
symbol.rule.on.remove(&connection);
if symbol.rule.asked == Some(connection)
&& let Some(other) = symbol.other_up(up, connection)
{
Self::ask_fresh_book(ticker, symbol, other, effects);
self.counts.resyncs += 1;
}
}
/// Its streams count no more: back up, it delivers fresh books, and one
/// asked from it is its answer.
fn on_down(
&mut self,
shell: &mut Shell<Clocked>,
connection: usize,
_effects: &mut Vec<Effect>,
) {
for symbol in shell.symbols.values_mut() {
symbol.rule.on.remove(&connection);
}
}
fn on_timer(
&mut self,
_shell: &mut Shell<Clocked>,
_now: DateTime<Utc>,
_effects: &mut Vec<Effect>,
) {
}
fn wakes_soon(&self) -> bool {
false
}
fn on_removed(&mut self, _ticker: &str) {}
/// While no connection delivers it, the book may be missing changes: no
/// Snapshot of it until one does.
fn has_book(symbol: &Clocked) -> bool {
symbol.at.is_some() && !symbol.on.is_empty()
}
fn may_be_quiet(symbol: &Clocked) -> bool {
symbol.at.is_some() && symbol.asked.is_none()
}
/// Asks again for a fresh book still not come, and for the book of a
/// symbol no connection delivers: its first, or one a reconnect didn't
/// bring.
fn on_minute(
&mut self,
up: &[bool],
ticker: &str,
symbol: &mut Symbol,
_now: DateTime<Utc>,
effects: &mut Vec<Effect>,
) {
let again = symbol.rule.asked.or_else(|| {
let up = |&connection: &usize| is_up(up, connection);
symbol
.rule
.on
.is_empty()
.then(|| symbol.connections.iter().copied().find(up))
.flatten()
});
if let Some(connection) = again {
Self::ask_fresh_book(ticker, symbol, connection, effects);
self.counts.resyncs += 1;
}
}
fn log_minute(&mut self, minute: &Minute) {
let counts = &self.counts;
log!(
"kalshi: book thread {}: {} symbols; so far {} changes written, {} duplicates, \
{} before their connection's book, {} ahead of the book, {} fresh books \
({} losses), {} below zero, \
{} resyncs, {} quiet subscriptions resubscribed, {} markets reported quiet again; \
this minute {} changes received, at most {} events waited",
minute.number,
minute.symbols,
counts.written,
counts.duplicates,
counts.unplaced,
counts.ahead,
counts.books,
counts.losses,
counts.below_zero,
counts.resyncs,
minute.quiet,
minute.quiet_reported,
minute.received,
minute.peak_waiting
);
}
}
#[cfg(test)]
mod tests {
use std::collections::HashMap;
use chrono::TimeDelta;
use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain;
use serde::Deserialize;
use super::*;
use crate::core::{Arrival, Received, feed::next_minute};
use crate::kalshi::wire;
use engineering_challenge::Side;
use engineering_challenge::book::Output;
type Feed = feed::Feed<Clock>;
fn at(millis: i64) -> DateTime<Utc> {
DateTime::from_timestamp_millis(1_794_422_000_000 + millis).unwrap()
}
fn feed_of(tickers: &[&str], connections: &[usize]) -> Feed {
let mut feed = Feed::new("kalshi", 0, Clock::default());
for ticker in tickers {
feed.add_market(
&Market {
ticker: ticker.to_string(),
},
connections,
at(0),
);
}
feed
}
fn book(ticker: &str, clock: u64, bids: &[(i64, i64)], asks: &[(i64, i64)]) -> Event {
Event::Book(BookMessage {
ticker: ticker.into(),
clock,
bids: bids.to_vec(),
asks: asks.to_vec(),
sent_at: at(0),
})
}
fn change(ticker: &str, clock: u64, side: Side, price: i64, delta: i64) -> Event {
Event::Change(ChangeMessage {
ticker: ticker.into(),
clock,
side,
price,
delta,
at: at(0),
})
}
impl Feed {
fn from(&mut self, connection: usize, millis: i64, event: Event) -> Vec<Effect> {
self.on_event(Arrival {
received_at: at(millis),
connection,
bytes: 0,
event,
})
}
}
fn deltas(effects: &[Effect]) -> Vec<&PriceLevelUpdateDomain> {
effects
.iter()
.filter_map(|effect| match effect {
Effect::Write(Output::Delta(delta)) => Some(delta),
_ => None,
})
.collect()
}
fn snapshots(effects: &[Effect]) -> usize {
effects
.iter()
.filter(|effect| matches!(effect, Effect::Write(Output::Snapshot(_))))
.count()
}
fn resyncs(effects: &[Effect]) -> Vec<(usize, &str)> {
effects
.iter()
.filter_map(|effect| match effect {
Effect::Send(connection, Command::Resubscribe(ticker)) => {
Some((*connection, ticker.as_str()))
}
_ => None,
})
.collect()
}
/// A market with the same book from both connections.
fn both_on(ticker: &str) -> Feed {
let mut feed = feed_of(&[ticker], &[0, 1]);
let first = feed.from(0, 1, book(ticker, 10, &[(400_000, 5_000_000)], &[]));
assert_eq!(snapshots(&first), 1);
let second = feed.from(1, 2, book(ticker, 10, &[(400_000, 5_000_000)], &[]));
assert!(second.is_empty(), "{second:?}");
feed
}
#[test]
fn a_change_is_written_once_from_whichever_delivers_it_first() {
let mut feed = both_on("M");
let first = feed.from(1, 5, change("M", 11, Side::Buy, 400_000, 2_000_000));
let written = deltas(&first);
assert_eq!(written.len(), 1);
// The absolute quantity, at the first copy's receive time.
assert_eq!(written[0].quantity, 7_000_000);
assert_eq!(written[0].backtest_timestamp, at(5));
assert!(
deltas(&feed.from(0, 6, change("M", 11, Side::Buy, 400_000, 2_000_000))).is_empty()
);
assert_eq!(feed.rule.counts.duplicates, 1);
}
#[test]
fn every_change_of_one_clock_is_applied_once() {
let mut feed = both_on("M");
// Two identical changes share a clock: both are real.
let twice = [
change("M", 11, Side::Sell, 600_000, 1_000_000),
change("M", 11, Side::Sell, 600_000, 1_000_000),
];
let mut written = Vec::new();
for event in twice {
written.extend(feed.from(0, 5, event));
}
let quantities: Vec<i64> = deltas(&written)
.iter()
.map(|delta| delta.quantity)
.collect();
assert_eq!(quantities, [1_000_000, 2_000_000]);
for event in [
change("M", 11, Side::Sell, 600_000, 1_000_000),
change("M", 11, Side::Sell, 600_000, 1_000_000),
] {
assert!(deltas(&feed.from(1, 6, event)).is_empty());
}
// The second connection's third change of that clock is new.
let third = feed.from(1, 7, change("M", 11, Side::Sell, 600_000, 1_000_000));
assert_eq!(deltas(&third)[0].quantity, 3_000_000);
}
#[test]
fn a_connection_counts_for_a_symbol_only_from_its_book() {
let mut feed = feed_of(&["M"], &[0, 1]);
feed.from(0, 1, book("M", 10, &[], &[]));
// Connection 1 hasn't delivered its book: its change has no place.
assert!(feed.from(1, 2, change("M", 11, Side::Buy, 1, 1)).is_empty());
assert_eq!(feed.rule.counts.unplaced, 1);
// Down and up again, connection 0 too waits for its book.
feed.on_received(Received::Down(0));
feed.on_received(Received::Up(0));
assert!(feed.from(0, 3, change("M", 12, Side::Buy, 1, 1)).is_empty());
assert_eq!(feed.rule.counts.unplaced, 2);
}
#[test]
fn a_fresh_book_replaces_ours_only_at_or_past_it_and_different() {
let mut feed = both_on("M");
feed.from(0, 3, change("M", 11, Side::Buy, 400_000, 1_000_000));
// Older than ours: connection 1 is only behind.
assert!(
feed.from(1, 4, book("M", 10, &[(400_000, 5_000_000)], &[]))
.is_empty()
);
// At our clock and the same: nothing.
assert!(
feed.from(1, 5, book("M", 11, &[(400_000, 6_000_000)], &[]))
.is_empty()
);
// Past ours and the same: nothing either.
assert!(
feed.from(1, 6, book("M", 15, &[(400_000, 6_000_000)], &[]))
.is_empty()
);
assert_eq!(feed.rule.counts.losses, 0);
// Past ours and different: it replaces ours, as a Snapshot.
let different = feed.from(1, 7, book("M", 20, &[(400_000, 1_000_000)], &[]));
assert_eq!(snapshots(&different), 1);
assert_eq!(feed.rule.counts.losses, 1);
assert_eq!(feed.shell.symbols["M"].book.bids[&400_000], 1_000_000);
// Its changes up to that book are duplicates now.
assert!(deltas(&feed.from(0, 8, change("M", 16, Side::Buy, 400_000, 1))).is_empty());
}
#[test]
fn a_level_below_zero_resyncs_and_the_answer_replaces_ours() {
let mut feed = both_on("M");
let effects = feed.from(0, 3, change("M", 11, Side::Buy, 400_000, -6_000_000));
assert_eq!(deltas(&effects)[0].quantity, 0);
assert_eq!(resyncs(&effects), [(0, "M")]);
assert_eq!(feed.rule.counts.below_zero, 1);
assert!(!Clock::may_be_quiet(&feed.shell.symbols["M"].rule));
// The answer replaces ours even behind it, and differing is a loss.
let answer = feed.from(0, 4, book("M", 9, &[(400_000, 9_000_000)], &[]));
assert_eq!(snapshots(&answer), 1);
assert_eq!(feed.shell.symbols["M"].book.bids[&400_000], 9_000_000);
assert_eq!(feed.shell.symbols["M"].rule.asked, None);
// The stream that answered goes on from its book.
let after = feed.from(0, 5, change("M", 10, Side::Buy, 400_000, 1_000_000));
assert_eq!(deltas(&after)[0].quantity, 10_000_000);
}
/// The answer to a resync can be behind what the other connection has
/// delivered: the book goes back to it, and only a stream that continues
/// from there is taken until the other fits again.
#[test]
fn after_an_answer_behind_ours_a_connection_ahead_waits_for_the_book() {
let mut feed = both_on("M");
feed.from(0, 3, change("M", 11, Side::Buy, 400_000, -6_000_000));
feed.from(1, 4, change("M", 11, Side::Buy, 400_000, -6_000_000));
feed.from(1, 5, change("M", 12, Side::Buy, 400_000, 1_000_000));
// Connection 0's answer holds clock 11, not yet 12.
let answer = feed.from(0, 6, book("M", 11, &[(400_000, 2_000_000)], &[]));
assert_eq!(snapshots(&answer), 1);
// Connection 1 is past it: its next change would skip clock 12.
let ahead = feed.from(1, 7, change("M", 13, Side::Buy, 400_000, 1_000_000));
assert!(deltas(&ahead).is_empty(), "{ahead:?}");
let twelve = feed.from(0, 8, change("M", 12, Side::Buy, 400_000, 1_000_000));
assert_eq!(deltas(&twelve)[0].quantity, 3_000_000);
let thirteen = feed.from(0, 9, change("M", 13, Side::Buy, 400_000, 1_000_000));
assert_eq!(deltas(&thirteen)[0].quantity, 4_000_000);
// The book has caught up with connection 1: it counts again.
let fourteen = feed.from(1, 10, change("M", 14, Side::Buy, 400_000, 1_000_000));
assert_eq!(deltas(&fourteen)[0].quantity, 5_000_000);
assert_eq!(feed.rule.counts.ahead, 1);
}
#[test]
fn a_first_book_asked_from_one_connection_can_come_from_the_other() {
let mut feed = feed_of(&["N"], &[0, 1]);
feed.on_received(Received::Up(0));
feed.from(0, 5, change("N", 1, Side::Buy, 1, 1));
assert_eq!(resyncs(&feed.tick(next_minute(at(10)))), [(0, "N")]);
let first = feed.from(1, 61_000, book("N", 20, &[(400_000, 5_000_000)], &[]));
assert_eq!(snapshots(&first), 1);
// Connection 0's answer, older and different, is only where it stands.
let answer = feed.from(0, 61_001, book("N", 19, &[(400_000, 4_000_000)], &[]));
assert!(answer.is_empty(), "{answer:?}");
assert_eq!(feed.rule.counts.losses, 0);
assert_eq!(feed.shell.symbols["N"].book.bids[&400_000], 5_000_000);
}
#[test]
fn a_symbol_no_connection_delivers_is_asked_for_at_the_minute() {
let mut feed = both_on("M");
feed.on_received(Received::Down(0));
feed.on_received(Received::Down(1));
feed.on_received(Received::Up(0));
feed.on_received(Received::Up(1));
// Its books didn't come with the reconnects.
assert_eq!(resyncs(&feed.tick(next_minute(at(10)))), [(0, "M")]);
}
#[test]
fn a_resync_asked_from_a_connection_that_leaves_is_asked_from_another() {
let mut feed = both_on("M");
feed.from(0, 3, change("M", 11, Side::Buy, 400_000, -6_000_000));
let market = Market { ticker: "M".into() };
let moved = feed.move_market(&market, 0, 2, at(4));
assert!(resyncs(&moved).is_empty());
// Connection 2's book ends the move: connection 0 leaves.
let ended = feed.from(2, 5, book("M", 12, &[], &[]));
assert!(ended.iter().any(|effect| matches!(
effect,
Effect::Send(0, Command::Unsubscribe(tokens)) if tokens == &["M".to_string()]
)));
// It came from connection 2, not the one asked: not the answer.
assert_eq!(resyncs(&ended), [(1, "M")]);
}
#[test]
fn a_fresh_book_still_not_come_is_asked_again_at_the_minute() {
let mut feed = both_on("M");
feed.from(0, 3, change("M", 11, Side::Buy, 400_000, -6_000_000));
let minute = feed.tick(next_minute(at(10)));
assert_eq!(resyncs(&minute), [(0, "M")]);
// A symbol with no book at all is asked for one too.
let mut none = feed_of(&["N"], &[0]);
none.on_received(Received::Up(0));
// An event starts the book thread's minutes.
none.from(0, 5, change("N", 1, Side::Buy, 1, 1));
assert_eq!(resyncs(&none.tick(next_minute(at(10)))), [(0, "N")]);
// One in order isn't.
let mut fine = both_on("F");
assert!(resyncs(&fine.tick(next_minute(at(10)))).is_empty());
}
#[test]
fn no_minute_snapshot_while_no_connection_delivers_the_symbol() {
let mut feed = both_on("M");
let minute = next_minute(at(10));
assert_eq!(snapshots(&feed.tick(minute)), 1);
feed.on_received(Received::Down(0));
feed.on_received(Received::Down(1));
assert_eq!(snapshots(&feed.tick(minute + TimeDelta::minutes(1))), 0);
// Back, its book compared: the same, so no Snapshot until the minute.
assert!(
feed.from(1, 70_000, book("M", 30, &[(400_000, 5_000_000)], &[]))
.is_empty()
);
assert_eq!(snapshots(&feed.tick(minute + TimeDelta::minutes(2))), 1);
}
/// The demo's recording on two connections (tests/fixtures/README.md).
fn recording() -> Vec<(DateTime<Utc>, usize, Event)> {
#[derive(Deserialize)]
struct Line {
received_at_us: i64,
connection: usize,
text: String,
}
let path = concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/kalshi-frames-two.jsonl"
);
std::fs::read_to_string(path)
.unwrap()
.lines()
.filter_map(|line| {
let line: Line = serde_json::from_str(line).unwrap();
let frame: wire::Frame = serde_json::from_str(&line.text).unwrap();
let event = wire::event(&frame).unwrap()?;
let received_at = DateTime::from_timestamp_micros(line.received_at_us).unwrap();
Some((received_at, line.connection, event))
})
.collect()
}
fn tickers(recording: &[(DateTime<Utc>, usize, Event)]) -> Vec<String> {
let mut tickers: Vec<String> = recording
.iter()
.map(|(_, _, event)| crate::core::Event::market(event).unwrap().to_string())
.collect();
tickers.sort();
tickers.dedup();
tickers
}
/// Each symbol's Deltas, as (side, price, quantity, receive time).
type Written = HashMap<String, Vec<(Side, i64, i64, DateTime<Utc>)>>;
/// Replays the connections in `on`.
fn replay(on: &[usize]) -> (Feed, Written) {
let recording = recording();
let tickers = tickers(&recording);
let tickers: Vec<&str> = tickers.iter().map(String::as_str).collect();
let mut feed = feed_of(&tickers, on);
let mut written: HashMap<_, Vec<_>> = HashMap::new();
for (received_at, connection, event) in recording {
if !on.contains(&connection) {
continue;
}
let arrival = Arrival {
received_at,
connection,
bytes: 0,
event,
};
for delta in deltas(&feed.on_event(arrival)) {
written.entry(delta.symbol.clone()).or_default().push((
delta.side,
delta.price_level,
delta.quantity,
delta.backtest_timestamp,
));
}
}
(feed, written)
}
/// Connection 0 asked for every book again every 3 s: each, compared
/// with ours folded from the changes, was the same.
#[test]
fn the_recorded_changes_fold_into_the_exchanges_books() {
let (feed, written) = replay(&[0]);
let counts = &feed.rule.counts;
assert!(counts.books > 700, "{counts:?}");
assert_eq!(
(counts.losses, counts.below_zero, counts.duplicates),
(0, 0, 0)
);
assert!(written.values().map(Vec::len).sum::<usize>() > 1_000);
// A NO bid arrived as the ask it is: no book is crossed. Taken at
// one minus its price again, a third of them would be.
let mut two_sided = 0;
for (ticker, symbol) in &feed.shell.symbols {
let best_bid = symbol.book.bids.keys().next_back();
let best_ask = symbol.book.asks.keys().next();
if let (Some(bid), Some(ask)) = (best_bid, best_ask) {
assert!(bid < ask, "{ticker}: bid {bid}, ask {ask}");
two_sided += 1;
}
}
assert!(two_sided >= 10, "{two_sided}");
}
/// The book and writer threads as main runs them, fed the recording:
/// every Snapshot is the one before it with the Deltas between.
#[test]
fn the_output_reconciles() {
let recording = recording();
let tickers = tickers(&recording);
let tickers: Vec<&str> = tickers.iter().map(String::as_str).collect();
let feed = feed_of(&tickers, &[0, 1]);
let (events_tx, events_rx) = crossbeam::channel::unbounded();
let (outputs_tx, outputs_rx) = crossbeam::channel::unbounded();
let (connection_tx, _connection_rx) = crossbeam::channel::unbounded();
let (_commands_tx, commands_rx) = crossbeam::channel::unbounded();
let (reports_tx, _reports_rx) = crossbeam::channel::unbounded();
for (received_at, connection, event) in recording {
let arrival = Arrival {
received_at,
connection,
bytes: 0,
event,
};
events_tx.send(Received::Event(arrival)).unwrap();
}
drop(events_tx);
let book = std::thread::spawn(move || {
feed::serve(
feed,
vec![connection_tx.clone(), connection_tx],
events_rx,
commands_rx,
outputs_tx,
reports_tx,
)
});
let path =
std::env::temp_dir().join(format!("kalshi-reconcile-{}.sbe", std::process::id()));
let written = crate::write(path.to_str().unwrap(), outputs_rx);
book.join().unwrap();
written.unwrap();
let report = engineering_challenge::reconcile::reconcile(&path, None);
std::fs::remove_file(&path).unwrap();
let report = report.unwrap();
// The opening Snapshots, and one per symbol at the minute crossed.
assert_eq!(report.snapshots, tickers.len() * 2, "{report:?}");
assert!(report.deltas > 1_000, "{report:?}");
assert_eq!(report.orphan_deltas, 0);
assert_eq!(report.mismatches, []);
}
/// Connection 1 left five markets for 5 s: connection 0 carried them.
#[test]
fn two_recorded_connections_write_the_changes_of_one_each_at_its_first_copy() {
let (one, alone) = replay(&[0]);
let (both, together) = replay(&[0, 1]);
assert_eq!(both.rule.counts.losses, 0);
assert!(both.rule.counts.duplicates > 900, "{:?}", both.rule.counts);
assert_eq!(alone.keys().len(), together.keys().len());
let mut earlier = 0;
for (symbol, changes) in &alone {
let content = |changes: &[(Side, i64, i64, DateTime<Utc>)]| {
changes
.iter()
.map(|&(side, price, quantity, _)| (side, price, quantity))
.collect::<Vec<_>>()
};
assert_eq!(content(changes), content(&together[symbol]), "{symbol}");
for (single, first) in changes.iter().zip(&together[symbol]) {
assert!(first.3 <= single.3);
earlier += usize::from(first.3 < single.3);
}
}
// Connection 1 was first for most changes in the recording.
assert!(earlier > 500, "{earlier}");
assert_eq!(one.rule.counts.written, both.rule.counts.written);
}
}
+334
View File
@@ -0,0 +1,334 @@
//! Kalshi's market listing: which markets are live, or about to be. Two fetcher threads page
//! it on the orchestrator's request and stream what they read; they hold no
//! state and no timer, as Gamma's (roadmap/06). Read without a key.
use std::time::Duration;
use chrono::{DateTime, TimeDelta, Utc};
use crossbeam::channel::{Receiver, Sender};
use serde::Deserialize;
use super::Market;
use crate::core::{Finding, Request, retried};
/// The listing's largest page.
const PAGE_SIZE: &str = "1000";
/// For every request, so SIGINT never waits on the listing longer than this.
const TIMEOUT: Duration = Duration::from_secs(10);
/// A page's tries, and the wait between them.
const TRIES: u32 = 5;
const RETRY_AFTER: Duration = Duration::from_secs(5);
/// A market not yet open is tracked this long before its opening: its book is
/// there, empty, from its first order. Three hourly rescans, so every market
/// is reached by a scan before it opens even when one scan fails.
const OPENING_AHEAD: TimeDelta = TimeDelta::hours(3);
/// A fetcher thread: answers each request until the orchestrator is gone.
pub fn run(listing: &str, requests: Receiver<Request>, findings: Sender<Finding<Market>>) {
let agent = ureq::Agent::new_with_config(
ureq::Agent::config_builder()
.timeout_global(Some(TIMEOUT))
.build(),
);
let markets = format!("{listing}/markets");
for request in requests {
let sent = match request {
Request::Rescan => rescan(&agent, &markets, &findings),
Request::Recent { since } => recent(&agent, &markets, since, &findings),
};
if sent.is_err() {
return;
}
}
}
/// `Err` once the orchestrator is gone.
type Sent = Result<(), crossbeam::channel::SendError<Finding<Market>>>;
fn rescan(agent: &ureq::Agent, markets: &str, findings: &Sender<Finding<Market>>) -> Sent {
findings.send(Finding::ScanStarted)?;
// Unopened first: a market that opens while the scan runs is then on one
// list at least. Open first, it could be on neither, and be removed.
for status in ["unopened", "open"] {
let query = [("status", status.to_string()), mve_excluded()];
let mut cursor = String::new();
loop {
let page = match retried(TRIES, RETRY_AFTER, || page(agent, markets, &query, &cursor)) {
Ok(page) => page,
Err(error) => {
log!("kalshi: rescan failed: {error}");
return findings.send(Finding::ScanFailed);
}
};
let last = page.markets.is_empty() || page.cursor.is_empty();
findings.send(Finding::ScanPage(tracked(page.markets, Utc::now())))?;
if last {
break;
}
cursor = page.cursor;
}
}
findings.send(Finding::ScanDone)
}
fn recent(
agent: &ureq::Agent,
markets: &str,
since: DateTime<Utc>,
findings: &Sender<Finding<Market>>,
) -> Sent {
let since = since.timestamp().to_string();
// Created, whatever their status: open and unopened alike.
let created = [("min_created_ts", since.clone()), mve_excluded()];
let closed = [
("status", "closed".to_string()),
("min_close_ts", since),
mve_excluded(),
];
let result = every_page(agent, markets, &created).and_then(|created| {
findings.send(Finding::Created(tracked(created, Utc::now())))?;
every_page(agent, markets, &closed)
});
match result {
Ok(closed) => {
let tickers = closed.into_iter().map(|market| market.ticker);
findings.send(Finding::Closed(tickers.collect()))?;
findings.send(Finding::RecentDone)
}
Err(error) => {
log!("kalshi: recent check failed: {error}");
findings.send(Finding::RecentFailed)
}
}
}
/// Combos (MVEs) are not in the challenge.
fn mve_excluded() -> (&'static str, String) {
("mve_filter", "exclude".to_string())
}
fn every_page(
agent: &ureq::Agent,
markets: &str,
query: &[(&str, String)],
) -> anyhow::Result<Vec<ListedMarket>> {
let mut listed = Vec::new();
let mut cursor = String::new();
loop {
let page = retried(TRIES, RETRY_AFTER, || page(agent, markets, query, &cursor))?;
let last = page.markets.is_empty() || page.cursor.is_empty();
listed.extend(page.markets);
if last {
return Ok(listed);
}
cursor = page.cursor;
}
}
#[derive(Deserialize)]
struct Page {
markets: Vec<ListedMarket>,
/// Empty on the last page.
#[serde(default)]
cursor: String,
}
#[derive(Deserialize)]
struct ListedMarket {
ticker: String,
/// `active` once open, `initialized` before.
status: String,
open_time: Option<String>,
}
fn page(
agent: &ureq::Agent,
markets: &str,
query: &[(&str, String)],
cursor: &str,
) -> anyhow::Result<Page> {
let mut request = agent
.get(markets)
.query("limit", PAGE_SIZE)
.query_pairs(query.iter().map(|(name, value)| (*name, value.as_str())));
if !cursor.is_empty() {
request = request.query("cursor", cursor);
}
let body = request.call()?.body_mut().read_to_string()?;
Ok(serde_json::from_str(&body)?)
}
/// The markets to track among `markets` at `now`: those live, and those
/// opening within [`OPENING_AHEAD`].
fn tracked(markets: Vec<ListedMarket>, now: DateTime<Utc>) -> Vec<Market> {
markets
.into_iter()
.filter(|market| match market.status.as_str() {
"active" => true,
"initialized" => market
.open_time
.as_deref()
.and_then(|time| DateTime::parse_from_rfc3339(time).ok())
.is_some_and(|opens| opens.to_utc() - now <= OPENING_AHEAD),
_ => false,
})
.map(|market| Market {
ticker: market.ticker,
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
fn fixture(name: &str) -> Page {
let path = format!("{}/tests/fixtures/{name}", env!("CARGO_MANIFEST_DIR"));
serde_json::from_str(&std::fs::read_to_string(path).unwrap()).unwrap()
}
fn time(text: &str) -> DateTime<Utc> {
DateTime::parse_from_rfc3339(text).unwrap().to_utc()
}
fn tickers(markets: &[Market]) -> Vec<&str> {
markets
.iter()
.map(|market| market.ticker.as_str())
.collect()
}
#[test]
fn open_markets_are_tracked_and_the_last_page_has_no_cursor() {
let page = fixture("kalshi-open.json");
assert!(!page.cursor.is_empty());
let live = tracked(page.markets, time("2026-11-11T19:30:00Z"));
assert_eq!(live.len(), 3);
assert!(live[0].ticker.starts_with("KX"));
let last: Page = serde_json::from_str(r#"{"markets":[],"cursor":""}"#).unwrap();
assert!(last.cursor.is_empty());
}
/// The fixture's unopened markets open at 19:19:08.
#[test]
fn an_unopened_market_is_tracked_from_three_hours_before_its_opening() {
let opening = time("2026-11-11T19:19:08Z");
let at = |now: DateTime<Utc>| tracked(fixture("kalshi-unopened.json").markets, now);
assert!(at(opening - OPENING_AHEAD - TimeDelta::seconds(1)).is_empty());
assert_eq!(at(opening - OPENING_AHEAD).len(), 3);
// Past its opening and still listed unopened: tracked, empty.
assert_eq!(
tickers(&at(opening + TimeDelta::minutes(20)))[0],
"KXEUROCUPTOTAL-26NOV121400MAXSKY-179"
);
}
/// A local listing: answers each request with the fixture its query asks
/// for (an empty last page after a cursor), and tells the queries it got.
fn listing(requests: usize) -> (String, std::thread::JoinHandle<Vec<String>>) {
use std::io::{BufRead, BufReader, Write};
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("http://{}", listener.local_addr().unwrap());
let server = std::thread::spawn(move || {
let mut queries = Vec::new();
for _ in 0..requests {
let (mut stream, _) = listener.accept().unwrap();
let mut line = String::new();
BufReader::new(&stream).read_line(&mut line).unwrap();
let query = line.split(' ').nth(1).unwrap().to_string();
let name = if query.contains("cursor=") {
None
} else if query.contains("status=unopened") {
Some("kalshi-unopened.json")
} else if query.contains("status=closed") {
Some("kalshi-closed.json")
} else {
Some("kalshi-open.json")
};
let body = name.map_or(r#"{"markets":[],"cursor":""}"#.to_string(), |name| {
let path = format!("{}/tests/fixtures/{name}", env!("CARGO_MANIFEST_DIR"));
std::fs::read_to_string(path).unwrap()
});
write!(
stream,
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
)
.unwrap();
queries.push(query);
}
queries
});
(url, server)
}
fn found(findings: &[Finding<Market>]) -> Vec<String> {
findings
.iter()
.map(|finding| match finding {
Finding::ScanPage(markets) => format!("page {}", tickers(markets).join(" ")),
Finding::Created(markets) => format!("created {}", tickers(markets).join(" ")),
Finding::Closed(closed) => format!("closed {}", closed.join(" ")),
other => format!("{other:?}"),
})
.collect()
}
#[test]
fn a_rescan_pages_the_unopened_markets_then_the_open_ones() {
let (url, server) = listing(4);
let (requests_tx, requests) = crossbeam::channel::unbounded();
let (findings_tx, findings) = crossbeam::channel::unbounded();
requests_tx.send(Request::Rescan).unwrap();
drop(requests_tx);
run(&url, requests, findings_tx);
let queries = server.join().unwrap();
// Unopened first: a market opening meanwhile is on one list at least.
assert!(queries[0].contains("status=unopened"), "{queries:?}");
assert!(queries[1].contains("status=unopened") && queries[1].contains("cursor="));
assert!(queries[2].contains("status=open") && !queries[2].contains("cursor="));
for query in &queries {
assert!(query.contains("mve_filter=exclude") && query.contains("limit=1000"));
}
let found = found(&findings.try_iter().collect::<Vec<_>>());
assert_eq!(found.len(), 6, "{found:?}");
assert_eq!(found[0], "ScanStarted");
assert!(found[1].starts_with("page KXEUROCUPTOTAL-26NOV121400MAXSKY-179 "));
assert_eq!(found[2], "page ");
assert!(found[3].starts_with("page KXNCAAWBTOTAL-26NOV11QUINSJU-106 "));
assert_eq!(found[5], "ScanDone");
}
#[test]
fn a_recent_check_reports_the_markets_created_and_those_closed() {
let (url, server) = listing(4);
let (requests_tx, requests) = crossbeam::channel::unbounded();
let (findings_tx, findings) = crossbeam::channel::unbounded();
let since = time("2026-11-11T19:00:00Z");
requests_tx.send(Request::Recent { since }).unwrap();
drop(requests_tx);
run(&url, requests, findings_tx);
let queries = server.join().unwrap();
let since = format!("={}", since.timestamp());
assert!(
queries[0].contains(&format!("min_created_ts{since}"))
&& !queries[0].contains("status=")
);
assert!(
queries[2].contains(&format!("min_close_ts{since}"))
&& queries[2].contains("status=closed")
);
let found = found(&findings.try_iter().collect::<Vec<_>>());
assert_eq!(found.len(), 3, "{found:?}");
assert!(found[0].starts_with("created KXNCAAWBTOTAL-26NOV11QUINSJU-106 "));
assert!(found[1].starts_with("closed KXFIBAECUPSPREAD-26NOV111230SZOZIG-SZO23 "));
assert_eq!(found[2], "RecentDone");
}
#[test]
fn a_closed_market_is_not_tracked() {
let closed = fixture("kalshi-closed.json");
assert_eq!(closed.markets[0].status, "determined");
assert!(tracked(closed.markets, time("2026-11-11T19:30:00Z")).is_empty());
}
}
+286
View File
@@ -0,0 +1,286 @@
//! Kalshi: its websocket's `orderbook_delta` channel (the connection's side of
//! it, signed with an API key), its market listing, and the clock rule its
//! book threads apply (docs/notes.md, Kalshi's websocket; roadmap/06).
pub mod feed;
pub mod listing;
pub mod sign;
pub mod wire;
use chrono::Utc;
use crossbeam::channel::{Receiver, Sender};
use serde_json::json;
use tungstenite::client::IntoClientRequest;
use crate::core::{self, Finding, Request, Undecoded};
use sign::Key;
/// Tickers per subscribe or unsubscribe message.
const CHUNK: usize = 500;
/// Errors after which the subscription is gone (Kalshi's docs: a channel
/// error, a subscription buffer overflow): the connection starts over.
const TERMINAL_ERRORS: [u32; 2] = [10, 25];
/// A tracked market: one book, named by its ticker, which is its id and its
/// only symbol.
#[derive(Debug, Clone)]
pub struct Market {
pub ticker: String,
}
impl core::Market for Market {
fn id(&self) -> &str {
&self.ticker
}
fn symbols(&self) -> &[String] {
std::slice::from_ref(&self.ticker)
}
}
/// The exchange: its websocket and listing, and the key that signs the
/// websocket's handshake.
pub struct Kalshi {
pub websocket: String,
pub listing: String,
pub key: Key,
}
impl Kalshi {
pub fn production(key: Key) -> Self {
Self {
websocket: "wss://external-api-ws.kalshi.com/trade-api/ws/v2".into(),
listing: "https://external-api.kalshi.com/trade-api/v2".into(),
key,
}
}
pub fn demo(key: Key) -> Self {
Self {
websocket: "wss://external-api-ws.demo.kalshi.co/trade-api/ws/v2".into(),
listing: "https://external-api.demo.kalshi.co/trade-api/v2".into(),
key,
}
}
}
impl core::Exchange for Kalshi {
type Market = Market;
type Event = wire::Event;
type Session = Session;
type Rule = feed::Clock;
const NAME: &'static str = "kalshi";
fn request(&self) -> anyhow::Result<tungstenite::handshake::client::Request> {
let mut request = self.websocket.as_str().into_client_request()?;
for (name, value) in self.key.headers(Utc::now().timestamp_millis()) {
request.headers_mut().insert(name, value.parse()?);
}
Ok(request)
}
fn session(&self) -> Session {
Session::default()
}
fn rule(&self) -> feed::Clock {
feed::Clock::default()
}
fn fetch(&self, requests: Receiver<Request>, findings: Sender<Finding<Market>>) {
listing::run(&self.listing, requests, findings)
}
}
/// A connection's subscription: one per connection, which every subscribe
/// adds to.
#[derive(Debug, Default)]
pub struct Session {
/// The last command's id.
command: u64,
/// The subscription's id, from its `subscribed` answer.
sid: Option<u64>,
/// The last message's `seq`: the next one must follow it.
seq: Option<u64>,
}
impl Session {
fn command(&mut self, cmd: &str, params: serde_json::Value) -> String {
self.command += 1;
json!({"id": self.command, "cmd": cmd, "params": params}).to_string()
}
}
impl core::Session for Session {
type Event = wire::Event;
/// Nothing: a subscribe can't name no market, so the first one creates the
/// subscription.
fn opening(&mut self) -> Vec<String> {
Vec::new()
}
/// `use_yes_price` makes a NO bid come as the ask it is; sent, since its
/// default is announced to flip.
fn subscribe(&mut self, tickers: &[&String]) -> Vec<String> {
tickers
.chunks(CHUNK)
.map(|chunk| {
let params = json!({
"channels": ["orderbook_delta"],
"market_tickers": chunk,
"use_yes_price": true,
"skip_ticker_ack": true,
});
self.command("subscribe", params)
})
.collect()
}
fn unsubscribe(&mut self, tickers: &[&String]) -> Vec<String> {
// Every fresh connection's subscription was 1 (docs/notes.md): an
// unsubscribe sent before its answer names it so.
let sid = self.sid.unwrap_or(1);
tickers
.chunks(CHUNK)
.map(|chunk| {
let params = json!({
"sids": [sid],
"market_tickers": chunk,
"action": "delete_markets",
"skip_ticker_ack": true,
});
self.command("update_subscription", params)
})
.collect()
}
/// None: Kalshi pings a connection it has nothing else to send to, and
/// tungstenite answers.
fn keepalive(&self) -> Option<&'static str> {
None
}
fn decode(&mut self, text: &str) -> Result<Vec<wire::Event>, Undecoded> {
let frame: wire::Frame =
serde_json::from_str(text).map_err(|error| Undecoded::Unreadable(error.to_string()))?;
if let Some(seq) = frame.seq {
let expected = self.seq.map_or(1, |last| last + 1);
if seq != expected {
return Err(Undecoded::Lost(format!(
"message {seq} came, {expected} expected"
)));
}
self.seq = Some(seq);
}
match frame.kind.as_str() {
"subscribed" => self.sid = self.sid.or(frame.msg.sid),
"error" => {
let (code, text) = (frame.msg.code.unwrap_or(0), frame.msg.msg.as_deref());
let id = frame.id.unwrap_or(0);
if TERMINAL_ERRORS.contains(&code) {
return Err(Undecoded::Lost(format!("error {code} ({text:?})")));
}
log!("kalshi: command {id} refused: error {code} ({text:?})");
}
_ => {}
}
wire::event(&frame)
.map(|event| event.into_iter().collect())
.map_err(Undecoded::Unreadable)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::Session as _;
fn decoded(session: &mut Session, text: &str) -> Result<usize, String> {
match session.decode(text) {
Ok(events) => Ok(events.len()),
Err(Undecoded::Lost(why)) => Err(format!("lost: {why}")),
Err(Undecoded::Unreadable(why)) => Err(format!("unreadable: {why}")),
}
}
const DELTA: &str = r#"{"type":"orderbook_delta","sid":1,"seq":SEQ,"msg":{"market_ticker":"M","market_id":"x","clock":5,"price_dollars":"0.5000","delta_fp":"1.00","side":"yes","ts_ms":1}}"#;
fn delta(seq: u64) -> String {
DELTA.replace("SEQ", &seq.to_string())
}
#[test]
fn a_skipped_seq_is_a_lost_message() {
let mut session = Session::default();
let subscribed =
r#"{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":3}}"#;
assert_eq!(decoded(&mut session, subscribed), Ok(0));
assert_eq!(session.sid, Some(3));
assert_eq!(decoded(&mut session, &delta(1)), Ok(1));
// An answer counts too.
let ok = r#"{"type":"ok","id":2,"sid":3,"seq":2,"msg":{}}"#;
assert_eq!(decoded(&mut session, ok), Ok(0));
assert_eq!(decoded(&mut session, &delta(3)), Ok(1));
assert_eq!(
decoded(&mut session, &delta(5)),
Err("lost: message 5 came, 4 expected".into())
);
// A connection's first message is 1.
assert!(decoded(&mut Session::default(), &delta(2)).is_err());
}
#[test]
fn a_terminal_error_starts_over_and_others_are_logged() {
let error = |code: u32| {
format!(r#"{{"type":"error","id":4,"seq":1,"msg":{{"code":{code},"msg":"x"}}}}"#)
};
assert_eq!(decoded(&mut Session::default(), &error(14)), Ok(0));
assert!(
decoded(&mut Session::default(), &error(25))
.unwrap_err()
.starts_with("lost")
);
assert!(
decoded(&mut Session::default(), "not json")
.unwrap_err()
.starts_with("unreadable")
);
}
#[test]
fn commands_carry_their_ids_and_the_subscription() {
let mut session = Session::default();
let tickers: Vec<String> = (0..501).map(|n| format!("T{n}")).collect();
let refs: Vec<&String> = tickers.iter().collect();
let sent: Vec<serde_json::Value> = session
.subscribe(&refs)
.iter()
.map(|text| serde_json::from_str(text).unwrap())
.collect();
assert_eq!(sent.len(), 2);
assert_eq!(sent[0]["id"], 1);
assert_eq!(sent[1]["id"], 2);
assert_eq!(sent[0]["cmd"], "subscribe");
assert_eq!(
sent[0]["params"]["market_tickers"]
.as_array()
.unwrap()
.len(),
500
);
assert_eq!(sent[0]["params"]["use_yes_price"], true);
session.sid = Some(7);
let unsubscribe: serde_json::Value =
serde_json::from_str(&session.unsubscribe(&refs[..1])[0]).unwrap();
assert_eq!(unsubscribe["id"], 3);
assert_eq!(unsubscribe["cmd"], "update_subscription");
assert_eq!(unsubscribe["params"]["sids"], json!([7]));
assert_eq!(unsubscribe["params"]["action"], "delete_markets");
assert_eq!(unsubscribe["params"]["market_tickers"], json!(["T0"]));
}
}
#[cfg(test)]
mod socket_tests;
+186
View File
@@ -0,0 +1,186 @@
//! The key that signs Kalshi's websocket handshake (docs/notes.md, Kalshi's
//! websocket): an Ed25519 key, read once from the files that the environment
//! names. Neither file's content is ever logged.
use std::path::Path;
use base64::Engine;
use ring::signature::Ed25519KeyPair;
use rustls_pki_types::PrivateKeyDer;
use rustls_pki_types::pem::PemObject;
/// The variables naming the two files.
pub const KEY_ID_FILE: &str = "KALSHI_KEY_ID_FILE";
pub const PRIVATE_KEY_FILE: &str = "KALSHI_PRIVATE_KEY_FILE";
/// What the handshake signs, after the time: the method and the path.
const SIGNED_PATH: &str = "GET/trade-api/ws/v2";
pub struct Key {
id: String,
pair: Ed25519KeyPair,
}
impl Key {
/// The key from the files [`KEY_ID_FILE`] and [`PRIVATE_KEY_FILE`] name.
pub fn from_env() -> anyhow::Result<Self> {
let path = |variable: &str| {
std::env::var_os(variable)
.ok_or_else(|| anyhow::anyhow!("{variable} is not set: it names a file of the key"))
};
Self::from_files(
path(KEY_ID_FILE)?.as_ref(),
path(PRIVATE_KEY_FILE)?.as_ref(),
)
.map_err(|error| {
anyhow::anyhow!("{error} (the files {KEY_ID_FILE} and {PRIVATE_KEY_FILE} name)")
})
}
/// The key id in `id_file`, the private key (a PKCS#8 PEM) in `key_file`.
pub fn from_files(id_file: &Path, key_file: &Path) -> anyhow::Result<Self> {
let id = std::fs::read_to_string(id_file)
.map_err(|error| anyhow::anyhow!("{}: {error}", id_file.display()))?
.trim()
.to_string();
anyhow::ensure!(!id.is_empty(), "{}: no key id in it", id_file.display());
let pem = std::fs::read(key_file)
.map_err(|error| anyhow::anyhow!("{}: {error}", key_file.display()))?;
// The errors name the file, never its content: the PEM reader's own
// error quotes what it read, so it is dropped.
let der = PrivateKeyDer::from_pem_slice(&pem)
.map_err(|_| anyhow::anyhow!("{}: not a PEM private key", key_file.display()))?;
let PrivateKeyDer::Pkcs8(der) = der else {
anyhow::bail!(
"{}: not a PKCS#8 key; Kalshi's Ed25519 keys are",
key_file.display()
);
};
let pair = Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der())
.map_err(|_| anyhow::anyhow!("{}: not an Ed25519 key", key_file.display()))?;
Ok(Self { id, pair })
}
/// The handshake's three headers at `millis` since the epoch.
pub fn headers(&self, millis: i64) -> [(&'static str, String); 3] {
let signature = self.pair.sign(format!("{millis}{SIGNED_PATH}").as_bytes());
[
("KALSHI-ACCESS-KEY", self.id.clone()),
("KALSHI-ACCESS-TIMESTAMP", millis.to_string()),
(
"KALSHI-ACCESS-SIGNATURE",
base64::engine::general_purpose::STANDARD.encode(signature.as_ref()),
),
]
}
}
/// RFC 8032's first test key (section 7.1), as PKCS#8: a known key, so a
/// known signature.
#[cfg(test)]
const TEST_KEY: &str = "-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEIJ1hsZ3v/VpguoRK9JLsLMREScVpezJpGXA7rAMcrn9g
-----END PRIVATE KEY-----
";
/// [`TEST_KEY`], with the id `a-key-id`.
#[cfg(test)]
pub fn test_key() -> Key {
let der = PrivateKeyDer::from_pem_slice(TEST_KEY.as_bytes()).unwrap();
let PrivateKeyDer::Pkcs8(der) = der else {
unreachable!("PKCS#8")
};
Key {
id: "a-key-id".into(),
pair: Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()).unwrap(),
}
}
#[cfg(test)]
mod tests {
use super::*;
fn write(dir: &Path, name: &str, text: &str) -> std::path::PathBuf {
let path = dir.join(name);
std::fs::write(&path, text).unwrap();
path
}
fn scratch(name: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir().join(format!("kalshi-sign-{name}-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
dir
}
#[test]
fn signs_the_time_method_and_path() {
let dir = scratch("signs");
let key = Key::from_files(
&write(&dir, "id", "a-key-id\n"),
&write(&dir, "key.pem", TEST_KEY),
)
.unwrap();
let [id, time, signature] = key.headers(1_669_149_841_234);
assert_eq!(id, ("KALSHI-ACCESS-KEY", "a-key-id".to_string()));
assert_eq!(
time,
("KALSHI-ACCESS-TIMESTAMP", "1669149841234".to_string())
);
assert_eq!(signature.0, "KALSHI-ACCESS-SIGNATURE");
// Checked against the RFC's public key for this one.
let public = ring::signature::UnparsedPublicKey::new(
&ring::signature::ED25519,
[
0xd7, 0x5a, 0x98, 0x01, 0x82, 0xb1, 0x0a, 0xb7, 0xd5, 0x4b, 0xfe, 0xd3, 0xc9, 0x64,
0x07, 0x3a, 0x0e, 0xe1, 0x72, 0xf3, 0xda, 0xa6, 0x23, 0x25, 0xaf, 0x02, 0x1a, 0x68,
0xf7, 0x07, 0x51, 0x1a,
],
);
let bytes = base64::engine::general_purpose::STANDARD
.decode(&signature.1)
.unwrap();
public
.verify(b"1669149841234GET/trade-api/ws/v2", &bytes)
.unwrap();
// Ed25519 is deterministic: the same time, the same signature.
assert_eq!(key.headers(1_669_149_841_234)[2], signature);
std::fs::remove_dir_all(dir).unwrap();
}
#[test]
fn a_missing_or_wrong_file_says_which_and_never_what_it_holds() {
let dir = scratch("errors");
let id = write(&dir, "id", "a-key-id");
let error = |id: &Path, key: &Path| Key::from_files(id, key).err().unwrap().to_string();
let missing = error(&id, &dir.join("absent.pem"));
assert!(missing.contains("absent.pem"), "{missing}");
// A key flattened onto one line: the PEM reader's own error quotes
// what it read, so ours must not carry it.
let flat = write(&dir, "flat.pem", &TEST_KEY.replace('\n', " "));
let flattened = error(&id, &flat);
assert!(flattened.contains("not a PEM private key"), "{flattened}");
assert!(flattened.ends_with("not a PEM private key"), "{flattened}");
let garbage = write(&dir, "garbage.pem", "not a key at all");
let unreadable = error(&id, &garbage);
assert!(
unreadable.ends_with("not a PEM private key"),
"{unreadable}"
);
// An RSA key in PKCS#1, as Kalshi generates its RSA keys: refused.
let rsa = write(
&dir,
"rsa.pem",
"-----BEGIN RSA PRIVATE KEY-----\nMAA=\n-----END RSA PRIVATE KEY-----\n",
);
assert!(error(&id, &rsa).contains("not a PKCS#8 key"));
let empty = write(&dir, "empty-id", "\n");
let key = write(&dir, "key.pem", TEST_KEY);
assert!(error(&empty, &key).contains("no key id"));
std::fs::remove_dir_all(dir).unwrap();
}
}
+159
View File
@@ -0,0 +1,159 @@
//! The connection loop (`src/core/socket.rs`) against a local server, with
//! Kalshi's codec: the signed handshake, the server's ping, and a skipped
//! `seq`.
use std::net::TcpListener;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread;
use std::time::Duration;
use crossbeam::channel::unbounded;
use tungstenite::handshake::server::{Request, Response};
use tungstenite::{Bytes, Message};
use super::{Kalshi, sign};
use crate::core::socket::{Command, Notice, SILENT_AFTER, run};
use crate::core::{BOOK_THREADS, Received};
const SNAPSHOT: &str = r#"{"type":"orderbook_snapshot","sid":1,"seq":1,"msg":{"market_ticker":"M","market_id":"x","clock":10,"yes_dollars_fp":[["0.4000","5.00"]]},"sending_ts_ms":1794422918369}"#;
fn delta(seq: u64) -> String {
format!(
r#"{{"type":"orderbook_delta","sid":1,"seq":{seq},"msg":{{"market_ticker":"M","market_id":"x","clock":{seq}0,"price_dollars":"0.4000","delta_fp":"1.00","side":"yes","ts_ms":1794422918370}}}}"#
)
}
#[test]
// The handshake callback's error type is tungstenite's, not ours.
#[allow(clippy::result_large_err)]
fn signs_its_handshake_answers_pings_and_reconnects_on_a_skipped_seq() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/trade-api/ws/v2", listener.local_addr().unwrap());
let server = thread::spawn(move || {
let mut seen = Vec::new();
for connection in 0..2 {
let (stream, _) = listener.accept().unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut headers = Vec::new();
let mut socket =
tungstenite::accept_hdr(stream, |request: &Request, response: Response| {
for name in [
"KALSHI-ACCESS-KEY",
"KALSHI-ACCESS-TIMESTAMP",
"KALSHI-ACCESS-SIGNATURE",
] {
let value = request
.headers()
.get(name)
.map(|value| value.to_str().unwrap().to_string());
headers.push(value);
}
Ok(response)
})
.unwrap();
let subscribe: serde_json::Value =
serde_json::from_str(socket.read().unwrap().to_text().unwrap()).unwrap();
assert_eq!(subscribe["cmd"], "subscribe");
assert_eq!(
subscribe["params"]["market_tickers"],
serde_json::json!(["M"])
);
socket
.send(Message::text(
r#"{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}"#,
))
.unwrap();
socket.send(Message::text(SNAPSHOT)).unwrap();
socket.send(Message::text(delta(2))).unwrap();
if connection == 1 {
socket.close(None).unwrap();
while socket.read().is_ok() {}
seen.push((headers, false));
break;
}
socket
.send(Message::Ping(Bytes::from_static(b"heartbeat")))
.unwrap();
let ponged = loop {
match socket.read().unwrap() {
Message::Pong(body) => break body == Bytes::from_static(b"heartbeat"),
_ => continue,
}
};
// 3 never comes.
socket.send(Message::text(delta(4))).unwrap();
while socket.read().is_ok() {}
seen.push((headers, ponged));
}
seen
});
let kalshi = Kalshi {
websocket: url,
listing: String::new(),
key: sign::test_key(),
};
let (book_threads, events): (Vec<_>, Vec<_>) = (0..BOOK_THREADS).map(|_| unbounded()).unzip();
let (commands_tx, commands) = unbounded();
commands_tx
.send(Command::Subscribe(vec!["M".into()]))
.unwrap();
let (notices_tx, notices) = unbounded();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let client = thread::spawn(move || {
run(
&kalshi,
0,
&book_threads,
&commands,
&notices_tx,
SILENT_AFTER,
&client_stop,
)
});
let seen = server.join().unwrap();
stop.store(true, Ordering::Relaxed);
client.join().unwrap();
for (headers, _) in &seen {
assert_eq!(headers[0].as_deref(), Some("a-key-id"));
assert!(
headers[1]
.as_deref()
.is_some_and(|time| time.parse::<i64>().is_ok())
);
assert!(
headers[2]
.as_deref()
.is_some_and(|signature| signature.len() == 88)
);
}
assert!(seen[0].1, "the ping got its pong");
let notices: Vec<Notice> = notices.try_iter().collect();
assert_eq!(
notices[..3],
[Notice::Up(0), Notice::Down(0), Notice::Up(0)]
);
// The book and the change before the gap reached the book thread; the
// change after it didn't.
let received: Vec<_> = events
.iter()
.flat_map(|receiver| receiver.try_iter())
.collect();
// Every book thread heard it go down each time: after the skipped
// `seq`, and when the server closed the second connection.
let downs = received
.iter()
.filter(|received| matches!(received, Received::Down(0)))
.count();
assert_eq!(downs, 2 * BOOK_THREADS);
let events: Vec<_> = received
.iter()
.filter(|received| matches!(received, Received::Event(_)))
.collect();
assert_eq!(events.len(), 4, "two per connection: {events:?}");
}
+213
View File
@@ -0,0 +1,213 @@
//! Kalshi's websocket messages, as the exchange writes them, and how a
//! connection reads them: each frame is one message, numbered by `seq` across
//! the connection (docs/notes.md, Kalshi's websocket).
use chrono::{DateTime, Utc};
use engineering_challenge::Side;
use engineering_challenge::book::parse_fixed;
use serde::Deserialize;
/// A message of the `orderbook_delta` channel, read.
#[derive(Debug, PartialEq, Eq)]
pub enum Event {
Book(BookMessage),
Change(ChangeMessage),
}
/// A market's full book: its YES bids are the bids; with `use_yes_price`,
/// its NO bids come as the asks they are, at one minus their price.
#[derive(Debug, PartialEq, Eq)]
pub struct BookMessage {
pub ticker: String,
/// The clock of the last change it holds (0 before the market opens).
pub clock: u64,
pub bids: Vec<(i64, i64)>,
pub asks: Vec<(i64, i64)>,
/// When the exchange sent it: a book has no time of its own.
pub sent_at: DateTime<Utc>,
}
/// A change of one level: its quantity moves by `delta`, which can be
/// negative.
#[derive(Debug, PartialEq, Eq)]
pub struct ChangeMessage {
pub ticker: String,
pub clock: u64,
pub side: Side,
pub price: i64,
pub delta: i64,
pub at: DateTime<Utc>,
}
impl crate::core::Event for Event {
fn market(&self) -> Option<&str> {
Some(match self {
Event::Book(book) => &book.ticker,
Event::Change(change) => &change.ticker,
})
}
fn is_book(&self) -> bool {
matches!(self, Event::Book(_))
}
}
/// What a frame holds, every field the channel's messages use.
#[derive(Debug, Deserialize)]
pub struct Frame {
#[serde(rename = "type")]
pub kind: String,
/// The command it answers, for an answer.
pub id: Option<u64>,
/// Counts every message of the connection's subscription but
/// `subscribed`, from 1.
pub seq: Option<u64>,
pub sending_ts_ms: Option<i64>,
#[serde(default)]
pub msg: Body,
}
#[derive(Debug, Default, Deserialize)]
pub struct Body {
pub market_ticker: Option<String>,
pub clock: Option<u64>,
pub yes_dollars_fp: Option<Vec<(String, String)>>,
pub no_dollars_fp: Option<Vec<(String, String)>>,
pub price_dollars: Option<String>,
pub delta_fp: Option<String>,
pub side: Option<String>,
pub ts_ms: Option<i64>,
/// `subscribed`: the subscription's id.
pub sid: Option<u64>,
/// `error`: its code and text.
pub code: Option<u32>,
pub msg: Option<String>,
}
/// The event a frame of the book channel holds; none for the others (answers
/// to commands), `Err` with what is wrong for a book or change unreadable.
pub fn event(frame: &Frame) -> Result<Option<Event>, String> {
let body = &frame.msg;
let ticker = || body.market_ticker.clone().ok_or("no market_ticker");
let clock = || body.clock.ok_or("no clock");
match frame.kind.as_str() {
"orderbook_snapshot" => {
let side = |levels: &Option<Vec<(String, String)>>| {
levels
.iter()
.flatten()
.map(|(price, quantity)| Some((parse_fixed(price)?, parse_fixed(quantity)?)))
.collect::<Option<Vec<_>>>()
.ok_or("a level is not two decimals")
};
let sent_at = frame
.sending_ts_ms
.and_then(DateTime::from_timestamp_millis)
.ok_or("no sending_ts_ms")?;
Ok(Some(Event::Book(BookMessage {
ticker: ticker()?,
clock: clock()?,
bids: side(&body.yes_dollars_fp)?,
asks: side(&body.no_dollars_fp)?,
sent_at,
})))
}
"orderbook_delta" => {
let side = match body.side.as_deref() {
Some("yes") => Side::Buy,
Some("no") => Side::Sell,
_ => return Err("side is neither yes nor no".into()),
};
let price = body.price_dollars.as_deref().and_then(parse_fixed);
let delta = body.delta_fp.as_deref().and_then(parse_signed);
let at = body.ts_ms.and_then(DateTime::from_timestamp_millis);
let (Some(price), Some(delta), Some(at)) = (price, delta, at) else {
return Err("price_dollars, delta_fp or ts_ms unreadable".into());
};
Ok(Some(Event::Change(ChangeMessage {
ticker: ticker()?,
clock: clock()?,
side,
price,
delta,
at,
})))
}
_ => Ok(None),
}
}
/// A signed decimal (`"-54.00"`, `"20.00"`), exactly.
fn parse_signed(text: &str) -> Option<i64> {
match text.strip_prefix('-') {
Some(magnitude) => parse_fixed(magnitude).map(|value| -value),
None => parse_fixed(text),
}
}
#[cfg(test)]
mod tests {
use super::*;
fn read(text: &str) -> Result<Option<Event>, String> {
event(&serde_json::from_str(text).unwrap())
}
#[test]
fn reads_a_book_and_a_change_as_recorded() {
let book = r#"{"type":"orderbook_snapshot","sid":1,"seq":2,"msg":{"market_ticker":"KXHIGHNY-26NOV12-T59","market_id":"01a1","clock":56884934192,"yes_dollars_fp":[["0.4800","80.00"],["0.6500","10.00"]],"no_dollars_fp":[["0.9100","80.00"],["0.6600","7.00"]]},"sending_ts_ms":1794422918369}"#;
assert_eq!(
read(book).unwrap(),
Some(Event::Book(BookMessage {
ticker: "KXHIGHNY-26NOV12-T59".into(),
clock: 56_884_934_192,
bids: vec![(480_000, 80_000_000), (650_000, 10_000_000)],
asks: vec![(910_000, 80_000_000), (660_000, 7_000_000)],
sent_at: DateTime::from_timestamp_millis(1_794_422_918_369).unwrap(),
}))
);
let change = r#"{"type":"orderbook_delta","sid":1,"seq":9,"msg":{"market_ticker":"M","market_id":"01a1","clock":56885842636,"price_dollars":"0.8100","delta_fp":"-1.50","side":"no","ts":"2026-11-11T18:36:37.012Z","ts_ms":1794422197012},"sending_ts_ms":1794422197023}"#;
assert_eq!(
read(change).unwrap(),
Some(Event::Change(ChangeMessage {
ticker: "M".into(),
clock: 56_885_842_636,
side: Side::Sell,
price: 810_000,
delta: -1_500_000,
at: DateTime::from_timestamp_millis(1_794_422_197_012).unwrap(),
}))
);
}
#[test]
fn an_empty_book_has_no_levels_and_answers_are_no_events() {
// An unopened market's book, sides omitted.
let empty = r#"{"type":"orderbook_snapshot","sid":1,"seq":2,"msg":{"market_ticker":"KXQUICKSETTLE-11NOV26H1400-2","market_id":"","clock":0},"sending_ts_ms":1794422918369}"#;
let Some(Event::Book(book)) = read(empty).unwrap() else {
panic!("not a book");
};
assert!(book.bids.is_empty() && book.asks.is_empty());
assert_eq!(book.clock, 0);
for answer in [
r#"{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1},"sending_ts_ms":1}"#,
r#"{"type":"ok","id":2,"sid":1,"seq":6,"sending_ts_ms":1,"msg":{}}"#,
r#"{"type":"error","id":2,"sid":1,"seq":860,"msg":{"code":14,"msg":"Market Ticker required"}}"#,
] {
assert_eq!(read(answer).unwrap(), None, "{answer}");
}
}
#[test]
fn an_unreadable_change_says_so() {
for bad in [
r#"{"type":"orderbook_delta","seq":1,"msg":{"market_ticker":"M","clock":1,"price_dollars":"0.81","delta_fp":"1e3","side":"no","ts_ms":1}}"#,
r#"{"type":"orderbook_delta","seq":1,"msg":{"market_ticker":"M","clock":1,"price_dollars":"0.81","delta_fp":"1.00","side":"maybe","ts_ms":1}}"#,
r#"{"type":"orderbook_delta","seq":1,"msg":{"clock":1,"price_dollars":"0.81","delta_fp":"1.00","side":"no","ts_ms":1}}"#,
] {
assert!(read(bad).is_err(), "{bad}");
}
assert_eq!(parse_signed("-0.01"), Some(-10_000));
assert_eq!(parse_signed("--1"), None);
}
}
+33 -10
View File
@@ -1,8 +1,9 @@
//! The collector (roadmap/04): an orchestrator keeps the tracked markets equal
//! to Gamma's live ones and places them on connections (read, stamp, decode,
//! route), which feed the book threads (fold, check, resync), all writing
//! through one writer (SBE).
//! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs
//! The collector (roadmap/04, roadmap/06): an orchestrator keeps the tracked
//! markets equal to the exchange's live ones and places them on connections
//! (read, stamp, decode, route), which feed the book threads (fold, check,
//! resync), all writing through one writer (SBE).
//! Usage: `engineering-challenge EXCHANGE [OUTPUT]`, EXCHANGE one of
//! `polymarket`, `kalshi` or `kalshi-demo`, OUTPUT defaulting to stdout; logs
//! go to stderr.
use std::sync::Arc;
@@ -19,6 +20,8 @@ macro_rules! log {
};
}
mod core;
mod kalshi;
mod polymarket;
use engineering_challenge::book::Output;
@@ -26,11 +29,31 @@ use engineering_challenge::book::Output;
/// `AlignedSbeReader::open` reads nothing else.
const CHUNK: usize = 64 << 20;
fn main() -> anyhow::Result<()> {
let output = std::env::args()
.nth(1)
.unwrap_or_else(|| "/dev/stdout".to_string());
const USAGE: &str = "usage: engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]";
fn main() -> anyhow::Result<()> {
let mut args = std::env::args().skip(1);
let exchange = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?;
let output = args.next().unwrap_or_else(|| "/dev/stdout".to_string());
// Before anything starts: a key that can't be read stops the run here.
match exchange.as_str() {
"polymarket" => collect(
polymarket::Polymarket {
url: polymarket::MARKET_CHANNEL_URL.to_string(),
},
output,
),
"kalshi" => collect(
kalshi::Kalshi::production(kalshi::sign::Key::from_env()?),
output,
),
"kalshi-demo" => collect(kalshi::Kalshi::demo(kalshi::sign::Key::from_env()?), output),
_ => anyhow::bail!("unknown exchange {exchange:?}; {USAGE}"),
}
}
/// Collects `exchange` into `output` until SIGINT.
fn collect<E: core::Exchange>(exchange: E, output: String) -> anyhow::Result<()> {
let stop = Arc::new(AtomicBool::new(false));
let on_sigint = Arc::clone(&stop);
ctrlc::set_handler(move || on_sigint.store(true, Ordering::Relaxed))?;
@@ -47,7 +70,7 @@ fn main() -> anyhow::Result<()> {
// connections; the book threads stop once every connection has, and the
// writer flushes once they all have, even after a panic upstream (the run
// still fails then).
let failed = polymarket::orchestrator::run(outputs_tx, &stop);
let failed = core::orchestrator::run(Arc::new(exchange), outputs_tx, &stop);
let written = writer
.join()
.map_err(|_| anyhow::anyhow!("the writer thread panicked"))?;
+544 -969
View File
File diff suppressed because it is too large. Load diff
+8 -43
View File
@@ -2,7 +2,6 @@
//! threads page it on the orchestrator's request and stream what they read;
//! they hold no state and no timer (roadmap/03).
use std::thread;
use std::time::Duration;
use chrono::{DateTime, Utc};
@@ -10,6 +9,7 @@ use crossbeam::channel::{Receiver, Sender};
use serde::Deserialize;
use super::Market;
use crate::core::{Finding, Request, retried};
const KEYSET_URL: &str = "https://gamma-api.polymarket.com/markets/keyset";
/// Gamma's page size, whatever a larger `limit` asks for.
@@ -23,31 +23,7 @@ const SCAN_RETRY_AFTER: Duration = Duration::from_secs(5);
/// null (checked on 4 such tokens, 2026-11-04).
const MISSING_MIN_ORDER_SIZE: &str = "0";
pub enum Request {
/// Every live market.
Rescan,
/// The live markets created and the markets closed since then.
Recent { since: DateTime<Utc> },
}
#[derive(Debug)]
pub enum Finding {
/// A rescan began: once it is done, a market known now and not seen in it
/// is gone.
ScanStarted,
ScanPage(Vec<Market>),
ScanDone,
ScanFailed,
/// Live markets created since the request's time.
Created(Vec<Market>),
/// The `conditionId`s of markets closed since the request's time.
Closed(Vec<String>),
RecentDone,
RecentFailed,
}
/// A fetcher thread: answers each request until the orchestrator is gone.
pub fn run(requests: Receiver<Request>, findings: Sender<Finding>) {
pub fn run(requests: Receiver<Request>, findings: Sender<Finding<Market>>) {
let agent = ureq::Agent::new_with_config(
ureq::Agent::config_builder()
.timeout_global(Some(TIMEOUT))
@@ -65,14 +41,16 @@ pub fn run(requests: Receiver<Request>, findings: Sender<Finding>) {
}
/// `Err` once the orchestrator is gone.
type Sent = Result<(), crossbeam::channel::SendError<Finding>>;
type Sent = Result<(), crossbeam::channel::SendError<Finding<Market>>>;
fn rescan(agent: &ureq::Agent, findings: &Sender<Finding>) -> Sent {
fn rescan(agent: &ureq::Agent, findings: &Sender<Finding<Market>>) -> Sent {
findings.send(Finding::ScanStarted)?;
let query = [("active", "true"), ("closed", "false")];
let mut cursor = None;
loop {
let page = match with_retries(|| page(agent, &query, cursor.as_deref())) {
let page = match retried(SCAN_TRIES, SCAN_RETRY_AFTER, || {
page(agent, &query, cursor.as_deref())
}) {
Ok(page) => page,
Err(error) => {
log!("polymarket: rescan failed: {error}");
@@ -88,20 +66,7 @@ fn rescan(agent: &ureq::Agent, findings: &Sender<Finding>) -> Sent {
}
}
fn with_retries(mut fetch: impl FnMut() -> anyhow::Result<Page>) -> anyhow::Result<Page> {
let mut tries = 1;
loop {
match fetch() {
Err(_) if tries < SCAN_TRIES => {
tries += 1;
thread::sleep(SCAN_RETRY_AFTER);
}
result => return result,
}
}
}
fn recent(agent: &ureq::Agent, since: DateTime<Utc>, findings: &Sender<Finding>) -> Sent {
fn recent(agent: &ureq::Agent, since: DateTime<Utc>, findings: &Sender<Finding<Market>>) -> Sent {
let created = [
("active", "true"),
("closed", "false"),
+91 -33
View File
@@ -1,25 +1,21 @@
//! Polymarket: which markets are live (Gamma), the orchestrator that places
//! them on connections, the connections, and the book threads they feed
//! Polymarket: its market channel (the connection's side of it), Gamma (which
//! markets are live), and the hash rule its book threads apply
//! (docs/adr/0003-first-one-wins.md).
pub mod feed;
pub mod gamma;
pub mod orchestrator;
pub mod socket;
pub mod wire;
/// How many book threads own the books. One handled ~30,000 frames a second
/// in sprint 02.
pub const BOOK_THREADS: usize = 16;
use crossbeam::channel::{Receiver, Sender};
use serde_json::json;
use tungstenite::client::IntoClientRequest;
/// The book thread that owns a market's books, from its `conditionId` (`0x`
/// and 64 hex digits, a hash: any 16 of them spread evenly). Computed, so the
/// orchestrator and every connection agree without a shared table.
pub fn book_thread(condition_id: &str) -> usize {
let digits = condition_id.get(2..18).unwrap_or_default();
// An id of another shape still gets a thread, always the same one.
u64::from_str_radix(digits, 16).unwrap_or(0) as usize % BOOK_THREADS
}
use crate::core::{self, Finding, Request, Undecoded};
pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
/// Tokens per subscribe or unsubscribe message: a first message of more than
/// 500 gets no opening books (docs/notes.md).
const CHUNK: usize = 500;
/// A live market: its symbols and the two inputs of their book hash that the
/// websocket doesn't carry (docs/notes.md).
@@ -34,27 +30,89 @@ pub struct Market {
pub neg_risk: bool,
}
impl core::Market for Market {
fn id(&self) -> &str {
&self.condition_id
}
fn symbols(&self) -> &[String] {
&self.tokens
}
}
/// The exchange, at its market channel's URL (the tests' local one in tests).
pub struct Polymarket {
pub url: String,
}
impl core::Exchange for Polymarket {
type Market = Market;
type Event = wire::Event;
type Session = Session;
type Rule = feed::Chain;
const NAME: &'static str = "polymarket";
fn request(&self) -> anyhow::Result<tungstenite::handshake::client::Request> {
Ok(self.url.as_str().into_client_request()?)
}
fn session(&self) -> Session {
Session
}
fn rule(&self) -> feed::Chain {
feed::Chain::default()
}
fn fetch(&self, requests: Receiver<Request>, findings: Sender<Finding<Market>>) {
gamma::run(requests, findings)
}
}
/// The market channel keeps no state of its own per connection.
pub struct Session;
impl Session {
/// The messages that subscribe or unsubscribe `tokens` on an open
/// connection.
fn operations(operation: &str, tokens: &[&String]) -> Vec<String> {
tokens
.chunks(CHUNK)
.map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string())
.collect()
}
}
impl core::Session for Session {
type Event = wire::Event;
/// Opened empty, then subscribed in chunks: one rule for every token.
fn opening(&mut self) -> Vec<String> {
vec![json!({"type": "market", "assets_ids": []}).to_string()]
}
fn subscribe(&mut self, tokens: &[&String]) -> Vec<String> {
Self::operations("subscribe", tokens)
}
fn unsubscribe(&mut self, tokens: &[&String]) -> Vec<String> {
Self::operations("unsubscribe", tokens)
}
fn keepalive(&self) -> Option<&'static str> {
Some("PING")
}
fn decode(&mut self, text: &str) -> Result<Vec<wire::Event>, Undecoded> {
wire::decode(text).map_err(|error| Undecoded::Unreadable(error.to_string()))
}
}
#[cfg(test)]
pub(crate) fn fixture_markets() -> Vec<Market> {
gamma::parse_markets(include_str!("../../tests/fixtures/gamma-markets.json"))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_market_always_gets_the_same_book_thread_and_markets_spread() {
let markets = fixture_markets();
let threads: std::collections::BTreeSet<usize> = markets
.iter()
.map(|market| book_thread(&market.condition_id))
.collect();
assert!(threads.iter().all(|&thread| thread < BOOK_THREADS));
// 20 markets over 16 threads: several threads, not one.
assert!(threads.len() > 5, "{threads:?}");
let id = &markets[0].condition_id;
assert_eq!(book_thread(id), book_thread(&id.clone()));
assert_eq!(book_thread("not an id"), 0);
}
}
mod socket_tests;
-600
View File
@@ -1,600 +0,0 @@
//! A connection: reads the market channel, stamps each frame's receive time,
//! decodes it and sends each event to the book thread of its market; and
//! sends the exchange what the book threads ask for (subscriptions, resyncs).
//! It reconnects when it drops, falls silent, or is recycled.
use std::collections::BTreeSet;
use std::io::ErrorKind;
use std::net::{TcpStream, ToSocketAddrs};
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread;
use std::time::{Duration, Instant};
use chrono::{DateTime, Utc};
use crossbeam::channel::{Receiver, Sender};
use serde_json::json;
use tungstenite::http::Uri;
use tungstenite::protocol::frame::coding::CloseCode;
use tungstenite::stream::MaybeTlsStream;
use tungstenite::{Message, WebSocket};
use super::book_thread;
use super::wire::{self, Arrival, Received};
pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
/// For the TCP connect and the TLS + websocket handshake, and for every send.
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
/// How often the loop wakes when no frame comes: tungstenite has no `split`,
/// so the reads and the writes below (PING, commands, close) share this thread.
const READ_TIMEOUT: Duration = Duration::from_millis(200);
const PING_EVERY: Duration = Duration::from_secs(10);
const RECONNECT_AFTER: Duration = Duration::from_secs(1);
const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
/// Nothing received this long, not even the answer to three keepalives: the
/// connection is silent, and reconnects.
pub const SILENT_AFTER: Duration = Duration::from_secs(30);
/// Tokens per subscribe or unsubscribe message: a first message of more than
/// 500 gets no opening books (docs/notes.md).
const CHUNK: usize = 500;
/// What a book thread asks a connection for.
#[derive(Debug, PartialEq, Eq)]
pub enum Command {
Subscribe(Vec<String>),
Unsubscribe(Vec<String>),
/// Unsubscribe and subscribe again: the server then sends a fresh book.
/// A resync asks for that book; a quiet subscription only for the new
/// subscription.
Resubscribe(String),
/// Close and reconnect, subscribing every token again.
Recycle,
}
/// What a connection tells the orchestrator.
#[derive(Debug, PartialEq, Eq)]
pub enum Notice {
/// Connected, its tokens subscribed.
Up(usize),
Down(usize),
/// Closed by the server with 1013: it couldn't send to us fast enough.
TooSlow(usize),
/// Closed by us: nothing came for [`SILENT_AFTER`].
Silent(usize),
}
type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
/// Runs until `stop` is set or a book thread is gone, reconnecting (and
/// subscribing its tokens again) whenever the connection drops, falls silent
/// for `silent_after` or is recycled, and telling every book thread and the
/// orchestrator (`notices`) when it comes up and goes down. `book_threads`
/// holds every book thread's channel, by number.
pub fn run(
url: &str,
number: usize,
book_threads: &[Sender<Received>],
commands: &Receiver<Command>,
notices: &Sender<Notice>,
silent_after: Duration,
stop: &AtomicBool,
) {
let tell = |received: fn(usize) -> Received, notice: fn(usize) -> Notice| {
for book_thread in book_threads {
_ = book_thread.send(received(number));
}
_ = notices.send(notice(number));
};
let mut tokens = BTreeSet::new();
while !stop.load(Ordering::Relaxed) {
// Asked for while disconnected: the subscribe on connect covers them,
// and its books are fresh, so resyncs have nothing left to do.
for command in commands.try_iter() {
_ = apply(&mut tokens, &command);
}
match connect(url, &tokens) {
Ok(mut socket) => {
log!(
"polymarket: connection {number}: connected, {} symbols",
tokens.len()
);
tell(Received::Up, Notice::Up);
let ended = read(
&mut socket,
number,
&mut tokens,
book_threads,
commands,
notices,
silent_after,
stop,
);
if !matches!(ended, Ok(Ended::Stopped)) {
tell(Received::Down, Notice::Down);
}
match ended {
Ok(Ended::Stopped) => return close(&mut socket, number, book_threads),
Ok(Ended::Recycled) => log!("polymarket: connection {number}: recycled"),
Ok(Ended::BookThreadGone) => {
// Only a panic ends a book thread before us: its
// markets would go unserved, so the run stops.
log!("polymarket: connection {number}: a book thread is gone, stopping");
return stop.store(true, Ordering::Relaxed);
}
Err(error) => {
log!("polymarket: connection {number}: connection lost: {error}")
}
}
}
Err(error) => log!("polymarket: connection {number}: connect failed: {error}"),
}
thread::sleep(RECONNECT_AFTER);
}
}
enum Ended {
Stopped,
BookThreadGone,
Recycled,
}
/// Updates the token set, and returns the messages that carry `command` out
/// on an open connection: none for a resync of a token not in the set.
#[must_use]
fn apply(tokens: &mut BTreeSet<String>, command: &Command) -> Vec<String> {
match command {
Command::Subscribe(added) => {
tokens.extend(added.iter().cloned());
operations("subscribe", added)
}
Command::Unsubscribe(removed) => {
removed.iter().for_each(|token| _ = tokens.remove(token));
operations("unsubscribe", removed)
}
// Subscribing a token already subscribed sends nothing back but
// `NO NEW ASSETS`: unsubscribing first makes the server send a fresh
// book.
Command::Resubscribe(token) if tokens.contains(token) => [
operations("unsubscribe", [token]),
operations("subscribe", [token]),
]
.concat(),
Command::Resubscribe(_) => Vec::new(),
// The read loop acts on it; disconnected, the reconnect does.
Command::Recycle => Vec::new(),
}
}
/// The messages that subscribe or unsubscribe `tokens` on an open connection.
fn operations<'a>(operation: &str, tokens: impl IntoIterator<Item = &'a String>) -> Vec<String> {
let tokens: Vec<_> = tokens.into_iter().collect();
tokens
.chunks(CHUNK)
.map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string())
.collect()
}
fn connect(url: &str, tokens: &BTreeSet<String>) -> anyhow::Result<Socket> {
let uri: Uri = url.parse()?;
let host = uri
.host()
.ok_or_else(|| anyhow::anyhow!("{url} has no host"))?;
// The market channel is `wss://` on TLS's port; the tests name theirs.
let port = uri.port_u16().unwrap_or(443);
// Timed, unlike `tungstenite::connect`, so a stalled connect or handshake
// can't keep SIGINT waiting. The name lookup itself has no timeout.
let address = (host, port)
.to_socket_addrs()?
.next()
.ok_or_else(|| anyhow::anyhow!("{host} has no address"))?;
let stream = TcpStream::connect_timeout(&address, CONNECT_TIMEOUT)?;
stream.set_read_timeout(Some(CONNECT_TIMEOUT))?;
stream.set_write_timeout(Some(CONNECT_TIMEOUT))?;
let (mut socket, _response) = tungstenite::client_tls(url, stream)
.map_err(|error| anyhow::anyhow!("handshake failed: {error}"))?;
let stream = match socket.get_ref() {
MaybeTlsStream::Plain(stream) => stream,
MaybeTlsStream::Rustls(stream) => stream.get_ref(),
_ => anyhow::bail!("unexpected TLS stream"),
};
stream.set_read_timeout(Some(READ_TIMEOUT))?;
// Opened empty, then subscribed in chunks: one rule for every token.
socket.send(Message::text(
json!({"type": "market", "assets_ids": []}).to_string(),
))?;
for message in operations("subscribe", tokens) {
socket.send(Message::text(message))?;
}
Ok(socket)
}
/// Decodes a frame and sends each event to the book thread of its market;
/// `false` once a book thread is gone.
fn route(
received_at: DateTime<Utc>,
text: &str,
number: usize,
book_threads: &[Sender<Received>],
) -> bool {
let events = match wire::decode(text) {
Ok(events) => events,
Err(error) => {
log!("polymarket: unreadable frame ({error}): {}", truncate(text));
return true;
}
};
let bytes = (text.len() / events.len().max(1)) as u64;
for event in events {
// An event of no market (none we ask for) has no book thread.
let Some(thread) = event.market().map(book_thread) else {
continue;
};
let arrival = Arrival {
received_at,
connection: number,
bytes,
event,
};
if book_threads[thread].send(Received::Event(arrival)).is_err() {
return false;
}
}
true
}
fn truncate(text: &str) -> &str {
text.char_indices()
.nth(200)
.map_or(text, |(end, _)| &text[..end])
}
// The connection's whole context, each a different part of it.
#[allow(clippy::too_many_arguments)]
fn read(
socket: &mut Socket,
number: usize,
tokens: &mut BTreeSet<String>,
book_threads: &[Sender<Received>],
commands: &Receiver<Command>,
notices: &Sender<Notice>,
silent_after: Duration,
stop: &AtomicBool,
) -> anyhow::Result<Ended> {
let mut last_ping = Instant::now();
let mut last_heard = Instant::now();
loop {
let message = socket.read();
// Stamp as soon as the frame is off the wire, before any processing.
let received_at = Utc::now();
if message.is_ok() {
last_heard = Instant::now();
}
match message {
Ok(Message::Text(text)) => {
if !route(received_at, text.as_str(), number, book_threads) {
return Ok(Ended::BookThreadGone);
}
}
Ok(Message::Close(frame)) => {
// 1013: the server couldn't send to us fast enough. The
// orchestrator splits a connection closed so twice.
if frame
.as_ref()
.is_some_and(|frame| frame.code == CloseCode::Again)
{
_ = notices.send(Notice::TooSlow(number));
}
anyhow::bail!("closed by the server: {frame:?}")
}
Ok(_) => {}
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
Err(error) => return Err(error.into()),
}
if stop.load(Ordering::Relaxed) {
return Ok(Ended::Stopped);
}
if last_heard.elapsed() >= silent_after {
_ = notices.send(Notice::Silent(number));
anyhow::bail!("silent for {} s", silent_after.as_secs_f64());
}
if last_ping.elapsed() >= PING_EVERY {
socket.send(Message::text("PING"))?;
last_ping = Instant::now();
}
for command in commands.try_iter() {
if command == Command::Recycle {
// Not waited on: whatever still comes would arrive from a
// connection already down for the book threads. The commands
// behind it are applied before the reconnect.
_ = socket.close(None);
_ = socket.flush();
return Ok(Ended::Recycled);
}
for message in apply(tokens, &command) {
socket.send(Message::text(message))?;
}
}
}
}
/// Closes the connection, forwarding what still arrives until the server
/// confirms or [`CLOSE_TIMEOUT`] passes.
fn close(socket: &mut Socket, number: usize, book_threads: &[Sender<Received>]) {
if let Err(error) = socket.close(None) {
return log!("polymarket: connection {number}: close failed: {error}");
}
let deadline = Instant::now() + CLOSE_TIMEOUT;
while Instant::now() < deadline {
let message = socket.read();
let received_at = Utc::now();
match message {
Ok(Message::Text(text)) => _ = route(received_at, text.as_str(), number, book_threads),
Ok(_) => {}
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
// ConnectionClosed: the close handshake is done.
Err(_) => break,
}
}
}
/// A read timeout shows up as `WouldBlock` on macOS and `TimedOut` elsewhere;
/// tungstenite keeps a partly read frame, so the next read resumes it.
fn timed_out(error: &std::io::Error) -> bool {
matches!(error.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut)
}
#[cfg(test)]
mod tests {
use std::net::TcpListener;
use std::sync::Arc;
use crossbeam::channel::unbounded;
use super::*;
/// The tokens each message of one connection subscribed, after the first.
fn subscribed(socket: &mut WebSocket<TcpStream>, tokens: usize) -> Vec<Vec<String>> {
let first = socket.read().unwrap().into_text().unwrap();
let first: serde_json::Value = serde_json::from_str(&first).unwrap();
assert_eq!(first, json!({"type": "market", "assets_ids": []}));
let mut chunks = Vec::new();
while chunks.iter().map(Vec::len).sum::<usize>() < tokens {
let text = socket.read().unwrap().into_text().unwrap();
let message: serde_json::Value = serde_json::from_str(&text).unwrap();
assert_eq!(message["operation"], "subscribe");
chunks.push(serde_json::from_value(message["assets_ids"].clone()).unwrap());
}
chunks
}
#[test]
fn a_reconnect_subscribes_every_token_again_in_chunks() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/ws/market", listener.local_addr().unwrap());
let tokens: Vec<String> = (0..1_200).map(|token| token.to_string()).collect();
let total = tokens.len();
let server = thread::spawn(move || {
let mut connections = Vec::new();
for _ in 0..2 {
let (stream, _) = listener.accept().unwrap();
// A client that never sends fails the test rather than hangs it.
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut socket = tungstenite::accept(stream).unwrap();
connections.push(subscribed(&mut socket, total));
// The client reconnects after this close; after the second,
// it finds no server and stops.
socket.close(None).unwrap();
while socket.read().is_ok() {}
}
connections
});
let (events_tx, _events_rx) = unbounded();
let (commands_tx, commands_rx) = unbounded();
commands_tx
.send(Command::Subscribe(tokens.clone()))
.unwrap();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let (notices, _notices_rx) = unbounded();
let client = thread::spawn(move || {
run(
&url,
0,
&[events_tx],
&commands_rx,
&notices,
SILENT_AFTER,
&client_stop,
)
});
let connections = server.join().unwrap();
stop.store(true, Ordering::Relaxed);
client.join().unwrap();
let mut tokens = tokens;
tokens.sort();
for chunks in connections {
assert!(chunks.iter().all(|chunk| chunk.len() <= CHUNK));
let mut subscribed = chunks.concat();
subscribed.sort();
assert_eq!(subscribed, tokens);
}
}
/// A local server that accepts two connections of `total` tokens, each
/// handled by `serve` once subscribed, and answers what the client
/// subscribed on each.
fn serving_twice(
total: usize,
serve: fn(&mut WebSocket<TcpStream>),
) -> (String, thread::JoinHandle<Vec<Vec<Vec<String>>>>) {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/ws/market", listener.local_addr().unwrap());
let server = thread::spawn(move || {
(0..2)
.map(|_| {
let (stream, _) = listener.accept().unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut socket = tungstenite::accept(stream).unwrap();
let chunks = subscribed(&mut socket, total);
serve(&mut socket);
chunks
})
.collect()
});
(url, server)
}
/// Runs a connection of `tokens` against `url` until `until` returns,
/// then stops it: the notices it sent.
fn client(
url: String,
tokens: Vec<String>,
silent_after: Duration,
until: impl FnOnce(&Sender<Command>),
) -> Vec<Notice> {
let (events_tx, _events_rx) = unbounded();
let (commands_tx, commands_rx) = unbounded();
commands_tx.send(Command::Subscribe(tokens)).unwrap();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let (notices, notices_rx) = unbounded();
let client = thread::spawn(move || {
run(
&url,
0,
&[events_tx],
&commands_rx,
&notices,
silent_after,
&client_stop,
)
});
until(&commands_tx);
stop.store(true, Ordering::Relaxed);
client.join().unwrap();
notices_rx.try_iter().collect()
}
#[test]
fn a_silent_connection_reconnects_with_its_tokens() {
let tokens: Vec<String> = (0..10).map(|token| token.to_string()).collect();
// Silent: reads what comes (the keepalives) and answers nothing,
// until the client closes.
let (url, server) = serving_twice(tokens.len(), |socket| while socket.read().is_ok() {});
let notices = client(url, tokens.clone(), Duration::from_millis(500), |_| {
thread::sleep(Duration::from_millis(2_500));
});
let connections = server.join().unwrap();
assert_eq!(connections.len(), 2);
assert!(connections.iter().all(|chunks| chunks.concat() == tokens));
assert_eq!(
notices[..3],
[Notice::Up(0), Notice::Silent(0), Notice::Down(0)]
);
assert_eq!(notices[3], Notice::Up(0));
}
#[test]
fn a_recycle_closes_and_subscribes_every_token_again() {
let tokens: Vec<String> = (0..10).map(|token| token.to_string()).collect();
// Answers the keepalives, as the exchange does, until closed.
let (url, server) = serving_twice(tokens.len(), |socket| {
while let Ok(message) = socket.read() {
if message.is_text() {
_ = socket.send(Message::text("PONG"));
}
}
});
let notices = client(url, tokens.clone(), SILENT_AFTER, |commands| {
thread::sleep(Duration::from_millis(300));
commands.send(Command::Recycle).unwrap();
thread::sleep(Duration::from_millis(1_800));
});
let connections = server.join().unwrap();
assert!(connections.iter().all(|chunks| chunks.concat() == tokens));
assert_eq!(notices, [Notice::Up(0), Notice::Down(0), Notice::Up(0)]);
}
/// The recording's first frame holds the opening books of 40 tokens, 20
/// markets: each book goes to its market's book thread, with its share of
/// the frame's bytes.
#[test]
fn a_frame_of_several_markets_reaches_each_markets_book_thread() {
#[derive(serde::Deserialize)]
struct Line {
text: String,
}
let path = concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/polymarket-frames.jsonl"
);
let recording = std::fs::read_to_string(path).unwrap();
let line: Line = serde_json::from_str(recording.lines().next().unwrap()).unwrap();
let (senders, receivers): (Vec<_>, Vec<_>) = (0..crate::polymarket::BOOK_THREADS)
.map(|_| unbounded())
.unzip();
assert!(route(Utc::now(), &line.text, 3, &senders));
let mut books = 0;
for (thread, receiver) in receivers.iter().enumerate() {
for received in receiver.try_iter() {
let Received::Event(arrival) = received else {
panic!("not an event: {received:?}");
};
let market = arrival.event.market().unwrap().to_string();
assert_eq!(book_thread(&market), thread);
assert_eq!(arrival.connection, 3);
assert_eq!(arrival.bytes, line.text.len() as u64 / 40);
books += 1;
}
}
assert_eq!(books, 40);
// Neither a keepalive's answer nor an unreadable frame reaches one.
assert!(route(Utc::now(), "PONG", 3, &senders));
assert!(route(Utc::now(), "not json", 3, &senders));
assert!(receivers.iter().all(|receiver| receiver.is_empty()));
// A book thread gone stops the connection.
drop(receivers);
assert!(!route(Utc::now(), &line.text, 3, &senders));
}
#[test]
fn commands_update_the_token_set_and_say_what_to_send() {
let mut tokens = BTreeSet::new();
let operation = |message: &String| -> (String, Vec<String>) {
let message: serde_json::Value = serde_json::from_str(message).unwrap();
let tokens = serde_json::from_value(message["assets_ids"].clone()).unwrap();
(message["operation"].as_str().unwrap().to_string(), tokens)
};
let sent = |messages: Vec<String>| messages.iter().map(operation).collect::<Vec<_>>();
let op = |name: &str, tokens: &[&str]| {
(
name.to_string(),
tokens.iter().map(|token| token.to_string()).collect(),
)
};
let subscribe = Command::Subscribe(vec!["a".into(), "b".into()]);
assert_eq!(
sent(apply(&mut tokens, &subscribe)),
[op("subscribe", &["a", "b"])]
);
assert_eq!(
sent(apply(&mut tokens, &Command::Resubscribe("a".into()))),
[op("unsubscribe", &["a"]), op("subscribe", &["a"])]
);
let unsubscribe = Command::Unsubscribe(vec!["a".into()]);
assert_eq!(
sent(apply(&mut tokens, &unsubscribe)),
[op("unsubscribe", &["a"])]
);
assert!(apply(&mut tokens, &Command::Resubscribe("a".into())).is_empty());
assert_eq!(tokens.into_iter().collect::<Vec<_>>(), ["b"]);
}
}
+272
View File
@@ -0,0 +1,272 @@
//! The connection loop (`src/core/socket.rs`) against a local server, with
//! Polymarket's codec.
use std::collections::BTreeSet;
use std::net::{TcpListener, TcpStream};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread;
use std::time::Duration;
use chrono::Utc;
use crossbeam::channel::{Sender, unbounded};
use serde_json::json;
use tungstenite::{Message, WebSocket};
use super::{CHUNK, Polymarket, Session};
use crate::core::Event;
use crate::core::socket::{Command, Notice, Routed, SILENT_AFTER, apply, route, run};
use crate::core::{BOOK_THREADS, Received, book_thread};
/// The tokens each message of one connection subscribed, after the first.
fn subscribed(socket: &mut WebSocket<TcpStream>, tokens: usize) -> Vec<Vec<String>> {
let first = socket.read().unwrap().into_text().unwrap();
let first: serde_json::Value = serde_json::from_str(&first).unwrap();
assert_eq!(first, json!({"type": "market", "assets_ids": []}));
let mut chunks = Vec::new();
while chunks.iter().map(Vec::len).sum::<usize>() < tokens {
let text = socket.read().unwrap().into_text().unwrap();
let message: serde_json::Value = serde_json::from_str(&text).unwrap();
assert_eq!(message["operation"], "subscribe");
chunks.push(serde_json::from_value(message["assets_ids"].clone()).unwrap());
}
chunks
}
#[test]
fn a_reconnect_subscribes_every_token_again_in_chunks() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/ws/market", listener.local_addr().unwrap());
let tokens: Vec<String> = (0..1_200).map(|token| token.to_string()).collect();
let total = tokens.len();
let server = thread::spawn(move || {
let mut connections = Vec::new();
for _ in 0..2 {
let (stream, _) = listener.accept().unwrap();
// A client that never sends fails the test rather than hangs it.
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut socket = tungstenite::accept(stream).unwrap();
connections.push(subscribed(&mut socket, total));
// The client reconnects after this close; after the second,
// it finds no server and stops.
socket.close(None).unwrap();
while socket.read().is_ok() {}
}
connections
});
let (events_tx, _events_rx) = unbounded();
let (commands_tx, commands_rx) = unbounded();
commands_tx
.send(Command::Subscribe(tokens.clone()))
.unwrap();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let (notices, _notices_rx) = unbounded();
let client = thread::spawn(move || {
run(
&Polymarket { url },
0,
&[events_tx],
&commands_rx,
&notices,
SILENT_AFTER,
&client_stop,
)
});
let connections = server.join().unwrap();
stop.store(true, Ordering::Relaxed);
client.join().unwrap();
let mut tokens = tokens;
tokens.sort();
for chunks in connections {
assert!(chunks.iter().all(|chunk| chunk.len() <= CHUNK));
let mut subscribed = chunks.concat();
subscribed.sort();
assert_eq!(subscribed, tokens);
}
}
/// A local server that accepts two connections of `total` tokens, each
/// handled by `serve` once subscribed, and answers what the client
/// subscribed on each.
fn serving_twice(
total: usize,
serve: fn(&mut WebSocket<TcpStream>),
) -> (String, thread::JoinHandle<Vec<Vec<Vec<String>>>>) {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/ws/market", listener.local_addr().unwrap());
let server = thread::spawn(move || {
(0..2)
.map(|_| {
let (stream, _) = listener.accept().unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut socket = tungstenite::accept(stream).unwrap();
let chunks = subscribed(&mut socket, total);
serve(&mut socket);
chunks
})
.collect()
});
(url, server)
}
/// Runs a connection of `tokens` against `url` until `until` returns,
/// then stops it: the notices it sent.
fn client(
url: String,
tokens: Vec<String>,
silent_after: Duration,
until: impl FnOnce(&Sender<Command>),
) -> Vec<Notice> {
let (events_tx, _events_rx) = unbounded();
let (commands_tx, commands_rx) = unbounded();
commands_tx.send(Command::Subscribe(tokens)).unwrap();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let (notices, notices_rx) = unbounded();
let client = thread::spawn(move || {
run(
&Polymarket { url },
0,
&[events_tx],
&commands_rx,
&notices,
silent_after,
&client_stop,
)
});
until(&commands_tx);
stop.store(true, Ordering::Relaxed);
client.join().unwrap();
notices_rx.try_iter().collect()
}
#[test]
fn a_silent_connection_reconnects_with_its_tokens() {
let tokens: Vec<String> = (0..10).map(|token| token.to_string()).collect();
// Silent: reads what comes (the keepalives) and answers nothing,
// until the client closes.
let (url, server) = serving_twice(tokens.len(), |socket| while socket.read().is_ok() {});
let notices = client(url, tokens.clone(), Duration::from_millis(500), |_| {
thread::sleep(Duration::from_millis(2_500));
});
let connections = server.join().unwrap();
assert_eq!(connections.len(), 2);
assert!(connections.iter().all(|chunks| chunks.concat() == tokens));
assert_eq!(
notices[..3],
[Notice::Up(0), Notice::Silent(0), Notice::Down(0)]
);
assert_eq!(notices[3], Notice::Up(0));
}
#[test]
fn a_recycle_closes_and_subscribes_every_token_again() {
let tokens: Vec<String> = (0..10).map(|token| token.to_string()).collect();
// Answers the keepalives, as the exchange does, until closed.
let (url, server) = serving_twice(tokens.len(), |socket| {
while let Ok(message) = socket.read() {
if message.is_text() {
_ = socket.send(Message::text("PONG"));
}
}
});
let notices = client(url, tokens.clone(), SILENT_AFTER, |commands| {
thread::sleep(Duration::from_millis(300));
commands.send(Command::Recycle).unwrap();
thread::sleep(Duration::from_millis(1_800));
});
let connections = server.join().unwrap();
assert!(connections.iter().all(|chunks| chunks.concat() == tokens));
assert_eq!(notices, [Notice::Up(0), Notice::Down(0), Notice::Up(0)]);
}
/// The recording's first frame holds the opening books of 40 tokens, 20
/// markets: each book goes to its market's book thread, with its share of
/// the frame's bytes.
#[test]
fn a_frame_of_several_markets_reaches_each_markets_book_thread() {
#[derive(serde::Deserialize)]
struct Line {
text: String,
}
let path = concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/polymarket-frames.jsonl"
);
let recording = std::fs::read_to_string(path).unwrap();
let line: Line = serde_json::from_str(recording.lines().next().unwrap()).unwrap();
let (senders, receivers): (Vec<_>, Vec<_>) = (0..BOOK_THREADS).map(|_| unbounded()).unzip();
let mut session = Session;
let mut routed = |text: &str, senders: &[Sender<Received<_>>]| {
route(&mut session, Utc::now(), text, 3, senders, "polymarket")
};
assert_eq!(routed(&line.text, &senders), Routed::Sent);
let mut books = 0;
for (thread, receiver) in receivers.iter().enumerate() {
for received in receiver.try_iter() {
let Received::Event(arrival) = received else {
panic!("not an event: {received:?}");
};
let market = arrival.event.market().unwrap().to_string();
assert_eq!(book_thread(&market), thread);
assert_eq!(arrival.connection, 3);
assert_eq!(arrival.bytes, line.text.len() as u64 / 40);
books += 1;
}
}
assert_eq!(books, 40);
// Neither a keepalive's answer nor an unreadable frame reaches one.
assert_eq!(routed("PONG", &senders), Routed::Sent);
assert_eq!(routed("not json", &senders), Routed::Sent);
assert!(receivers.iter().all(|receiver| receiver.is_empty()));
// A book thread gone stops the connection.
drop(receivers);
assert_eq!(routed(&line.text, &senders), Routed::BookThreadGone);
}
#[test]
fn commands_update_the_token_set_and_say_what_to_send() {
let mut tokens = BTreeSet::new();
let mut session = Session;
let operation = |message: &String| -> (String, Vec<String>) {
let message: serde_json::Value = serde_json::from_str(message).unwrap();
let tokens = serde_json::from_value(message["assets_ids"].clone()).unwrap();
(message["operation"].as_str().unwrap().to_string(), tokens)
};
let sent = |messages: Vec<String>| messages.iter().map(operation).collect::<Vec<_>>();
let op = |name: &str, tokens: &[&str]| {
(
name.to_string(),
tokens.iter().map(|token| token.to_string()).collect(),
)
};
let subscribe = Command::Subscribe(vec!["a".into(), "b".into()]);
assert_eq!(
sent(apply(&mut session, &mut tokens, &subscribe)),
[op("subscribe", &["a", "b"])]
);
assert_eq!(
sent(apply(
&mut session,
&mut tokens,
&Command::Resubscribe("a".into())
)),
[op("unsubscribe", &["a"]), op("subscribe", &["a"])]
);
let unsubscribe = Command::Unsubscribe(vec!["a".into()]);
assert_eq!(
sent(apply(&mut session, &mut tokens, &unsubscribe)),
[op("unsubscribe", &["a"])]
);
assert!(apply(&mut session, &mut tokens, &Command::Resubscribe("a".into())).is_empty());
assert_eq!(tokens.into_iter().collect::<Vec<_>>(), ["b"]);
}
+6 -30
View File
@@ -1,35 +1,8 @@
//! The market channel's messages, as the exchange writes them, and how a
//! frame's text becomes events (docs/notes.md).
use chrono::{DateTime, Utc};
use serde::Deserialize;
/// What a connection sends a book thread.
// Nearly every message is an event: boxing it to shrink the two rare markers
// would cost an allocation per event for nothing.
#[allow(clippy::large_enum_variant)]
#[derive(Debug)]
pub enum Received {
Event(Arrival),
/// The connection came up: its subscriptions are being sent.
Up(usize),
/// The connection went down: nothing more comes from it until it is up.
Down(usize),
}
/// One event of a frame on its way to its market's book thread.
#[derive(Debug)]
pub struct Arrival {
/// The frame's receive time, stamped right after the socket read.
pub received_at: DateTime<Utc>,
/// The connection that delivered it.
pub connection: usize,
/// Its share of the frame's bytes: a frame is one market's change, or
/// books of several, each taking its share.
pub bytes: u64,
pub event: Event,
}
/// The events of a frame's text; none for the keepalive's `PONG` and for
/// `NO NEW ASSETS`.
pub fn decode(text: &str) -> Result<Vec<Event>, serde_json::Error> {
@@ -42,9 +15,8 @@ pub fn decode(text: &str) -> Result<Vec<Event>, serde_json::Error> {
}
}
impl Event {
/// The market it belongs to, which decides its book thread.
pub fn market(&self) -> Option<&str> {
impl crate::core::Event for Event {
fn market(&self) -> Option<&str> {
match self {
Event::Book(book) => Some(&book.market),
Event::PriceChange(change) => Some(&change.market),
@@ -54,6 +26,10 @@ impl Event {
Event::Other => None,
}
}
fn is_book(&self) -> bool {
matches!(self, Event::Book(_))
}
}
#[derive(Debug, Deserialize)]