Archived
feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
This commit is contained in:
24 files changed
+8984
-1214
No files matched your search
+40
-15
@@ -32,28 +32,53 @@ The collector's in-memory order book for one symbol: its bid and ask price
|
||||
levels, each with its absolute quantity.
|
||||
_Avoid_: Order book state, depth
|
||||
|
||||
**Lane**:
|
||||
A set of symbols whose books one book thread owns, fed by its own
|
||||
connections: one path from the wire to the writer. Every live symbol is in
|
||||
exactly one lane.
|
||||
_Avoid_: Shard (a technique, not this thing), group, partition
|
||||
**Connection**:
|
||||
One websocket to the exchange. Every live symbol is subscribed on two
|
||||
connections, chosen independently of each other.
|
||||
_Avoid_: Socket (for the concept; fine for the code), stream, lane (the
|
||||
retired pairing of one connection with one book thread)
|
||||
|
||||
**Load**:
|
||||
How much a market weighs on its connection: the bytes per second of its
|
||||
frames. A lane's load is the sum of its markets'.
|
||||
How much a market weighs on a connection: the bytes per second of its frames
|
||||
there. A connection's load is the sum of its markets'.
|
||||
_Avoid_: Activity, volume (the exchange's traded amount, unrelated)
|
||||
|
||||
**Connection**:
|
||||
One websocket to the exchange, carrying the symbols of one lane.
|
||||
_Avoid_: Socket (for the concept; fine for the code), stream
|
||||
**First one wins**:
|
||||
The rule for a symbol's two connections: a change is taken from whichever
|
||||
delivers it first, with that receive time, and the other's copy is dropped.
|
||||
_Avoid_: Source, primary, standby, failover (no connection has a rank)
|
||||
|
||||
**Duplicate**:
|
||||
A change another connection already delivered.
|
||||
|
||||
**Verified**:
|
||||
Said of a symbol whose book gives the exchange's book hash after every
|
||||
change. It stops being verified when a change fits on no connection in time.
|
||||
|
||||
**Held**:
|
||||
Said of a change kept unwritten because it doesn't fit the book yet: a change
|
||||
before it is still to come, from this connection or the other.
|
||||
_Avoid_: Buffered, pending, queued
|
||||
|
||||
**Rescued**:
|
||||
Said of a change one connection missed and the other delivered: the output
|
||||
has it, where a single connection would have had a loss.
|
||||
|
||||
**Follow**:
|
||||
To apply one connection's changes unchecked, while a symbol can't be
|
||||
verified and until a fresh book says where it stands.
|
||||
|
||||
**Split**:
|
||||
Moving half the markets off a connection the exchange keeps closing.
|
||||
|
||||
**Receive time**:
|
||||
When a message came off the socket, by the local clock.
|
||||
_Avoid_: Arrival time, local timestamp
|
||||
|
||||
**Loss**:
|
||||
Any message for a symbol that the collector did not apply, which leaves its
|
||||
book silently wrong.
|
||||
A change for a symbol that no connection delivered in time, which leaves its
|
||||
book silently wrong. A change only one connection missed is rescued, not
|
||||
lost.
|
||||
_Avoid_: Gap (implies sequence numbers)
|
||||
|
||||
**Resync**:
|
||||
@@ -63,9 +88,9 @@ one differs: a loss, or a subscription gap.
|
||||
_Avoid_: Re-snapshot, rebuild, refresh
|
||||
|
||||
**Subscription gap**:
|
||||
The changes our own subscribe keeps from us, whether it opens, resyncs or
|
||||
moves a symbol: the exchange holds them in the book it sends but never sends
|
||||
them as changes.
|
||||
The changes our own subscribe keeps from us, whether it opens or resyncs a
|
||||
symbol: the exchange holds them in the book it sends but never sends them as
|
||||
changes. A connection that stays subscribed meanwhile has no such gap.
|
||||
_Avoid_: Resync gap (the older, narrower name); calling it a loss (the state
|
||||
stays right; only those updates are missing)
|
||||
|
||||
|
||||
Reference in new issue
Block a user