Inside the Obermind market data engine: shared aggregation and synthetic order types

Two pieces of Obermind Stream do most of the work on a busy trading day. The market data engine turns raw venue updates into the market data each account is subscribed to, and the Aggregator decides where an order goes and what comes back. Platform 4.9 reworks both.

The market data engine now aggregates each instrument once for the whole process and computes top of book and Level 2 depth once per distinct liquidity profile, rather than repeating that work for every connected session. On top of that engine we added synthetic order types: a way for Obermind Stream to rest any order type on the platform and execute it at a venue when the trigger price is reached, so the order types available to a trader no longer depend on what the venue behind their liquidity pool happens to accept.

This article covers both, the arithmetic behind the change, and how to configure it.

Where the market data engine sits

Obermind Stream receives market data from every configured venue in a deployment's liquidity pool — the liquidity providers, trading venues and exchanges a firm has connected — and maintains that venue state once per instrument, consolidating it into a top of book best bid and offer with the Level 2 depth behind it. Three consumers subscribe to that book:

• Live sessions on Obermind Web, Obermind Terminal, the mobile apps and the API
• Resting orders waiting on a trigger price
• Downstream integrations such as the FIX API Engine and snapshot services

Why two accounts are quoted differently

Each account is served through a liquidity profile. A profile is not just a markup applied to one universal price — it defines which venues in the pool that segment actually aggregates, and only then how the resulting prices are dressed.

Four things vary between profiles, and any one of them alone is enough to make the quotes differ:

• Venue composition. A profile aggregates a chosen subset of the pool. An institutional segment might aggregate three tier one liquidity providers plus a crypto exchange, while a retail segment aggregates one provider. Different venue sets produce a different best bid and offer and different Level 2 depth, because they are literally different books.
• Per-venue weighting. Venues inside a profile do not have to count equally. A profile can lean on a primary provider and treat the rest as secondary, so two profiles built from the same venues can still resolve to a different top of book once weighting is applied.
• Size-tiered aggregation. Depth is aggregated in size tiers, so the price a profile shows for one unit of size is not the price it shows for ten. A profile that aggregates deeper tiers quotes a wider top of book but sustains larger orders, and two segments can be tiered differently on identical venues.
• Commercial treatment. On top of its own aggregate, a profile carries its own markup, spread floor and depth presentation, which is how a firm prices its segments differently on the same underlying venues.

So a customised liquidity pool is exactly that: the same deployment, the same venue connections, but a per-segment view assembled from the venues that segment is entitled to, weighted and tiered the way that segment is priced. Two accounts on the same liquidity profile are subscribed to identical market data. Two accounts on different profiles may be quoted from different venues entirely.

That is the whole basis of the optimisation. The natural unit of pricing work is the liquidity profile, not the account, and a deployment has a handful of profiles no matter how many accounts it runs.

From per session aggregation to a shared market data engine

The shape of the work

The original model gave every connected session its own aggregation. A venue update reached each session's Aggregator instance, and each one rebuilt the same venue state, derived its own top of book and Level 2 depth, and pushed the result back through the full event pipeline before it reached that one subscriber. Two sessions on the same liquidity profile did that identical work twice.

The shared market data engine collapses the repeated part. One update maintains one venue book, each liquidity profile aggregates its own venues once, and then a light delivery step reaches every subscribed session.

Fan out to sessions is deliberately direct rather than back through the event bus. Routing each delivery through the bus would reinstate the per session pipeline cost, which is the cost the engine exists to remove.

The cost model

The definitions below are per instrument, and the two expressions are the work each model performs every second.

A   connected sessions subscribed to the instrument
P   distinct liquidity profiles across those sessions
R   venue update rate, updates per second
Ch  cost of one heavy unit: book rebuild plus full event pipeline re-entry
Cp  cost of aggregating one liquidity profile's weighted, tiered venue set
    into its own top of book
Cl  cost of delivering an already computed price to one subscribed session

per session aggregation   W_before = R * A * Ch
shared market data engine W_engine = R * (Ch + P * Cp + A * Cl)

speedup                   S = W_before / W_engine
                            = (A * Ch) / (Ch + P * Cp + A * Cl)

Cl is a dictionary lookup, a struct copy and a queue write. Cp is arithmetic on two numbers. Both are small next to Ch, which walks the book and re-enters the pipeline. Writing k = Cp / Ch and m = Cl / Ch, with k and m well below one, the ratio simplifies — and when session count is moderate relative to 1/m it approaches a clean form.

S ≈ A / (1 + P*k + A*m)

S ≈ A / (1 + P)        when A is moderate relative to 1/m

The reading is straightforward. The old cost scaled with session count. The new cost scales with liquidity profile count, and profile count is a property of how a deployment segments its clients and its venue entitlements, not of how many clients are subscribed.

Allocation and Level 2 depth

Two secondary quantities move the same way. The second is the consolidated depth rebuild, which is the expensive part of serving a Level 2 view. In both, Pa is the number of profiles that actually shift the price.

objects allocated per update      Alloc_before = A
                                  Alloc_engine = 1 + Pa

Level 2 depth rebuilds            L2_before = A
per update                        L2_engine = 1 + Pa

Sessions on a liquidity profile that does not shift the price receive the same object, because there is nothing to change. The engine only allocates a copy when a profile genuinely produces a different price, which happens when its venue set, its weighting, its size tiers or its markup make it diverge. Depth behaves identically: profiles that do not shift the top of book share one Level 2 aggregate, and each profile that does needs its own, because prices are written in place.

A worked example

Take a mid sized deployment on one currency pair: 500 sessions subscribed, four liquidity profiles across them, three of which shift the price, and 40 updates per second from the liquidity pool.

per session aggregation
  heavy units per second        40 * 500      = 20,000
  allocations per second        40 * 500      = 20,000
  Level 2 rebuilds per second   40 * 500      = 20,000

shared market data engine
  heavy units per second        40 * 1        = 40
  top of book computations      40 * 4        = 160
  allocations per second        40 * (1 + 3)  = 160
  Level 2 rebuilds per second   40 * (1 + 3)  = 160

Heavy work drops by a factor of 500 on that instrument. Allocation and Level 2 rebuild drop by a factor of 125. Delivery to each subscribed session still happens 20,000 times per second, but each delivery is now a lookup and a queue write rather than a full pipeline pass.

Scale that across an instrument set and the shape is the same: the term that used to multiply by session count is gone.

What stayed exactly the same

The design constraint that picked this approach was backwards compatibility, and it holds by construction rather than by migration.

• Each account keeps its own Aggregator instance, under the same identifier, owning the same portfolio and the same order maps.
• Order identity, ownership and persistence are byte for byte unchanged. Orders written before the change load unchanged, and orders written after load on an earlier build.
• The wire contract is preserved. Sessions were always quoted by their own algorithm instance and that identifier is on the wire, so the engine stamps it back onto the quote rather than letting a venue identifier through.
• What moved out of the per account instance is market data only: subscriptions, consolidated book, top of book and Level 2 caches, and trigger evaluation.

The engine ships behind a single setting and is off by default. Turning it off and restarting returns a deployment to per session aggregation, with no data written differently in either mode.

Configuring the liquidity pool

Which venues make up the aggregated liquidity pool is now configuration rather than a compiled list. It is an Aggregator property, set once per Stream instance.

<Algorithm>
  <FileName>OberMind.Algos.dll</FileName>
  <TypeName>Aggregator</TypeName>
  <Load>true</Load>
  <Properties>
    <Property name="LiquidityProviders">88,75</Property>
  </Properties>
</Algorithm>

Venues can be given by numeric id or by name. Leaving it empty keeps the built in list, so an existing deployment behaves exactly as before until it opts in.

This property is the pool, not the per-segment view: it declares every liquidity provider, trading venue and exchange the deployment aggregates. Which of those venues a given segment is quoted from, at what weighting and in what size tiers, stays a property of its liquidity profile — so widening the pool here makes new venues available to profiles rather than pushing them at every client.

Two further venue behaviours became configuration in the same release, both of which used to be fixed per venue in code.

VenueTimeInForce declares the time in force values each venue accepts, optionally followed by the value to use for market orders. An order arriving with anything else is rewritten to the first accepted value, or to the market default when it is a market order.

VenueSizeChecks lists the venues that enforce the instrument minimum, maximum and step size at order acceptance. It takes venue ids or names.

<Properties>
  <Property name="VenueTimeInForce">88:Day,GoodTillCancel:Day;75:ImmediateOrCancel</Property>
  <Property name="VenueSizeChecks">88,75</Property>
</Properties>

Empty keeps the previous behaviour in both cases. Adding a venue to a deployment is now an edit in Obermind Dash rather than a build.

Price triggers move into the market data engine

Once the engine owns the consolidated book, it also becomes the right place to evaluate resting price triggers.

A resting order registers a trigger with the engine. Whenever its liquidity profile's top of book changes, the engine calls back with that profile's own bid and offer, aggregated from that profile's own venues. An order is never evaluated against another profile's price, and the instrument stays subscribed for as long as any session or any resting trigger needs it.

Two properties of this matter operationally:

• Demand is the union of subscribed sessions and resting triggers. An instrument no session is subscribed to, but on which an order is resting, stays subscribed. When the last session logs out and the last trigger clears, the subscription is released.
• A venue dropping out does not fire triggers. Reconsolidating the book across the remaining venues can look like a large move. The next real market update evaluates triggers against a price the market actually made.

That second point is the foundation for the feature in the next section.

Synthetic order types

Not every venue accepts every order type. A venue may take market and limit orders and decline stops, or take market orders only. Historically that meant the order types offered to a trader were the intersection of what every venue in their liquidity pool supported.

Synthetic order types remove that constraint. Obermind Stream rests the order itself, watches the trader's own price, and sends a market order to the venue for the held quantity when the trigger condition is met. The venue fill is reported back as a fill of the original order.

Lifecycle

The held order is an ordinary order in every respect that matters to the rest of the platform. It is written to the order series, appears as working to the client and in Obermind Dash, and accepts cancel and replace.

The held order itself never fills. The venue market order carries the fill, so the position is booked exactly once and the client report is rebuilt from the original command with that account's normal pricing treatment applied.

Cancel and replace are accepted while the order is held and handled locally. While a venue order is in flight the order is briefly locked, and a cancel or replace arriving in that window is rejected rather than racing the venue.

Fill price policies

When a held order triggers, the market order takes whatever the venue is showing. The price reported back to the client is then governed by a fill price policy, chosen per deployment through SyntheticFillPricePolicy, so a firm can decide how slippage between the trigger and the venue fill is presented on the client order.

One rule constrains every policy: no policy may change the quantity. The client is always filled for exactly what the venue executed. Price differences accumulate as algorithm versus client performance and are visible as such, which keeps the invariant that a venue position always has a client order against it.

Venue attribution

While an order rests it is held by an internal provider, which is how Obermind Dash shows that an order is resting on the platform rather than working at a venue. Once it triggers, the venue order and the resulting fill report are stamped with the destination venue, so fills, positions and transaction rows all attribute the execution to where it actually happened. The hold appears in the order message history and in the trigger log line, which records order id, type, quantity, destination and the exact prices the decision used.

Durability

A held order survives everything a working order survives.

Concretely, a held order continues to work through a client logging out, repeated amendments that move it toward the market, an Obermind Stream restart, and a restart where the owning account never connects at all. A venue order that was in flight across a restart is re-attached rather than orphaned.

Fill semantics worth knowing

Holding a type changes how it fills, and it is worth being explicit with clients about this. A stop the venue itself supports rests at the venue and fills at its stop price. The same stop held on the platform executes as a market order sent on trigger, so it takes the price the venue is showing at that moment and can slip. This is inherent to the mechanism: the reason to hold is that the venue cannot rest the type.

The trade off is normally worth it. The alternative is that the order type is unavailable to that trader.

Configuration

Order type support per venue is one property, in the same Aggregator block.

<Algorithm>
  <FileName>OberMind.Algos.dll</FileName>
  <TypeName>Aggregator</TypeName>
  <Load>true</Load>
  <Properties>
    <Property name="SyntheticOrderProviders">88:Market,Limit;75:Market,Limit,Stop</Property>
    <Property name="SyntheticFillPricePolicy">Actual</Property>
    <Property name="SyntheticTriggerBufferTicks">0</Property>
  </Properties>
</Algorithm>

Read it as "this venue accepts these types". Anything not listed is held. Market orders are never held whatever the entry says, and a venue with no entry passes every order type through unchanged, which is what makes the empty default a no op.

SyntheticTriggerBufferTicks requires the market to be through the trigger by a given number of ticks before firing, which is useful on instruments where a single print can touch a level and step straight back.

Execution without a live session

Both features needed one more thing to be true: an account must be able to trade when nobody is logged into it.

Orders arrive for accounts with no session all the time. They come from the FIX API Engine, from admin and risk tools, and from resting orders that trigger while the owner is offline. Obermind Stream now builds the execution context for such an account on demand, so those orders are accepted and routed normally rather than waiting for a session.

With the market data engine on, they also route better. Order routing picks the venue behind the best price, and a sessionless account previously had no book of its own to consult, so routing fell back to a fixed order of preference. The engine's consolidated book exists whether anyone is connected or not, so the same order now routes to the venue actually showing the best top of book. Where no book exists the previous fallback still applies, so behaviour degrades to the old path rather than failing.

Operating it

Everything new is off or empty by default. A deployment that changes no settings behaves exactly as before.

The instrument trace is the tool for verifying the engine is doing what you expect. It prints the consolidated top of book and, beside it, the price each liquidity profile derived from it, so a deployment can confirm in one line that its profiles are aggregating the venues they should and that a session landed in the profile intended. Note that it samples at roughly one line per second per side rather than printing every update, so read it as a sample. The trigger log is the authoritative record of any individual order decision, and prints the exact prices used.

Every one of these is editable from Obermind Dash under Framework, alongside a new Configurations Guide tab documenting each configuration file, key and setting with copy ready XML.

Suggested rollout

1. Deploy the build with UseAggregationEngine left false. Nothing changes.
2. Set LiquidityProviders to the venues the deployment actually routes to.
3. Turn the engine on in staging and check, in order: top of book and Level 2 arrive for a subscribed instrument, two accounts on different liquidity profiles are subscribed to prices built from their own venue sets, a FIX API Engine session receives the profile it was assigned, a resting order still triggers after its owner logs out, and a resting order still triggers after a restart with no session at all.
4. Introduce SyntheticOrderProviders for one venue and one order type, and confirm the trigger and fill against a test account before widening it.

What this changes for an operator

The practical results:

• Capacity is decoupled from session count. Adding connected traders adds delivery cost, not aggregation cost. The consolidated book for an instrument is computed once regardless of how many sessions are subscribed to it, whether they arrive on the web platform, the mobile apps or the FIX API Engine.
• Segmentation is cheap. Adding a liquidity profile adds one top of book and one Level 2 computation per update, not one per account on it. A firm can give each segment its own venue set, weighting, size tiers and commercial treatment without paying for it in aggregation throughput.
• Order types stop depending on venue capability. A trader gets stops and stop limits whether or not the venue behind their liquidity pool rests them.
• Resting orders are durable. They survive logout and restart, and they work for accounts that are never connected, which is the normal state for API-driven and managed accounts.
• Venue configuration is operational. The liquidity pool, the accepted order types, the time in force rules and the size checks are all settings now.

The market data engine and synthetic order types ship in Obermind Platform 4.9 and are available on Obermind Stream deployments today.