The Decision Engine

The decision loop#

flowchart TD
    C["1 Collect<br/>connectors write a dated signal snapshot"] --> D["2 Detect<br/>evaluate numeric rules per entity"]
    D --> E["3 Explain<br/>turn the receipt into a sentence"]
    E --> P["4 Propose<br/>rank the matched set"]
    P --> G["5 Guardrail<br/>blocking checks on the real row"]
    G --> A["6 Approve<br/>a person decides, with a reason"]
    A --> X["7 Execute<br/>a person acts, off-platform"]
    X --> M["8 Measure<br/>machine verdict at the declared horizon"]
    M --> F["9 Confirm<br/>the owner ratifies or overrides"]
    F --> L["10 Learn<br/>rebuild the track record"]
    L -->|"changes the rank at step 4"| P

Every step writes to the same append-only ledger, so the loop is readable after the fact rather than reconstructed from logs.

Two facts about that table are load-bearing. No machine verdict is ever learned from - only an owner-confirmed outcome moves the track record. And a worker may write the proposal and the measurement; it may never write an approval, an activation, a rejection or a confirmation. Those are human transitions, each behind an ownership gate and an audit-log entry.

The loop is not uniformly implemented. Collect, propose, guardrail, approve, execute, measure, confirm and learn all run. Detect writes its own ledger entry on the off-take lever plane but not on the marketing rule plane, where the eligible set rides inside the proposal record instead. Explain writes no record of its own by design: the explanation is the frozen receipt, not a second artefact.

Two layers never mix. Choosing which playbook applies is layer one, a numeric rule evaluation. Learning how much to trust a lever and how much it moved the outcome is layer two, statistics over measured results. Neither layer uses a language model or semantic matching.

1. The numeric rule DSL#

A playbook is selected by evaluating numeric predicates over named signals: an observed value compared against a threshold. The evaluation is three-valued (true, false, unknown): if a signal is missing, its predicate is unknown and the rule does not match, so a missing signal fails closed rather than guessing. The selection is never done by an LLM and never by vector similarity.

Every selection emits a per-predicate receipt: the predicate that fired, the observed value, and the threshold at decision time. The receipt is a record of why the rule matched, reproducible from the stored inputs. The human-readable description attached to a rule is display text only; correcting its wording does not change which rule fires.

2. Off-take statistics#

The statistics live in one pure module, so no surface re-derives a number a different way. All values are stored as signed integers scaled by a fixed factor, so no floating-point drift enters the math.

• Correlation. The engine computes a lagged partial-rank correlation between social levers and GMV, controlled for campaign activity, over the org's own weekly history. Inputs are winsorized before ranking.
• Walk-forward evaluation. Forecasts are evaluated walk-forward on held-out later periods, not scored on the same data they were fit on.
• Forecast. Prediction is an ordinary-least-squares regression of GMV on the lagged lever, campaign, and season, blended with a cold-start prior when history is short. It is a regression forecast on the org's own history, with confidence bands. It is not a trained machine-learning model.
• Confidence. Confidence is scored from the strength of the correlation weighted by the amount of evidence, and stored as an integer basis-points value.

3. Abstain gates#

The abstain gates are the most rigorous part of the engine. Each returns a null or an "insufficient" state rather than a number when its condition is not met:

• Fewer than twenty weeks of history returns a "learning" state.
• A correlation below the reliability threshold returns "no reliable signal."
• A lever that does not vary returns nothing (a constant lever carries no information).
• A recommended discount above the cap is rejected by the guardrail.
• The diagnostic scan and the off-policy evaluation each have their own floor.

Everything downstream inherits these gates: if the statistics abstain, no recommendation is produced.

4. The policy learner#

The policy learner ranks levers from measured results. It maintains per-lever posteriors (a normal-conjugate model) in a policy table, and today selects deterministically from those posteriors: a seeded, reproducible draw, which is reproducible selection, not exploration. The draw uses no system clock and no ambient randomness, so the same inputs always produce the same choice.

The ledger records the true probability of the chosen action for every decision. Because selection is deterministic, that probability is always certainty, and the alternatives are recorded at zero - they were unreachable under a total order. The engine also holds the mathematics for the honest alternative: a randomized selection that spreads a declared exploration share across the eligible set, with the probabilities summing exactly to the whole and the random draw injected rather than generated inside, so a replay can reproduce it. That code path exists, is tested, and nothing calls it.

That bounds what the off-policy estimator can honestly claim. A doubly-structured estimator is implemented and runs as a floor guardrail, but over a log where every probability is certainty it has no support in the statistical sense (Sachdeva, Su, and Joachims, KDD 2020), so it is scaffolding awaiting randomized selection, not a working evaluation. The learner reports its own state honestly as cold-start, insufficient, or acting, and the Maturity page grades this layer as substrate.

What did change is the log's integrity. A decision only enters the estimator's input when it carries a real recorded probability inside the valid range, a real variance, and a known lever; rows that disagree with themselves are dropped. An earlier version imputed a probability from an exploration flag - a fabricated weight that made every term downstream of it baseless - and that imputation is gone.

5. Guardrails#

Before a recommendation is surfaced, it passes guardrails: a hard cap on the recommended discount, an objective-versus-cap comparison, and the off-policy floor. Guardrails gate the recommendation. They are not a what-if scenario simulator; the engine does not simulate outcomes across a space of levers.

6. Human approval#

Every decision mutation passes an owner-and-approval gate on the real decision row, and the gate is safe against a check-then-act race. Each mutation is written to an audit log. The platform recommends; operators act, and every action is approved by a human. Nothing in the engine pushes a change back to an ad platform or a channel.

7. The append-only ledger#

Each lifecycle event is written as one immutable row to an append-only event ledger. A database trigger blocks updates and truncation. Each row carries the eligible set of choices, the probability of the chosen action, hashes of the context and the receipt, and a version pin of the decision kernel. This ledger is MSO Cloud's own design: no published industry architecture was adopted for it, and it is stricter than the run logs of commercial rule engines, which typically retain days of history rather than an immutable record. The full contract a row satisfies is in the next section.

After activation, the outcome is measured against the org's first-party orders (or its chosen conversion), and the confirmed verdict and lesson are stored as separate metadata. That measurement feeds the next cycle.

8. The episode record#

Each lifecycle event is one row in the ledger, written under a contract that is stricter than "log what happened". The purpose is that a decision can be judged later without trusting anyone's memory of it.

An episode records:

• Which event it is: evaluated, proposed, approved, rejected, activated, cancelled, measured, outcome confirmed, lesson confirmed, or abstained.
• The whole eligible set at that step, not only the winner - every playbook that was in contention, each with a three-valued outcome (pass, fail, or unknown because a signal it names was not computed) and its rank in the deterministic order.
• The action chosen, and the true probability the selecting policy gave it, alongside the identifier of the selection rule and the name and version of the logging policy. Probabilities across the eligible set sum exactly to the whole; nothing is imputed.
• The reward window, declared before any outcome exists: which conversion metric, which days, and a late cutoff three days after the window closes. A result arriving after the cutoff is discarded rather than folded in, because a delayed result quietly credited to a decision is how a loop learns the wrong lesson. A window that is known at decision time to be ungradeable is marked as such.
• Version pins and hashes: the engine version, the rule version, a hash of the signals the decision saw, and a hash of the receipt it produced.
• Why it abstained, when it did: still learning, no reliable signal, a missing signal, below the evidence floor, provenance is an estimate, no rule matched, or no conversion data.

Two mechanical properties make the record trustworthy rather than merely present. It is idempotent: a re-run of the same scan writes nothing new, because each event carries a key unique within the organization and brand. The accepted cost is stated rather than hidden - a lifecycle transition that genuinely happens twice, such as approve, cancel, approve, records the first occurrence unless the caller deliberately distinguishes them. And it is immutable at the database: a trigger rejects any update or truncation of the ledger with a privilege error, and no runtime role holds delete permission. A recorded event cannot be quietly rewritten after the fact.

9. Replay#

Replay is the executable form of "we can show our work". Given an episode, the engine takes the signal values that episode froze, re-runs the rule body at the version the episode pinned, rebuilds the receipt, hashes it, and compares that hash to the one the episode stored.

The verdict is three-valued, and that is the point:

• Identical - the receipt re-derives exactly.
• Divergent - it does not, and the report names the leaf and the field that differ: the signal, the operator, the threshold, the observed value, the outcome, the resolved threshold, or the baseline.
• Unreplayable - with a stated reason: the episode was an abstention and froze no receipt, it carries no rule version, the corpus no longer holds that version, or it was decided by a different engine version. An unreplayable episode is never counted as a pass and never as a failure.

This runs as a build gate alongside the tenancy checks. The gate passes only on zero divergences. When there are no episodes that froze a selection receipt, it reports that it was skipped rather than reporting success - a gate that goes green over an empty table proves nothing, and saying so is part of the contract.

What replay does not yet establish: whether today's warehouse would still compute those same signal values from the same window. That needs a per-source watermark pinned at decision time, which the ledger does not carry. Replay proves that the kernel re-derives the receipt; it does not yet prove that the inputs are reproducible from the warehouse.

10. Memory#

The engine can cite what happened the last time a rule matched the same thing. That memory is derived at read time from confirmed outcomes; there is no separate memory store to drift from the ledger.

The filters come before any ranking, and they are what make the citation worth reading:

• Only outcomes an owner confirmed are eligible. A machine verdict never becomes a memory.
• Only hit, partial and miss count. Abstentions and learning states are not evidence.
• Only the same playbook, within the last year, not superseded, and not sample data.

What survives is ranked by three things added together: how specifically it matches (the same entity beats the same segment beats the same rule), how recent the confirmation is (decaying at a rate that halves in about five weeks), and a bonus when the owner wrote a lesson rather than only a verdict. Ties break deterministically. By default three citations are offered.

Supersession is derived, not stored: for a given rule, segment and period, the later confirmation wins, the earlier one stays in the record, and the successor is named. Nothing is rewritten - that is the same discipline the ledger enforces, applied one layer up.

The shape follows the published retrieval scoring of generative-agent memory (recency, importance, relevance, equally weighted), with two departures stated rather than hidden: relevance here is lexical specificity, not an embedding similarity, because a vector index has to beat a lexical baseline on a measured benchmark before it earns its complexity; and recency decays from confirmation, not from last retrieval, because retrievals are not logged.

The honest limit: memory is a citation on the decision card, not an input to the proposal. It does not change which rule fires or which draft is primary. When nothing is confirmed, the citation returns nothing and the card shows nothing, rather than a hedge.

11. Track record#

A playbook's measured history may reorder how playbooks are ranked. It may never rewrite one.

Confirmed outcomes are scored with partial credit - a hit counts one, a partial counts a half, a miss counts nothing - and folded into a conservative prior. What the ranking uses is not the resulting mean but the lower bound of its 5th percentile. The reason is a specific failure it prevents: on the mean, thirty hits and zero misses outranks three hundred hits and twenty misses, because a small sample can look perfect. On the lower bound, a wide uncertainty pays for its own width.

Two gates decide what that history is allowed to do:

Both numbers are documented product judgements, not citations: there is no published threshold for when a win rate stops being more misleading than a raw count. What is citable, and is applied at every sample size, is that a rate always ships with an interval, and that a zero-event rate uses the rule of three rather than claiming certainty.

Three properties hold by construction. With no sufficient evidence anywhere, the order is byte-identical to the rule-based one. Ranking permutes; it never filters - eligibility stays a pure numeric predicate, so learning can never quietly remove a rule from contention. And a well-evidenced rule cannot jump over an unevidenced rule it was already behind.

When a specific segment has not reached the threshold, the denominator falls back to the rule pooled across segments, and the receipt records which denominator was used - a pooled number is never presented as a segment one.

Nothing here edits a rule. Learning changes ranking and priors; a rule body changes only through draft, review and activation by a person, as a new version beside the archived old one.

What the engine does not do#

These limits are part of the design, not gaps to be talked around.

• No trained machine-learning model runs in production. Prediction is regression on the org's own history.
• No randomized or switchback experiments run live. Pre and post readouts are descriptive and are disclosed as confounded; they are directional evidence, never a causal result.
• There is no trade-off simulator.
• Selection is still deterministic, so off-policy evaluation still cannot be trusted. The ledger now records the true probability of the chosen action, and the mathematics for a randomized selection that logs its own probability is written and tested. Nothing produces a randomized selection yet, so every recorded probability is certainty, and a doubly-structured estimator over such a log degenerates into a shrunken on-policy mean. The estimator runs as a floor guardrail and is honest about being scaffolding.
• Off-policy evaluation has no diagnostics. There is no effective-sample-size measure, no self-normalized estimator, and no confidence interval on an off-policy number. Until those exist, an off-policy figure is not something to promote a policy on.
• Replay proves the kernel, not the inputs. Receipt re-derivation runs as a gate; a pinned data watermark that would prove the same window still yields the same values does not exist.
• Champion and challenger policies do not run side by side in production. The comparison function and the promotion margin exist in code but are exercised only by their tests, and there is no promotion report, approval step or rollback path.
• Memory has no deletion audit. Expiry and supersession exist; a record of what was removed and by whom does not.
• The engine recommends; it does not execute. Generic action execution is a table-and-types stub, and the one real outbound path (audience sync) has no channel connector registered, so it runs in dry-run mode.

Roadmap: methods adopted from industry practice#

Each item below is a published method with a named source, selected against the engine's actual state, in dependency order. Items are roadmap unless the item says which half of it now exists.

1. Randomized lever selection with logged propensities. The explore, log, learn, deploy contract of the Microsoft Decision Service: every decision records the candidate set, the action taken, and the probability the logging policy assigned it. Thompson sampling produces those probabilities as a by-product (Chapelle and Li, NIPS 2011); updates are batched to match weekly, delayed marketing outcomes. Every later item depends on this one. The logging half is done: the record now carries the candidate set, the chosen action, the true probability, and the logging policy's name and version, and a data gate re-derives in the database that the probabilities across a candidate set sum exactly to the whole. The randomization half is not: selection is still deterministic, so every logged probability is certainty. The record had to come first, because the ledger refuses updates - a field not written at decision time can never be added later.
2. Support diagnostic before any off-policy number. Following Sachdeva, Su, and Joachims (KDD 2020), the estimator reports effective sample size and abstains with "cannot evaluate" when action coverage is deficient, instead of printing an uninterpretable value. Not built: there is no effective-sample-size measure in the code today.
3. Self-normalized estimation, benchmarked before trusting. SNIPS (Swaminathan and Joachims, NIPS 2015) alongside the doubly-structured estimator, with the choice made against the Open Bandit Pipeline ground-truth logs rather than by assertion.
4. Shadow mode and backtest for rules. A rule can run in an evaluate-only state that logs the decisions it would have made without acting, the pattern of Uber's Mastermind, and a new rule is backtested against the org's own history before it is enabled, the discipline documented for Stripe Radar.
5. Guardrail alerting as equivalence testing. Per Microsoft's experimentation platform, alerting on every statistically significant adverse move makes an alerting system useless; each guardrail declares a tolerance band, alerts on rejection of "inside the band," and the set is corrected for multiplicity.
6. Counterfactual receipts. A rule receipt gains the flip point: the smallest input change that would have changed the decision (Wachter, Mittelstadt, and Russell). With named numeric thresholds this is nearly free, and it turns a receipt into an explanation a client can contest.
7. Impact-ranked signals. Anomalies ranked by explanatory power and surprise, the share of the money moved they account for, rather than by statistical deviation (Adtributor, NSDI 2014). Revenue and orders are additive, which is exactly this method's precondition.
8. Alert budgets with published precision. Each signal class declares a precision target, a detection window, and a reset window, following the Google SRE workbook; alert quality becomes measurable instead of anecdotal.
9. Snapshot pinning in the ledger. Each episode pins the data state it decided on (rollup watermark plus checksum) with a stated retention window, closing the gap between an append-only record and honest replay. This is now the named blocker on replay: the receipt re-derives, the inputs are not yet provably reproducible.
10. Abstain coverage as an operating metric. The share of decisions abstained, per gate, is published; an abstain gate without a coverage number cannot be audited (the selective-prediction framing of Geifman and El-Yaniv, NIPS 2017).
11. The LLM stays off numeric authorship. Assistant queries compile through the governed metric layer; a wrong question fails with an error instead of returning a plausible wrong number. This is already the architecture; it is stated here as doctrine because the industry evidence shows the failure asymmetry is the point.

Behind gates, after the above: champion/challenger promotion with a three-state judgment (pass, marginal with human approval, fail) plus a promotion report, an approval identity and a rollback path - none of which exists today, and all of which is meaningless until selections carry real probabilities; synthetic-counterfactual incrementality readouts (time-based regression / CausalImpact) replacing naive pre/post where a credible control series exists; geo experiments with a power analysis first; variance reduction (CUPED) only after randomized assignment exists to attach it to; and media-mix modeling only on top of a standing experiment program, since its own documentation requires experiment calibration and two to three years of clean weekly history.

Related#

• Governance and Honest Numbers
• Measurement in MSO Cloud
• Maturity Model
• Marketing Playbooks
• Morning Brief

Sources#

• packages/metrics/src/decision-episodes.ts - the episode record, its idempotency and hashing
• packages/schema/src/tables/mso-episodes.ts, packages/schema/sql/append-only-guards.sql - the ledger's shape and its immutability trigger
• packages/metrics/src/decision-replay.ts, scripts/audit/decision-replay.ts - replay and the gate it runs in
• packages/metrics/src/decision-memory.ts - what the engine remembers and what it refuses to remember
• packages/metrics/src/playbook-stats-rank.ts - the track record and its two gates
• packages/metrics/src/policy.ts, policy-loop.ts - the policy learner and the probability contract
• packages/metrics/src/offtake-loop.ts - the single writer of the ledger, and outcome measurement
• packages/metrics/src/playbook-rules.ts - the rule grammar and three-valued evaluation
• apps/web/src/server/routers/offtake.ts - the human transitions and their gates