On this page
In short#
The decision engine turns an organization's own signals into ranked, recorded proposals that a person approves, carries out and later confirms. It abstains when the evidence cannot support a number, and never executes an action.
Selecting a playbook is a numeric rule evaluation; trusting a playbook or lever is statistics over confirmed results. Neither uses a language model or semantic similarity. How far each number can be trusted is graded on the evidence ladder. Upstream: From signal to decision. Outcomes: Measurement and honest numbers. Screens: Decision engine screens.
Rule selection#
Numeric predicates compare an observed signal with a threshold. Evaluation is three-valued and unknown never matches, so a missing signal fails closed. Each selection emits a per-predicate receipt (predicate, observed value, threshold) re-derivable from stored inputs; display text is not part of the rule. Grammar: Marketing playbooks.
Off-take statistics#
On the commerce lever plane one pure module holds the statistics, as scaled integers, so no surface re-derives them. The forecast is a regression with confidence bands, not a trained model.
| Method | What it is |
|---|---|
| Correlation | Lagged partial-rank correlation of a lever with GMV, controlled for campaigns, winsorized inputs |
| Forecast | Least-squares regression on lever, campaign and season; cold-start prior; walk-forward evaluation |
| Confidence | Correlation strength weighted by the amount of evidence, in basis points |
Abstain gates#
Each gate returns a named state instead of a number; downstream inherits it, so no recommendation is produced.
| Condition | Result |
|---|---|
| Fewer than twenty weeks of history | learning |
| Correlation below the reliability threshold | no reliable signal |
| A lever that never varied | Nothing: a constant carries no information |
| Recommended discount above the cap | Rejected by the guardrail |
| No data from the declared conversion feed | Recorded as learning, with the reason |
| Diagnostic scan or off-policy evaluation below floor | Abstains with the floor named |
| No lever with confirmed positive lift | cold start: no lever is recommended |
Policy learner and off-policy evaluation#
Each lever's normal posterior starts at 0 bp and folds the measured lift of owner-confirmed, treatment-arm decisions only; a lever is served once it has at least three observations and a mean above zero. Selection is deterministic via a seeded draw (no clock, no ambient randomness), and the ledger records the true probability: one for the chosen action, zero for the rest. A randomized selector with a replayable draw is tested; no production path calls it. Off-policy evaluation reports IPS, self-normalized IPS and doubly-robust estimates with intervals and an effective-sample-size check, or refuses with a named reason (deterministic_log, zero_overlap, ess_below_floor). On deterministic logs it refuses, so no policy is promoted on an off-policy figure; a decision enters the estimator only with an in-range recorded probability, a real variance and a known lever (learned policy on Project status).
Guardrails and approval#
Blocking guardrails run before display: a hard discount cap, an objective-against-cap check and the off-policy floor. They gate; they do not simulate. Each mutation checks owner or org admin on the stored row and claims it with a status-predicated update before any command is written, so a repeated approve inserts nothing. One decision in ten, chosen by a hash of the decision, is held out as control: its commands are stored as held out, the board tells the owner not to run it by hand, its outcome abstains, and confirming it or approving its lesson is refused. An approved action becomes manual steps; the engine writes to no marketplace, ad account or channel. A playbook started by an alert or another system trigger proposes as a viewer, so each of its steps waits for a person.
The episode ledger#
Each lifecycle event is one immutable row. Writes are idempotent per organization and brand, so a re-run scan writes nothing new.
| Field | Records |
|---|---|
| Eligible set | Every candidate, its three-valued outcome and its rank in the deterministic order |
| Chosen action | Its true probability and the logging policy's name and version; probabilities sum to one |
| Reward window | Conversion, days and a late cutoff three days after close, declared before any outcome |
| Version pins | Engine and rule version, a hash of the signals seen, a hash of the receipt |
| Abstention reason | Learning, no reliable signal, missing signal, below floor, estimate, no match, no data |
Replay#
Replay re-runs the pinned rule version on the frozen signal values and compares receipt hashes. Where the episode pins the champion policy that ranked it, replay re-ranks the candidates with that configuration and reports reproduced or divergent; an episode ranked with the defaults reports unpinned.
| Verdict | Meaning |
|---|---|
| Identical | The receipt re-derives exactly |
| Divergent | Names the leaf and field that differ: signal, operator, threshold, observed value, outcome |
| Unreplayable | Abstained, no rule version, version gone, or another engine version; not a pass |
Memory#
The card cites what happened the last time a rule matched. Memory is derived at read time from confirmed outcomes, so no separate store can drift. Relevance is lexical, not embedding similarity, and memory never changes which rule fires.
| Aspect | Rule |
|---|---|
| Eligible | Owner-confirmed hit, partial or miss; same playbook; last year; not superseded; not sample data |
| Ranked by | Specificity (entity, segment, rule), recency halving in about five weeks, a bonus for a lesson |
| Supersession | Same rule, segment and period: the later confirmation wins and names its predecessor |
| Shown | Three citations by default; nothing when nothing is confirmed |
What it reads and what it never reads#
| Direction | Scope |
|---|---|
| Reads | First-party signals, the declared conversion, the playbook corpus, owner confirmations |
| Never reads as evidence | Third-party market estimates; a diagnostic may cite one as a hypothesis (Data planes) |
| Never writes | Approvals, activations, rejections or confirmations from a machine path; anything outside the platform |
Where it stops#
- No trained machine-learning model runs in production; prediction is regression on the organization's own history.
- Intervention readouts are observational: a before-and-after readout is disclosed as confounded and is directional evidence. The measured lift compares each decision with its own earlier window and is not yet adjusted by the control arm. Geo designs are out of scope.
- Memory has expiry and supersession but no deletion audit.
- There is no trade-off simulator. A randomized logged selection, a data watermark writer and a causal compute service are not built; dated status per capability is on Project status.
Decisions#
| Decision | Why | Enforced by |
|---|---|---|
| Every lifecycle event is one row in an append-only ledger | A field not captured at decision time can never be added honestly afterwards | An update-and-truncate trigger, no runtime delete grant, verify:data checking probabilities |
| A person approves before anything is activated; the worker proposes and measures | A loop that approves its own proposals, or learns from its own verdicts, grades itself | Router ownership checks, an audit-log entry per mutation, confirmed-only track record |
| Every selection receipt can be replayed from frozen inputs | An explanation that cannot be re-derived is a story | The replay build gate: zero divergences to pass, "skipped" (not "passed") when nothing replays |
Sources#
- Microsoft Decision Service: explore, log, learn, deploy
- Sachdeva, Su and Joachims, off-policy evaluation with deficient support, KDD 2020
- Swaminathan and Joachims, self-normalized estimator, NeurIPS 2015
- Wachter, Mittelstadt and Russell, counterfactual explanations
- Adtributor, NSDI 2014
- Google SRE workbook, alerting on SLOs
- Geifman and El-Yaniv, selective prediction, NeurIPS 2017