On this page
In short#
A decision is judged against the organization's own first-party data, on the one conversion it declared. When the evidence cannot support a number, every path returns a named state (no data, learning, insufficient) instead of a value.
Measurement has three jobs: reconcile an activated decision against first-party results, keep every displayed number traceable to one definition, and refuse a number the data cannot support. The same rules decide when a playbook's measured history may change its rank.
| Check | Rule |
|---|---|
| Confidence | Evidence-weighted, capped below certainty, stored in basis points; the card shows the computed number |
| Calibration | Stated against realised hit rate per bucket; a thin bucket returns insufficient |
| Drift | Recent half against history: smoothed PSI and two-sample KS; too short to split returns insufficient |
| Proof-carrying numbers | Each narrative number traces to a fact, each quote to the corpus, or output blocks |
The data assistant applies the same rule and drops any sentence whose number is not traceable to a supplied fact (Data assistant).
The definable conversion#
Each organization declares exactly one conversion; it counts events or sums their value. An org admin sets it in Marketing Systems OS > Operating settings (Decision engine screens).
| Conversion | Measures | Source |
|---|---|---|
orders |
Count of first-party orders | Marketplace orders |
gmv_ |
Value of first-party orders (VND) | Marketplace orders |
aov |
GMV divided by all placed orders | Marketplace orders |
leads |
Count of marketing-qualified leads | CRM leads |
deals |
Count of won deals | CRM deals |
deal_ |
Value of won deals | CRM deals |
Reconciling a decision#
Commerce conversions count every placed order in the window, cancelled and returned ones included, so the aov conversion is not the AOV the reports show (NMV divided by confirmed orders). Leads resolve through one CRM qualified-lead definition, deals through won CRM deals. The outcome screen shows expected against measured, the difference, a verdict inside the organization's tolerance bands, and a system-health block where each field says measured, abstention or insufficient.
One approved decision in ten is held out as control and not carried out. At window close the worker writes the lift of both arms: the conversion after execution against the equal-length window before it. A control decision's outcome abstains, so it reaches no calibration bucket, track record or policy fit. The lift is per decision and is not yet adjusted by the control arm.
Commerce data is the verification mechanism, not the subject: a marketing hypothesis is checked against recorded purchases, leads and deals the organization did not narrate. Data validation is separate: it checks that the pipeline read an uploaded export correctly, and does not measure decisions.
When measured history may change a ranking#
A confirmed hit scores one, a partial a half, a miss zero, folded into a conservative prior. Ranking uses the posterior's lower bound, not its mean, so a small perfect record does not outrank a large good one. Ranking permutes and never filters, so eligibility stays a numeric predicate.
Below the ranking threshold the pooled cross-segment record is used and the receipt names its denominator. Learning never edits a rule: a rule changes only through draft, review and activation by a person, as a new version beside the archived one.
Policy versions: champion and challenger#
A ranking policy is a named, versioned configuration. A challenger runs in shadow; a three-state promotion gate then reads its off-policy report, and marginal means a person may promote with the reasons on screen. Promotion and rollback are the same human-approved transition, each recorded as an episode. The scan ranks with the champion's configuration when exactly one valid champion exists, otherwise with the defaults, and the receipt pins the version it used. No promotion has run in production: selection is deterministic, so the off-policy report refuses today (Project status). The champion and challenger cell on the outcome screen compares the current policy with an exploration-weighted variant of itself, not a second policy serving traffic.
What it reads and what it never reads#
| Direction | Scope |
|---|---|
| Reads | First-party orders, CRM leads and deals, owner confirmations, the organization's signal history |
| Never reads as anchor | Third-party market estimates: own plane, labelled, never a KPI or outcome (Data planes) |
| Does not compute | Brand-health composites or framework cascades; from social data, only net sentiment |
Where it stops#
Readouts are before-and-after on first-party data, labelled confounded; they are not causal estimates. Drift produces a status and demotes a challenger; it does not change what serves. Dated status per capability: Project status.
Decisions#
| Decision | Why | Enforced by |
|---|---|---|
| A number the evidence cannot support is returned as a named state | A plausible wrong number is acted on; an honest "not enough data" is not | Nullable metric return types, a per-field honesty flag, assertions that fail on absence-as-zero |
| Each metric, statistic and conversion is defined in one module | Two definitions drift; a decision cannot be checked against a dashboard computed differently | Single-source audits over statistics, bindings, signals and conversion, plus verify:data |
| Thresholds 8 and 30 are product choices, overridable per organization | No published threshold; at 8 data outweighs the prior, at 30 a rate stops misleading | One ranking module holds both; a rate below 30 is never rendered |