Skip to content
MSO CloudDocumentation

Measurement and honest numbers

On this page

In short#

A decision is judged against the organization's own first-party data, on the one conversion it declared. When the evidence cannot support a number, every path returns a named state (no data, learning, insufficient) instead of a value.

Proposed numberA metric, rate orrecommendationEnough evidence?Confidence inbasis pointsyesEvery number grounded?Numbers and quotestrace to a sourceyesOwner approves?Owner ororg adminyesLedger and monitorsEvent and evidence;calibration, driftnononoNamed state, no numberno data, learning, insufficientRecordednothing activated

Measurement has three jobs: reconcile an activated decision against first-party results, keep every displayed number traceable to one definition, and refuse a number the data cannot support. The same rules decide when a playbook's measured history may change its rank.

Check Rule
Confidence Evidence-weighted, capped below certainty, stored in basis points; the card shows the computed number
Calibration Stated against realised hit rate per bucket; a thin bucket returns insufficient
Drift Recent half against history: smoothed PSI and two-sample KS; too short to split returns insufficient
Proof-carrying numbers Each narrative number traces to a fact, each quote to the corpus, or output blocks

The data assistant applies the same rule and drops any sentence whose number is not traceable to a supplied fact (Data assistant).

The definable conversion#

Each organization declares exactly one conversion; it counts events or sums their value. An org admin sets it in Marketing Systems OS > Operating settings (Decision engine screens).

Conversion Measures Source
orders Count of first-party orders Marketplace orders
gmv_vnd Value of first-party orders (VND) Marketplace orders
aov GMV divided by all placed orders Marketplace orders
leads Count of marketing-qualified leads CRM leads
deals Count of won deals CRM deals
deal_value Value of won deals CRM deals

Reconciling a decision#

Commerce conversions count every placed order in the window, cancelled and returned ones included, so the aov conversion is not the AOV the reports show (NMV divided by confirmed orders). Leads resolve through one CRM qualified-lead definition, deals through won CRM deals. The outcome screen shows expected against measured, the difference, a verdict inside the organization's tolerance bands, and a system-health block where each field says measured, abstention or insufficient.

One approved decision in ten is held out as control and not carried out. At window close the worker writes the lift of both arms: the conversion after execution against the equal-length window before it. A control decision's outcome abstains, so it reaches no calibration bucket, track record or policy fit. The lift is per decision and is not yet adjusted by the control arm.

Commerce data is the verification mechanism, not the subject: a marketing hypothesis is checked against recorded purchases, leads and deals the organization did not narrate. Data validation is separate: it checks that the pipeline read an uploaded export correctly, and does not measure decisions.

When measured history may change a ranking#

A confirmed hit scores one, a partial a half, a miss zero, folded into a conservative prior. Ranking uses the posterior's lower bound, not its mean, so a small perfect record does not outrank a large good one. Ranking permutes and never filters, so eligibility stays a numeric predicate.

0Nothing: rule-based rank, no percentage1 to 7Shown as counts; may not move a rank8 to 29May reorder playbooks; counts only, no rate30 or moreA rate may be shown, always with its interval8 · minimum to rank30 · minimum to show a rateowner-confirmed outcomesRank by the lower bound, not the meanHit 1, partial 0.5, miss 0, folded into a conservative prior; ranking uses the posterior's 5th percentile,so 6 of 9 cannot outrank 600 of 1000. Zero hits of n shows 0 to 3/n.
What confirmed history may do, by count. 8 is where the data outweighs the prior (strength 4); 30 is where a percentage is allowed on screen. Both are product choices, overridable per organization.

Below the ranking threshold the pooled cross-segment record is used and the receipt names its denominator. Learning never edits a rule: a rule changes only through draft, review and activation by a person, as a new version beside the archived one.

Policy versions: champion and challenger#

A ranking policy is a named, versioned configuration. A challenger runs in shadow; a three-state promotion gate then reads its off-policy report, and marginal means a person may promote with the reasons on screen. Promotion and rollback are the same human-approved transition, each recorded as an episode. The scan ranks with the champion's configuration when exactly one valid champion exists, otherwise with the defaults, and the receipt pins the version it used. No promotion has run in production: selection is deterministic, so the off-policy report refuses today (Project status). The champion and challenger cell on the outcome screen compares the current policy with an exploration-weighted variant of itself, not a second policy serving traffic.

1Challenger, day 0created, shadows the champion2Shadow floor, day 14and at least 30 episodes3Gatereads the off-policy reportshadow: every episode recorded with its probabilitypassmargin clears the floormarginalshadow floor not met, or watchfailreport refused, drift, no marginPromoteowner approvesmarginal: reasonsshown on screenkeeps the championWhat never happens on its ownThe champion is never demoted automatically; sustained drift demotes the challenger. Promotion writes policy_promoted with the approving identity;rollback is the same transition reversed (policy_rolled_back). Until a randomized selector serves traffic the report refuses on a deterministic log, so the gate fails.
A challenger's path. Nothing moves without a person: the gate grades, an owner promotes, and the same recorded transition reversed is the rollback.

What it reads and what it never reads#

Direction Scope
Reads First-party orders, CRM leads and deals, owner confirmations, the organization's signal history
Never reads as anchor Third-party market estimates: own plane, labelled, never a KPI or outcome (Data planes)
Does not compute Brand-health composites or framework cascades; from social data, only net sentiment

Where it stops#

Readouts are before-and-after on first-party data, labelled confounded; they are not causal estimates. Drift produces a status and demotes a challenger; it does not change what serves. Dated status per capability: Project status.

Decisions#

Decision Why Enforced by
A number the evidence cannot support is returned as a named state A plausible wrong number is acted on; an honest "not enough data" is not Nullable metric return types, a per-field honesty flag, assertions that fail on absence-as-zero
Each metric, statistic and conversion is defined in one module Two definitions drift; a decision cannot be checked against a dashboard computed differently Single-source audits over statistics, bindings, signals and conversion, plus verify:data
Thresholds 8 and 30 are product choices, overridable per organization No published threshold; at 8 data outweighs the prior, at 30 a rate stops misleading One ranking module holds both; a rate below 30 is never rendered