Measurement and honest numbers

What it is#

Measurement has three jobs: reconcile an activated decision against first-party results (the Measure Spine), keep every displayed number traceable to one definition, and refuse to show a number the data cannot support. The same rules decide when a playbook's measured history is allowed to change its rank.

The definable conversion

Each organization declares exactly one conversion that decisions are measured on. A conversion either counts events or sums their value.

A commerce organization measures on orders or GMV, a lead-generation organization on leads, a deal-driven organization on won deals. An org admin sets it in Marketing Systems OS > Operating settings; see Decision engine screens.

How it works#

Reconciling a decision

flowchart TD
  D["Activated decision"] --> C{"Declared conversion"}
  C -->|"orders, GMV, AOV"| O["First-party orders<br/>same query as the dashboard"]
  C -->|"leads"| L["CRM qualified leads"]
  C -->|"deals, deal value"| W["Won CRM deals"]
  O --> M["Measured outcome"]
  L --> M
  W --> M
  L -.->|"feed absent"| AB["null, never zero"]
  W -.->|"feed absent"| AB
  M --> V["Expected vs measured<br/>difference and verdict"]

Commerce conversions read the same first-party orders query the dashboards use, so a decision's measured result cannot disagree with the report beside it. Leads resolve through one CRM qualified-lead definition, deals through won CRM deals. When a feed is absent or thin, the resolver returns null: a missing lead feed is not zero leads. The outcome screen shows expected against measured, the difference, a verdict inside the organization's tolerance bands, and a system-health block whose every field says whether it is measured, an abstention or insufficient.

Commerce data is the verification mechanism, not the subject. Signals, diagnoses and memory are not ground truth; a marketing hypothesis is checked against behaviour the organization did not narrate, and recorded purchases, leads and deals are the closest truth available.

Separately, Data validation compares an uploaded export with what connectors ingested. It checks that the pipeline read the source correctly; it does not measure decisions.

Abstain over fabricate

Every metric path can return null, not-applicable or insufficient. Thin data does not become a confident number, and a missing value is stored as missing, never as 0. The engine's own gates are listed on The decision engine. A proposed number passes this sequence before it is shown or acted on:

flowchart TD
  R["Proposed number or recommendation"] --> AG{"Enough evidence?"}
  AG -->|"no"| NA["Named state, no number"]
  AG -->|"yes"| CS["Confidence in basis points"]
  CS --> PCN{"Every number and quote grounded?"}
  PCN -->|"no"| NA
  PCN -->|"yes"| HG{"Owner or org admin approves?"}
  HG -->|"no"| STOP["Recorded, nothing activated"]
  HG -->|"yes"| LED["Ledger records event and evidence"]
  LED --> MON["Calibration and drift read outcomes back"]

Confidence, calibration and drift

• Confidence is evidence-weighted, capped below certainty and stored in basis points, so the number on a card is the number the learning layer computed.
• Calibration buckets confirmed outcomes and compares stated hit rate with realised hit rate per bucket. A bucket with too few outcomes returns insufficient.
• Drift compares a signal's recent half with its historical half using a smoothed Population Stability Index and a two-sample Kolmogorov-Smirnov test, classified against standard bands at request time. A series too short to split returns insufficient, never stable.

Proof-carrying numbers

A number in generated narrative must trace to the fact that produced it, and a quoted phrase must be found in the source corpus; output that fails is blocked. The data assistant drops any sentence whose number is not traceable to a supplied fact (Data assistant).

When measured history may change a ranking

Confirmed outcomes score a hit as one, a partial as a half and a miss as zero, folded into a conservative prior. The ranking uses the lower bound of the posterior, not its mean, so a small perfect record does not outrank a large good one.

Both thresholds are product judgements, not values from the literature. What is applied at every size is that a rate ships with an interval and a zero-event rate uses the rule of three. With no sufficient evidence the order is identical to the rule-based one. Ranking permutes and never filters, so eligibility stays a numeric predicate. When a segment is below the threshold the pooled cross-segment record is used and the receipt names which denominator it used. Learning never edits a rule: a rule body changes only through draft, review and activation by a person, as a new version beside the archived one.

Policy versions: champion and challenger

A ranking policy is a named, versioned configuration (packages/metrics/src/policy-versions.ts). A challenger runs in shadow for at least fourteen days and thirty episodes, then a three-state promotion gate (pass, marginal, fail) reads its off-policy report; marginal means a person may promote with the reasons on screen. Promotion and rollback are the same human-approved transition in opposite directions, each recorded as an episode. Sustained drift demotes the challenger and raises an item; the champion is never demoted automatically.

The mechanism exists in code and has not run in production; because selection is deterministic, the off-policy report it depends on refuses today. The champion and challenger cell on the outcome screen is a computed comparison of the current policy with an exploration-weighted variant of itself, not a second policy serving traffic.

What it reads and what it never reads#

• Reads: first-party orders, CRM leads and deals, owner confirmations, and the organization's own signal history.
• Never reads as an anchor: third-party market estimates. They sit on their own plane, carry a third-party label, and are never a KPI or a decision outcome (Data planes and the warehouse).
• Does not compute: brand-health composite scores or framework cascades; from social data the platform computes net sentiment as one audience metric.

Limits#

• Lead and deal feeds are often thin; when they are, the resolver abstains and the decision stays in learning.
• Readouts are before-and-after on first-party data and are labelled confounded; they are not causal estimates.
• Drift produces a status and demotes a challenger; it does not change what serves.

Design intent#

Abstain over fabricate

• Decision. A number that the evidence cannot support is returned as a named state, and the interface renders that state.
• Why. A plausible wrong number is acted on; an honest "not enough data" is not.
• What it costs. Screens show empty and learning states often, especially for new organizations and thin feeds, and every surface needs copy for them.
• How it is enforced. Nullable return types through the metric layer, a per-field honesty discriminator on the governance read, and reconciliation assertions that fail when absence is read as zero.

One definition per metric

• Decision. Each metric, statistic and conversion is defined in one module; dashboards, routers, measurement and narrative import it.
• Why. Two definitions of one number drift apart, and a decision measured one way cannot be checked against a dashboard computed another.
• What it costs. A surface that needs a variant has to add it to the owning module, not compute it locally.
• How it is enforced. Single-source static audits over the statistics, playbook bindings, signal registry and conversion metric, and the metric reconciliation gate verify:data against externally held ground truth, all in the pre-push and CI gate.

Related#

• The decision engine
• From signal to decision
• Data planes and the warehouse
• Why the numbers can be trusted
• Decision engine screens
• Project status