On this page
MSO Cloud reads market and business signals, turns them into diagnoses and recommendations, and records every decision with its reasoning. What separates it from a reporting tool is a single rule enforced in code: when the evidence cannot support a number, the system says so instead of producing one.
The governance and truth layer enforces that rule. Capability grades for the whole platform live on the Maturity Model page.
Abstain over fabricate#
Every metric path can return null, na, or insufficient rather than a value. Thin data does not become a confident-looking number. The rule is applied at each stage that could invent one:
- The statistics loop abstains when a series has fewer than 20 weeks of history, or when the correlation it found is weaker than a fixed threshold. It reports
learningorno_reliable_signalinstead of a coefficient. - A constant-lever guard stops a correlation from being read off a variable that never moved.
- The recommendation policy has an off-policy evaluation floor and refuses to act when propensity data is missing.
- The conversion resolver returns null when a first-party feed is absent, rather than reading absence as zero.
Honest absence is the default across the codebase, not a special case. A missing value is stored as missing, never as 0.
The validation flow#
A proposed number passes through a fixed sequence of gates before it can be shown or acted on. Any gate can stop it.
flowchart TD
R["Reasoning output: a proposed number or recommendation"] --> AG{"Abstain gate: is the evidence sufficient?"}
AG -->|"no"| NA["Return null / na / insufficient. No number is shown."]
AG -->|"yes"| CS["Confidence scoring: evidence-weighted, stored in basis points"]
CS --> PCN{"Proof-carrying gate: is every number and quote grounded?"}
PCN -->|"no"| NA
PCN -->|"yes"| HG{"Human approval gate: owner or org admin approves"}
HG -->|"rejected"| STOP["Recorded and stopped. Nothing is activated."]
HG -->|"approved"| LED["Append-only ledger records the event and the evidence it used"]
LED --> MON["Drift and calibration monitoring read outcomes back"]
MON -.->|"informs the next decision"| RConfidence scoring#
Every coefficient the statistics loop reports carries a confidence value. It is evidence-weighted: more corroborating observations raise it, and it is capped below certainty. The value is stored as an integer in basis points, so the number shown on a recommendation card is the same number the learning layer computed, not a re-derivation. There is no free-floating confidence label detached from the evidence that produced it.
Calibration#
Calibration checks whether the confidence the system stated matches the outcomes that actually occurred. It buckets confirmed decision outcomes and compares the stated hit-rate against the realised hit-rate per bucket. When a bucket holds too few confirmed outcomes to judge, it returns insufficient rather than a calibration score. This keeps the system honest about its own honesty: a confidence number is only trusted once outcomes have been measured against it.
Drift monitoring#
Drift monitoring detects when the data feeding a decision has moved away from the distribution the decision was learned on. Two measures run: a Laplace-smoothed Population Stability Index (PSI) and a two-sample Kolmogorov-Smirnov (KS) test, each compared against standard interpretation bands. Both are computed at request time against current data, not read from a stale cached job. When inputs drift, the monitoring surface shows it.
Proof-carrying numbers#
Numbers and narrative quotes are gated before they reach a reader. A number must trace to the evidence that produced it, and a quoted phrase in an audience narrative must be grounded in the source corpus. The gate blocks output that fails this check. A sentence in a generated narrative cannot cite a statistic or a quotation that the underlying data does not support.
Single-source math#
Each metric definition lives in one module. No dashboard, router, or narrative re-derives a correlation, a conversion count, or a rollup on its own. This is enforced mechanically: static audits (single-source checks over the statistics loop, the playbook rule bindings, the signal registry, and the conversion metric) plus data reconciliation assertions run in the pre-push and CI gate. A second definition of the same number fails the build. The result is that a figure means the same thing on every surface that shows it.
The append-only decision ledger#
Every decision lifecycle event is written once to an event ledger. A row records the event, the eligible set considered, the propensity, and a hash of the context and the machine receipt used at decision time, pinned to a version. A database trigger blocks UPDATE and TRUNCATE on the ledger, so a recorded event cannot be quietly rewritten after the fact. This is the foundation for reproducibility: the evidence a decision used is preserved with the decision.
Replay
The next step now exists. Given a recorded decision, the platform re-runs the rule body at the version that decision pinned, over the signal values that decision froze, rebuilds the receipt, and compares its hash to the hash the record stored.
The verdict has three states, and keeping them apart is the honest part. Identical means the receipt re-derived exactly. Divergent names the condition and the field that differ. Unreplayable means the decision cannot be judged this way at all, with the reason stated - it was an abstention and froze no receipt, it names no rule version, the corpus no longer holds that version, or a different engine version produced it. An unreplayable decision is never counted as a pass, and never as a failure.
This runs as a build gate, and it passes only on zero divergences. When there is nothing to replay it reports that it was skipped, rather than reporting success over an empty table.
What replay establishes is that the decision kernel re-derives its own receipt. What it does not yet establish is that the warehouse would still produce those signal values from the same window; that needs a per-source watermark pinned at decision time, which the record does not carry. The distinction is deliberate and is stated here rather than glossed.
Human approval gates#
The platform recommends; a person decides. Every decision mutation passes an ownership and approval gate before it can move forward. The gate requires an owner or an org admin, checks the approval on the real decision row in a way that is safe against a race between check and write, and writes an audit-log entry for the action. Activation of a recommendation is a human-approved step, recorded.
Multi-tenant isolation at the claim level#
Every governed number is scoped to one organization. Row-level security isolates each tenant's rows, so one tenant's decisions, signals, and claims are never readable by another. On top of that, the governance surface carries an honesty discriminator on every field: the value it reports says whether it is a real measured number, an abstention, or insufficient data. A user interface reading that surface cannot paint a confident green over a value that was never measured.
What the engine remembers, and what it refuses to remember#
The engine can cite what happened the last time the same rule matched the same thing. Three filters decide what is eligible to be cited, and each of them exists to stop a specific dishonesty.
- Only an outcome an owner confirmed can become a memory. The machine's own verdict is display; it never feeds back.
- Only a hit, a partial or a miss counts. An abstention is not evidence of anything.
- Only the same rule, within the last year, not superseded by a later confirmation.
Supersession is derived rather than stored: for the same rule, segment and period, the later confirmation wins, the earlier one stays in the record, and the successor is named. Nothing is rewritten, which is the ledger's discipline applied one layer up.
The limit worth stating: a citation appears on the decision card. It does not influence which rule fires or which draft is primary. When nothing has been confirmed, the card shows nothing rather than a hedge.
When measured history is allowed to change a ranking#
A playbook's confirmed history can reorder how playbooks are ranked. Two thresholds govern it, and both are documented product judgements rather than citations, because no published threshold exists for either.
- Below eight confirmed outcomes, the history is computed and shown but may not move a rank. The record states that the engine looked and declined to use it, which is a checkable claim rather than silence.
- Below thirty confirmed outcomes, no percentage is rendered at all; the surface shows raw counts instead. A rate on a small sample implies a precision the sample does not have.
The ranking uses the lower bound of the posterior rather than its mean, so a small perfect record does not outrank a large good one. Every rate that is rendered ships with an interval, including the zero-event case. Ranking permutes the candidate list and never filters it, so eligibility stays a pure numeric predicate; and learning never rewrites a rule, only its rank and its priors. A rule body changes through draft, review and activation by a person, as a new version beside the archived old one.
Champion and challenger#
Policy monitoring has a foundation in place, including a challenger scaffold: an in-place variant of the recommendation policy used to watch for drift. This is not two policies running side by side in production and being compared on live traffic. The side-by-side champion and challenger design is a roadmap item; today the scaffold supports monitoring, not a production A/B of policies.
What is shipped and what is not#
Shipped and enforced in code today: abstain gates, confidence scoring, calibration, PSI and KS drift monitoring, proof-carrying number and quote gating, single-source math enforced by audits and reconciliation, the append-only decision ledger with a database trigger, a per-decision logging contract covering the whole candidate set and a reward window declared in advance, executable receipt replay enforced as a build gate, confirmed-outcome memory with derived supersession, evidence-gated ranking with its two thresholds, human approval gates with audit logging, and per-tenant isolation.
Foundation or roadmap, labeled as such: replay of the inputs as well as the kernel, which needs a pinned data watermark; a production champion and challenger comparison, where a scaffold exists and side-by-side policies do not, together with the promotion report, approval and rollback that would have to accompany it; off-policy evaluation diagnostics - effective sample size, a self-normalized estimator, confidence intervals - none of which exist; randomized selection, without which every recorded probability is certainty; drift that triggers an operational action rather than a chart; and a deletion audit for memory.
Sources#
packages/metrics/src/decision-replay.ts,scripts/audit/decision-replay.ts- replay and its build gatepackages/metrics/src/decision-memory.ts- confirmed-outcome memory and supersessionpackages/metrics/src/playbook-stats-rank.ts- the eight and thirty thresholds, and the lower-bound rankingpackages/metrics/src/decision-episodes.ts,packages/schema/sql/append-only-guards.sql- the ledger and its immutabilitypackages/metrics/src/policy.ts- the probability contract and what off-policy evaluation still lacksscripts/audit/data-reconciliation.ts- the data gate that re-derives the logging contract in SQL