Data Foundation

Three data planes that never mix#

Mixing data across planes is the most common class of bug this platform has had to close off, so the separation is enforced as an invariant, not a convention.

• 1st-party marketplace. Orders and inventory from marketplace connectors and CSV/XLSX upload. This is the money plane: it is the platform's source of truth for revenue and volume, and it is checked in an automated reconciliation gate against a ground truth kept outside the codebase.
• 1st-party marketing. Ad spend, web behavior, and search behavior pulled from the org's own ad, web-analytics, and search-console accounts. This plane has its own reconciliation and is never blended into marketplace figures; a marketing platform's own conversion counts do not sum across platforms and are never presented as marketplace revenue.
• 3rd-party market intelligence. Category and competitor estimates, and social listening, sourced from third parties. This data is stored as an estimate or a signal. It can inform a hypothesis or a recommendation, but it never feeds a first-party KPI and it is excluded from the reconciliation gate that covers the first two planes, because a market estimate has no external ground truth to reconcile against.

Audience data sits inside the first-party bucket even though the underlying listening feed is a third-party crawl: it is a client-approved deliverable, so it is treated, and reconciled, as first-party rather than as a fourth plane.

flowchart TB
  subgraph FP["1st-party: reconciled against ground truth"]
    ECOM["ecom · marketplace orders"]
    MKT["marketing · ad, web, search"]
    AUD["audience · client-approved corpus"]
    CRMP["crm · leads, deals (thin)"]
  end
  subgraph TP["3rd-party: estimate or signal only"]
    SOC["social · third-party listening"]
    MI["market · category & competitor estimates"]
  end
  FP --> KPI["First-party KPIs and Measure Spine"]
  TP -.->|"signal only, never a KPI"| DEC["Decision engine"]
  KPI --> DEC

The six pillars#

Connector families#

Connectors group by the plane they feed. Marketplace connectors (TikTok Shop, Shopee, Lazada, plus CSV import for Shopee and Lazada) feed the ecom pillar. Marketing connectors (Google Ads, Meta Ads, TikTok Ads, Shopee Ads, Google Analytics 4, Google Search Console, Google Sheets, Brevo, LinkedIn Page) feed the marketing and crm pillars. Social and market connectors (social listening and market-data providers) feed the social and market pillars and never cross into the first-party planes.

The medallion warehouse#

First-party data moves through three layers. Bronze holds raw, append-only ingest, deduplicated by content hash. Silver holds typed facts produced by per-stream promoters that are idempotent, so replaying Bronze reproduces the same Silver deterministically. Gold holds the materialized rollups dashboards actually read. Third-party estimates skip this pipeline entirely: they land directly in their own store, carry an estimate label wherever they are shown, and are never promoted into a first-party Gold table. The full mechanics, including the reconciliation rules that make Gold numbers trustworthy, are in Data Warehouse.

The signal registry#

Underneath the pillars, MSO Cloud keeps a signal registry: a versioned, org-scoped catalog of named signals, each with a schema, a unit, a time grain, and a provenance tier of first-party, third-party, or derived. A companion feature store holds the actual per-period values for each signal, and a signal with no data for a period is stored as absent rather than as zero. Every decision the platform proposes reads from this registry, so a decision's evidence is always traceable to a named, provenance-tagged signal rather than an unlabeled number.

Related#

• Platform Overview
• Data Warehouse
• Technology
• System Architecture