Skip to main content

Governance Pipeline

Metatate transforms human-readable policy definitions into machine-enforceable governance rules through a six-stage pipeline. Each stage is deterministic and auditable, producing artifacts that feed into the next.

Pipeline Stages​

1. Policy Definition​

Authors create governance policies using a structured YAML format. Each policy declares its scope (which tables it applies to) and a set of typed instructions covering classification, masking, usage guidance, AI governance, retention, and access control.

Policies are versioned. The editor validates structure before submission.

2. Instruction Extraction​

The compiler parses the policy YAML and normalizes each instruction into a canonical form. This stage handles:

  • Expanding shorthand notation into full instruction records
  • Assigning unique instruction IDs
  • Resolving column references against the catalog metadata
  • Validating parameter schemas per instruction type

The output is a flat list of typed, validated instructions keyed by table and instruction type.

3. Diff Calculation​

The diff engine compares the compiled instructions against the current deployed state in the governance tables. It produces a change plan with three operation types:

  • Add — new instructions not present in current state
  • Update — instructions where parameters, priority, or scope changed
  • Delete — instructions removed from the policy or orphaned by scope changes

The diff is table-level: each affected table gets its own set of operations. Unchanged tables are skipped entirely.

4. Plan Validation​

The policy manager validates the change plan before execution:

  • Conflict resolution — when multiple policies target the same table and column, priority and enforcement mode determine which instruction takes precedence
  • Completeness check — ensures no table is left in a partial state
  • Dependency validation — masking instructions require a corresponding classification

The plan is presented to the user for review before deployment.

5. Materialization​

The materializer executes the approved plan using an atomic write strategy. All changes are validated and written to the governance tables in a single transaction, ensuring they are never in an inconsistent state. Each deployment creates a snapshot and logs every decision for auditability.

If any step fails, the live tables remain unchanged.

6. MCP Tool Exposure​

Once materialized, the governance state is immediately available through the MCP (Model Context Protocol) tools registered in the core schema. These tools provide read access to:

  • Table discovery and catalog browsing
  • Policy and instruction lookup per table
  • Column classification and masking details
  • Usage rule evaluation and data meaning retrieval
  • Query validation against deployed rules

External AI agents and internal Cortex-powered features query these tools to make governance-aware decisions at runtime.

Design Principles​

  • Deterministic — the same policy YAML always produces the same governance state, regardless of deployment order
  • Auditable — every deployment creates a snapshot and decision log entries
  • Atomic — governance tables are never partially updated
  • Idempotent — redeploying an unchanged policy produces no changes
  • Incremental — only affected tables are touched during deployment