Skip to content

Data Plane

Vinxi Kernel — Data Plane: Architecture & Design

Status: Full design, 2026-07-02. Open questions in §10. Owner: Kernel team. Supersedes: the 2026-07-02 morning draft of this document (dense ledger form — see git history) and the external “Query Fabric” material (adjudicated; preserved under docs/_archive/adjudicated-inputs/).


1. Introduction

1.1 Purpose and audience

This document describes the data plane of the Vinxi Kernel: everything that happens to a read between an SDK call and its answer — what kinds of questions the system answers, what data those answers are made of, how a query is planned and executed, how fresh an answer is guaranteed to be, and how the same design scales from a single Postgres node to a Jio-scale deployment to a phone in a trench.

Primary audience: engineers building or extending the kernel, including someone joining the team with no prior context. Secondary audience: technical reviewers evaluating the design. The document is self-contained: every kernel term it uses is defined in §1.4, and each section states its “why,” not just its “what.” Decisions are collected in an explicit decision log (§9) rather than woven into prose.

1.2 Scope

In scope: the read side — read species, the query pipeline, truth tiers and serving engines, the registry (the read planner’s control plane), consistency and read-your-write, the offline read path, tiles and standing queries, caching, and how everything scales up and down.

Out of scope, covered elsewhere: the write path (LLD §6, §9 — it appears here only where it manufactures read structures or defines read visibility), branch merge semantics (LLD §15), the sync protocol and conflict adjudication for offline devices (LLD §22), authorization internals (LLD §13.2), and the ontology/type system (LLD §19–20).

DocumentWhat it fixes
the-seam.mdThe six verbs and the governed seam — the system’s physics
lld.mdPort-by-port contracts (P0–P26), planes, type system, branches
ports-and-adapters.mdDeployment profiles and the adapter matrix
docs/superpowers/specs/2026-07-02-networkaccess-pilot-program-design.mdPilot scope, entity census, performance gates
docs/superpowers/plans/2026-07-02-m1-walking-skeleton.mdThe first build increment (M1)
docs/_archive/adjudicated-inputs/query-fabric-external-draft-2026-07.md, .../lepton-data-fabric.mdExternal material this design adjudicated (§9.4)

1.4 Glossary

Read this once; every later section assumes it.

TermMeaning
RealmA bounded slice of reality (one customer’s world, one city). Every read is scoped to a realm.
BranchA fork of a realm’s reality for what-if simulation, planning, or offline work. main is canon.
Entity / TraitThe ontology’s nouns. An entity is a typed thing (Tower, WorkOrder); a trait is a named, typed property or link on it (tower.height, cable.splicedTo).
VerbsThe six operations the governed seam admits: Query (read), Act (a governed decision that changes state), Assert (a declaration of observed evidence), Call (a crossing to an external system), Subscribe, Schedule.
StatementOne piece of evidence: (subject entity, trait, value, valid-time, provenance, confidence). Asserts carry them; the system stores them, never overwrites them.
FusionThe declared policy that combines multiple sources’ statements about one trait into a single resolved answer (e.g. trust-weighted mean).
ResolvedValueWhat a read returns for a trait: value + uncertainty + settledness + the statements it derives from. The World plane never stores a naked scalar.
SettlednessPer-trait truth status: SETTLED (sources agree), CONTESTED (they disagree), STALE (evidence too old for the trait’s freshness class), BLIND (no evidence). Queryable, never hidden.
PlanesThe five layers of the kernel: Process (transactional state, port P1), Truth (append-only logs, P2), World (resolved current interpretation, P23), Serving (indexes/caches — rebuildable, authoritative for nothing), Materialization (P8 stream / P7 batch — builds the others).
Ports (P-numbers)The kernel’s stable interfaces. Relevant here: P0 ontology registry · P1 state store · P2 truth logs · P3 lakehouse · P4 time-series store · P5 search · P7 batch transform · P8 stream materializer · P12–P15 authorization stack · P16/P17 external boundary/drivers · P23 world store · P24 tiles · P26 realtime gateway. Adapters (Postgres, FDB, ClickHouse…) implement ports per profile.
Truth tierWhich copy of truth answers a read: operational (P1/P23 current state), hot (LVC latest values), warm (windows/recent history), cold (as-of/canonical lakehouse), analytical (branch reads), derived (search/vector indexes), serving (tiles).
LVCLast-Value Cache — the hot tier: the latest value per (entity, trait) for high-volume streams, milliseconds fresh.
OrderToken{cell, seq} minted when an Act commits. The total order of decisions within a cell; every downstream store applies the same order.
WatermarkHow far a downstream store has applied that order: “applied through token T.” Every serving structure has one, because all consume the same ordered stream.
CellA shard/scale unit. Ordering and freshness promises are per-cell (see the Cells doc for placement).
CapabilityA signed, scoped token proving policy was asked (P14). Every request carries one.
Workload classThe budget class of a caller: interactive UI · dashboard · agent-exploratory (tightest) · standing query · materialization job · export.
Freshness classDeclared per trait in P0: hot(ms) / warm(s) / cold(min). Determines which tier serves “latest” and when a trait is marked STALE.
Query IRThe single logical plan language every read compiles to, regardless of surface. Node set in §4.1.
ProfilesDeployment shapes: Lean (one node, small team), Scale (Jio-scale), Air-gap (self-hosted offline site), Edge/Device (phone or field laptop). Same contracts, different adapters.
Working set / sync manifestThe compiled, signed scope of data, traits, and actions a device may hold and use offline.
RegistryThe data plane’s control plane: the catalog of what serving artifacts exist, their statistics, lineage, freshness, and lifecycle state (§4.5).
SeamThe single governed doorway between userland (apps, agents, dashboards) and the kernel. Nothing reads a store directly.

2. Overview

2.1 What the data plane is

Userland — apps, agents, dashboards, BI tools — asks questions through the seam. Every question, whatever surface it arrives on, compiles to one logical plan language (the Query IR), gets policy woven into it exactly once, is routed to the tier of truth it semantically asked for, and is executed by engines that only compute. The write path appears at the top of the picture only because it manufactures what reads consume: every committed Act and admitted Assert flows down one ordered stream into the stores the router chooses between.

USERLAND: apps · agents · dashboards · BI
SDK · GraphQL · MCP · governed SQL · standing queries
═══════════════════════════ GOVERNED SEAM ═══════════════════════════
QUERY IR
┌──────────────────────────────┴──────────────────────────────┐
│ THE PIPELINE (§4.4) │
│ 1 BIND → 2 SECURE → 3 ROUTE → 4 OPTIMIZE → 5 EXECUTE → 6 ACCOUNT
└──────┬────────┬──────────┬──────────┬───────────┬───────────┘
│ │ │ │ │
┌───▼───┐┌───▼────┐┌────▼─────┐┌───▼────┐┌─────▼──────┐
│ HOT ││ OPERA- ││ WARM ││ COLD ││ DERIVED / │
│ LVC ││ TIONAL ││ windows/ ││ as-of, ││ SERVING │
│latest ││ P1/P23 ││ history ││ canon ││ search·ANN │
│values ││ current││ (P4) ││ (P3) ││ graph·tiles│
└───▲───┘└───▲────┘└────▲─────┘└───▲────┘└─────▲──────┘
│ │ │ │ │
└────────┴────── one ordered stream ───────┘
P8 materializer (fusion, projections, indexes)
WRITES: Acts (P1 + outbox) · Asserts (P4 gate)

Three facts orient everything else:

  1. One door. Every read enters as Query IR. That is the only place policy can be enforced once, and the reason no engine, index, or cache can become an authorization bypass.
  2. Tiers are semantics. “Latest speed” (hot), “speed history last week” (warm), “state as of March 3” (cold), and “state in my what-if branch” (analytical) are different questions, not different performance options. The router reads the question type off the plan; it never chooses a tier because it’s cheaper.
  3. One ordered stream. All tiers consume the same commit order. They can lag; they cannot diverge. Lag is measured by watermarks and is visible in the answer, never hidden.

2.2 Design principles

Numbered so later sections and the decision log can cite them.

  • PR-1 · The engines only compute. Meaning lives in the ontology (P0); trust lives in the registry; policy lives in the compiled plan. Any engine that accumulates one of those is a leak, and a bug.
  • PR-2 · One IR, policy injected once. All surfaces compile to the same Query IR; SECURE rewrites policy into the plan before any engine sees it. There is no second read path.
  • PR-3 · Routing is semantics first, cost second. The truth tier is read off the plan. Cost-based choice exists only inside a tier (anchor ordering, index choice) — never between tiers.
  • PR-4 · Policy compiles into the plan, or the plan is invalid. A policy predicate that no backend can evaluate and the kernel cannot retain post-retrieval makes the plan invalid — not slow, invalid.
  • PR-5 · A Query never leaves the owned planes. External systems feed the planes (through the Assert path); the planes answer queries. No read fans out to an external API.
  • PR-6 · Projectable is not filterable. A lazily-mirrored (on_read) trait can be read on entities you already found; it can never be used to find entities, because the landed evidence is an incomplete set. Promotion to streaming buys the filter.
  • PR-7 · One plan, one tier. A single query never silently joins two tiers (now-data with March-data). Callers who want that compose two queries in userland, visibly.
  • PR-8 · Barriers wait on watermarks; the overlay only accelerates. Freshness guarantees are expressed as waits on watermarks (§4.6). The recent-write overlay is machinery that makes those waits ~zero at Scale — it never becomes the semantic contract.
  • PR-9 · SERVING or invisible; promotion is an Act. A materialization, index, or package the planner may use is one in lifecycle state SERVING (§4.5). Everything else does not exist to the planner. Moving an artifact into service is a governed, audited Act.
  • PR-10 · Declared materializations, semantic invalidation — or no cache at all. Every precomputed answer is declared, versioned, and lineage-tracked, so invalidation is a graph walk, not a guess. Ad-hoc result caching is banned (§7.2).
  • PR-11 · Offline is the same pipeline with the first two stages precompiled. A device runs no policy engine; it holds policy’s output, signed. Local reads are ROUTE+EXECUTE over a synced base plus the device’s own pending writes.
  • PR-12 · Two planners, two altitudes, one door. The dataset world (SQL over datasets, for analysts and pipelines) keeps its own planner below the seam. Entity reads never route through it; it never defines what the seam can express.
  • PR-13 · Lag and staleness are queryable, never aspirational. STALE is a property an agent checks before acting; watermarks are numbers in the response, not a dashboard someone watches.

2.3 A read’s life at a glance

ctx.query( workOrders.assignedTo(me).near(here, 5km) ) ── SDK builder
[1] BIND parse → IR; resolve ontology/realm/branch; reject bad plans
[2] SECURE policy → plan: region predicate injected, columns masked
[3] ROUTE "current operational state" → operational tier (P23 tables)
[4] OPTIMIZE spatial index vs assignee index? registry stats say assignee
[5] EXECUTE index scan → ID set → resolve traits (ResolvedValues)
[6] ACCOUNT cost within interactive-UI budget; usage attributed
answer: entities + per-trait { value, settledness, freshness } + explain

Six stages, one of which (SECURE) carries the correctness burden, one of which (OPTIMIZE) carries the cleverness, and four of which are deliberately boring. Details in §4.4; four fully-worked traces in §8.


3. Requirements

3.1 The journeys the data plane must serve

These journeys (from the acceptance scenarios and the pilot program) are the demand side of this design. Each is a compressed statement of what it reads.

J-1 · NetworkAccess field operations (the pilot). A NOC operator watches a wallboard of alarms (standing query with diff-push); an ops agent traces a dead circuit upstream to a candidate cut (trace), checks the segment’s as-built location and finds it CONTESTED, so it refuses to dispatch blind (point read + settledness); a field engineer sees her assigned work orders near her (list + spatial), completes one, and it leaves her queue immediately (read-your-write on a list); a post-incident review reconstructs the network as of 02:14 and asks why the system believed the joint was where it said (as-of + explain). Volume: ~1M entities in the pilot cell, growing to 5–10M amplified.

J-2 · Trafficure corridor management. Speed feeds from three providers disagree; the map renders the corridor hatched (CONTESTED on tiles); latest speeds are milliseconds fresh (LVC); an engineer forks a what-if branch, an agent forks forty (branch reads), and the forty are ranked in one comparative read (multi-branch compare); the winning intervention renders as a diff-overlay on the live map (branch-diff tiles).

J-3 · SmartBuild construction. A months-lived plan branch is reviewed against reality as a diff (plan-branch read); trench progress is read per chainage (linear-referenced traits); claimed progress and photo-derived progress diverge and the span flips CONTESTED, opening a dispute (plan-vs-evidence standing query); crews work offline for days and their devices answer “my jobs near me” instantly from local state (offline reads).

J-4 · Agents everywhere. Every agent read is budget-capped (tightest workload class), policy-compiled like any read, and answers carry settledness + quality warnings the agent must check before acting. Retrieval for agents (search, similarity) is governed: policy restricts the universe before retrieval, and every hit resolves to a governed entity reference.

3.2 Performance gates

From the pilot program design; p99, measured against the pilot cell (~1M entities, 300k+ connectivity ports, amplified to 5–10M by M3).

ReadGate (p99)
Point read / read-your-write< 50 ms
List / worklist (validation queue)< 200 ms
Circuit / dependency trace< 500 ms
Map tile< 150 ms
Latest high-volume value (LVC)milliseconds
Others (search, aggregate, as-of, branch)no pilot gate; engineered per §5

Write-side context that shapes the read design: 500 Acts/s sustained per cell, 0.5–2k Asserts/s, 1M-row bulk migration under 4 hours.

3.3 Deployment profiles

The same design must run in four shapes. This is a hard requirement, not an aspiration — half the market is on-prem or air-gapped, and the pilot itself deploys from signed bundles on customer infrastructure.

ProfileShapeData-plane consequence
Leanone node, one team, moderate scaleOne Postgres serves seven read species (§6.1); DuckDB reads the lakehouse; LVC is in-process; no JVM anywhere in the read path
ScaleJio-scale, many cellsEngines split per workload class; per-species Scale adapters (§6.2); graduations are evidence-gated (§6.3)
Air-gapself-hosted, no internetIdentical shape to Scale with self-hosted adapters; baked tiles as offline base; offline-verifiable capabilities
Edge / Devicephone, field laptop§4.7 — SQLite + precompiled scope; the smallest honest deployment of the same pipeline

A customer graduates column-by-column (swap one adapter at a time); no plan shape, Surface, or SDK changes. The species table (§5) is the contract; the engine columns are engineering.


4. Architecture

4.1 Surfaces and the Query IR

Every read surface compiles to one closed, kernel-owned logical plan language (PR-2). The surfaces:

  • the typed SDK query builder (primary — generated per-realm from the ontology),
  • GraphQL, generated per-realm,
  • MCP tools for agents (the tool surface is a projection of the ontology — a hallucinated tool fails to exist, not at runtime),
  • governed SQL views — per-principal schemas of filtered entity views for BI tools; never raw tables, never a raw SQL door (raw SQL is an IR bypass),
  • standing-query registration (§4.8).

The IR node set (closed; growing it is a design event, LLD §10):

NodeQuestion it expresses
Select(entityType, predicate)find entities
`Project(traits, resolvedraw)`
`Traverse(linkfamily
`Trace(upstreamdownstream
`Spatial(withinnear
`Temporal(latestwindow
Aggregate(groupBy, measures)grouped measures
Similar(embedding, k)vector similarity
TextMatch(query)full-text search
Compare(branches[], plan, alignKeys)run one plan across N branches, aligned by key (LLD §24.5)
Subscribe(view)standing query over an IR-defined view

4.2 Truth tiers and serving stores

A read’s Temporal node names the kind of truth it wants; each kind has a home:

You asked forTierServed fromFreshness
Current state of an entityoperationalP1 (just-committed) / P23 (resolved world)ms behind commit
Latest value of a fast-changing traithotLVCmilliseconds
A window or history of valueswarmP4 time-series storeseconds
State as of a past timecoldP3 lakehouse snapshots (bitemporal)minutes (by design)
State in a branchanalyticalbranch overlay + base snapshotbranch-local
Text/similarity hitsderivedP5 / ANN indexesindex watermark
Map tilesservingP24 tile artifactspublish watermark

Why tiers can never disagree: every one of them consumes the same ordered stream of committed facts (§2.1). A tier can only be behind, and “how far behind” is its watermark — a number the response can carry (PR-13). This is also what makes read barriers (§4.6) implementable everywhere: every store knows exactly which token it has applied through.

The hot path in one line (LLD §9): driver Asserts → cheap authority check → one ordered statement stream → three consumers at three speeds — LVC (ms), warm store (s), lakehouse (min). Only traits declared hot earn LVC entries; fusion policies used in the LVC must be incrementally computable.

4.3 Source classes — where answers can possibly come from

Every trait’s evidence has exactly one of two origins, declared in its P0 manifest (locked decision DP-3):

  • Owned — truth born inside, written by Acts. Full tier routing, replayable, branchable.
  • Mirrored — external truth entering as Asserts under a source manifest. Lands in the time-series store, fuses into the World plane, and its staleness is visible (STALE/BLIND are queryable settledness states).

There is no third class. “Virtual” or “federated live” traits dissolve into a materialization policy on the mirror manifest:

PolicyMechanismTypical source
pushCDC / streaming driver emits continuouslyERP, OSS, sensor feeds
scheduledperiodic sweep emits diffsslow-changing registries
on_reada read finding the trait BLIND/STALE beyond its freshness class triggers a driver fetch; the fetched value enters through the ordinary Assert path; the read is answered from the planelong-tail attributes not worth streaming

The structural rule that keeps every invariant intact: a Query never leaves the owned planes (PR-5). on_read hydration is maintenance of the mirror, not an answer path. Consequences, all deliberate: policy still compiles into indexes we own; as-of stays honest (“we were BLIND then” is a true answer); branches never hydrate (a simulation never phones the outside world); and external non-determinism stays quarantined in the Assert path where it already lives.

Two promise deltas for on_read traits, and only two:

  1. Freshness — they spend more of their life STALE; a caller may attach a freshness barrier (§4.6) and pay for the wait.
  2. Filterability — PR-6. A filter over an on_read trait would scan only landed evidence: every value honest, the set silently incomplete. So on_read traits are projectable, not filterable, statically enforced at BIND. Filterability is what promotion to push/scheduled buys.

Operational guard: hydration is batched, rate-limited through the egress port, and capped per query — a 10k-entity projection must not become a thundering herd against a customer’s ERP.

What correctly remains outside the Query surface:

  • *Read-Call* — when a workflow needs the external system’s own answer at time T as a recorded, correlated fact. A hydrated Query returns plane truth; a read-Call returns a crossing. Two questions, two verbs.
  • Dataset federation — analysts querying external sources in place happens below the seam (§4.9), on datasets, not entities.

4.4 The query pipeline — six stages

Query IR → 1 BIND → 2 SECURE → 3 ROUTE → 4 OPTIMIZE → 5 EXECUTE → 6 ACCOUNT → answer

The complexity budget is deliberately lopsided: SECURE carries correctness, OPTIMIZE carries the only cost-based search, and the other four are simple by design.

Stage 1 · BIND — validate everything statically checkable, reject early

Parse the surface into IR. Resolve ontology version, realm, branch, and as-of time against P0. Validate every trait reference and the kind-compatibility of every predicate. Enforce the static rules: PR-6 (no filters on on_read traits) and barrier coherence (§4.6 — a barrier on an as-of, branch, or tile read is invalid, not slow). Stamp the plan-cache key (IR shape · ontology version · policy version).

Technology: Rust, against an in-memory ontology snapshot. Cost: microseconds. Design intent: everything rejectable without touching data is rejected here, so nothing downstream needs defensive checks.

Stage 2 · SECURE — compile policy into the plan, once

The correctness-critical stage; everything downstream is allowed to be dumb because this stage is not.

  • Coarse markings (realm / region / classification band) and row-level rules become indexable predicates on Select — cheap, applied to every plan, evaluated by ordinary indexes.
  • Trait-level denials become column masking on Project.
  • Project(raw) demands its elevated capability now, not at fetch time.
  • Fine-grained relationship checks (ReBAC) run only on the candidate set that survives the marking predicates — the two-tier model of LLD §13.2. List queries are never N per-row checks.

Two invariants:

  1. Residual-predicate discipline (PR-4): every injected predicate must have a proven evaluation site — pushed to a backend that can evaluate it, or retained kernel-side for post-retrieval filtering. A plan where a policy predicate has no site is invalid.
  2. SECURE precedes ROUTE and OPTIMIZE, so policy predicates participate in index selection rather than being bolted on after.

Because predicates are compiled into the plan, they apply identically to composed rows — a row patched by the recent-write overlay (§4.6) passes the same predicates as a base row. The overlay therefore adds no policy surface.

Stage 3 · ROUTE — deterministic tier and plane selection

A lookup, deliberately not clever (PR-3). Truth tier = the plan’s Temporal node × the trait’s declared freshness class (latest→LVC · window→warm · asOf→cold · point→operational). Species→plane per the §5 table. A cost model choosing between tiers would be a bug: the tier is what kind of truth you asked for.

A read barrier never changes routing; it gates execution of the routed subplan on that plane’s watermark (§4.6).

Output: subplans pinned to planes.

Stage 4 · OPTIMIZE — the one genuinely hard problem, kept small

Within the pinned backends only:

  • (a) Anchor selection. A multi-modal plan — “entities in this polygon, with CONTESTED speed, similar to E, two hops downstream of X” — can start at the spatial index, the settledness scan, the ANN index, or the traversal; the wrong anchor is orders of magnitude slower. The search space is 2–5 candidates, so this is System-R-lite: greedy (or tiny-DP) ordering over per-kind selectivity estimates from the registry (§4.5). No plan memo, no Cascades, no join-order explosion.
  • (b) Materialized-view rewrite. If the registry holds a SERVING materialization that subsumes a subplan and is fresh enough for the caller’s workload class, substitute it. This is the reason common questions are fast (§4.5). Artifacts in any other lifecycle state are invisible to the planner (PR-9).
  • (c) Pushdown and pruning. Predicates and projections pushed into each backend leaf; partition, H3-cell, and snapshot pruning for cold-tier leaves.

Graduation path if cross-store ordering ever outgrows greedy: an optd-style plan memo — evidence-gated like every adapter (§6.3).

Stage 5 · EXECUTE — mechanical by design

v0: backend leaves return entity-ID sets plus trait columns; the kernel composes them with ID-set algebra (intersect / union / semi-join) in Rust; Project resolves the survivors against the World store, composing the recent-write overlay where one exists (§4.6). The entity ID is the join currency — engines never join across each other.

Graduation (evidence-gated, target M3+): embed DataFusion as the federated executor — each plane a TableProvider leaf, Traverse/Trace as custom plan nodes producing ID-set streams, Arrow end-to-end. Calcite never enters this path — no JVM in the read hot path (the dataset world keeps its Calcite, below the seam, §4.9).

Stage 6 · ACCOUNT — governance wearing an optimizer’s numbers

Before execution: the estimated cost (OPTIMIZE’s selectivity numbers doing double duty) is checked against the caller’s workload-class budget. Over budget → reject with the estimate, offer async execution (a batch job), or degrade to approximate where the surface declared tolerance. During: row / time / hydration caps; barrier waits are charged and capped per workload class; branch reads bill against the branch’s class envelope (a sweep’s forty branches meter as one aggregate budget — LLD §23.1). After: actual cost attributed to tenant / actor / workload — the pricing meter and the abuse detector are the same instrument.

4.5 The registry — the planner’s control plane

Without a control plane that knows what exists, how fresh it is, how it derives, and what it costs, the router is blind. The registry is that control plane — and it is built as a system Realm: the kernel’s own reality, populated with kernel-owned entity types, governed by the same verbs as everything else.

Dataset · Projection · Index · Materialization · EmbeddingIndex ·
Metric · SourceManifest · FreshnessContract · QualityContract ·
EngineProfile · StatisticsSnapshot · LineageEdge · OfflinePackage

Registering an index is an Act. Lineage is a link traversal. “Which materializations are stale?” is an ordinary Query. The catalog UI is a Surface. There is no parallel metadata store and no second security model.

Three load-bearing roles:

  1. Access-path inventory, per entity type — OPTIMIZE’s input. Example for Tower: primary key → Postgres · geometry → PostGIS GiST · H3 → partition key · text → search index · embedding → pgvector (model@version) · adjacency → link index · latest → LVC · history → lakehouse. Anchor selection is a read over this inventory plus statistics.
  2. Statistics as a materializer projection — cardinalities, per-trait distinct counts and null rates, settledness distribution, H3-cell density, link-degree distribution (supernodes are quantified before anyone traverses them), partition layouts, stream rates. Declared, versioned, rebuildable — never a side-channel ANALYZE with its own freshness semantics. The materializer sees every statement anyway, so these stats are fresher and finer than any generic optimizer’s.
  3. Lineage drives semantic invalidation — the caching rule (§7.2) and re-materialization both walk the same LineageEdge graph. “Which caches does a work-order change touch?” is a Trace, not tribal knowledge.

The artifact lifecycle (decision DP-9). Every registry artifact — materialization, index, embedding index, tile layer, offline package — carries a lifecycle state as an ordinary trait:

DRAFT → BUILDING → VALIDATING → SERVING → DEPRECATED → ARCHIVED
failure states: FAILED · QUARANTINED

Three rules do all the work (no seventh pipeline stage):

  1. Only SERVING artifacts exist to the planner (PR-9). MV-rewrite, projections, and tile serving consult the state; everything else is never a candidate, so it never needs “blocking.”
  2. Promotion is an Act. Package-declared materializations auto-promote when their declared contracts pass — the attestation a build produces (schema, quality, freshness, lineage checks — §7.4) is the evidence. Human- and agent-authored artifacts require a steward’s promotion Act. Agents never self-certify: an agent’s draft artifact serves only inside its own branch/project scope, under its quota, until promoted.
  3. Names are aliases. Plans reference the registry name; the name resolves to the current SERVING version; promotion repoints atomically; rollback is a repoint. Physical artifact identifiers never appear in a plan — which makes silent metric drift structurally impossible: a definition change is a new version behind the same name, and the version served is recorded in the execution explain (§7.1).

Bootstrap ladder (resolved): a file-backed manifest (M1, pre-kernel — the planner reads a file) → the same content as plain Postgres tables → system-Realm entities at M2, absorbed via an ordinary bulk load. Each rung answers the planner identically; only the storage moves.

Materialization discipline (DP-10): platforms feel fast because common questions are already answered. The ladder — raw statements → fused world state → derived projections → metric tables → serving projections (search, tiles, LVC) — is the materializer’s job, and every rung is declared (a registry entry: definition as Query IR, target, freshness contract, owner), versioned, lineage-tracked, and monitored against its contract. Never an ad-hoc pipeline. Four execution modes: streaming (statement-driven), incremental batch, full rebuild (also the disaster-recovery story), and on-demand (first read materializes, registry caches). The operator declares target lag; the system chooses the mode — freshness contracts are the API, schedules are the adapter’s problem.

4.6 Consistency — barriers, the overlay, and read-your-write

The questions this section answers: when I write something, when do my reads see it? What may I rely on? What does it cost?

Baseline: eventual, with visible lag. All serving stores apply the same ordered stream (§4.2); a read with no freshness request is served from wherever ROUTE sends it, and the response’s data-explain carries the store’s watermark. Lag is a number, not a surprise (PR-13).

The read barrier — the freshness contract (decision DP-6). Any coherent read may carry:

barrier: { afterToken: OrderToken } -- "after my write"
| { freshness: Duration } -- "no staler than"
onTimeout: serve | fail -- default: serve

Semantics: the router holds the affected subplan until the routed store’s watermark satisfies the condition. Every serving structure has a watermark — the world store’s applied token, the search index’s, the adjacency index’s, the LVC’s — because all consume the same ordered stream. On timeout under serve (the default), the read executes anyway and the data-explain carries visible_through < requested: lag explicit, UI responsive. Under fail, the caller gets a typed error — for agent loops and workflow preconditions whose correctness needs the wait. Waits are charged by ACCOUNT and capped per workload class.

One primitive, two former problems: read-your-write (afterToken) and awaited freshness on lazily-mirrored traits (freshness) are the same mechanism. Barriers wait on watermarks (PR-8).

Coverage law. Where barriers apply, and how they are satisfied:

SpeciesBarrier?Satisfied by
Point / resolved entity / listyesoverlay composition (Scale) or watermark wait (Lean)
Search, traverse, traceyeswait on that index’s watermark (a post-splice Trace(afterToken) waits for the adjacency index to pass the token — “is my splice live?” gets an honest yes)
Aggregate / windowyeswait on the warm store’s watermark
As-of, branch reads, tilesno — rejected at BINDyou cannot be “after your write” in March; tiles are publish-time artifacts

The overlay never patches a text-match, a traversal frontier, or an aggregate: the moment the accelerator owns per-species correctness, it has become a second planner.

The recent-write overlay — machinery, not contract (decision DP-7). At Scale, the materializer lags seconds, and pure waiting would breach the list gate. The overlay keeps barrier waits at ~zero for entity-shaped reads. It is the Act-side analogue of the LVC: the latest committed trait patches keyed by entity, fed from the same outbox stream the materializer consumes, each entry retiring as the materializer’s watermark passes its token.

  • Global and policy-filtered, never session-scoped. Everyone composes the same overlay; SECURE’s compiled predicates evaluate on composed rows exactly as on base rows, so there is no two-users-two-truths anomaly and no new policy surface.
  • Lean ships without it. Materializer lag there is milliseconds; waits are already ~zero. The overlay graduates in on evidence: sustained barrier waits breaching the list gate at Scale (§6.3).
  • The overlay accelerates; it never defines truth. Truth is the log and the planes; the overlay is fresh visibility over them.

Read-your-write, end to end. An Act returns its OrderToken; the SDK carries it. A point read presenting it is served from the Process plane (P1), which committed it — guaranteed by construction. An entity-shaped or index read presenting it gets barrier semantics: the field engineer’s completed work order leaves her queue because her next list read carries her token, and either the overlay composes the patch (Scale) or the watermark passes in milliseconds (Lean). Reads with no token remain eventually consistent with visible lag — strong exactly where the loop needs it, receipts everywhere else.

Per-cell honesty. OrderTokens are {cell, seq}; a barrier binds the minting cell’s watermark only. A scatter-gather read across cells promises per-cell freshness, never a global cut — stated in the explain, not hidden. (Cell mechanics: the Cells doc.)

One tier per plan (PR-7, unchanged): a single query never joins hot-now against cold-March. The SDK may sugar a two-query composition; the seam sees two plans, and the incoherence — if wanted — is userland’s explicit, visible choice.

4.7 The offline / edge read path

Offline is a branch (LLD §22): a disconnected device is an implicit device branch — base = its last-synced replica, branch log = its queued writes — and reconnection is an ordinary branch merge. This section fixes what its reads are. One rule keeps offline part of the same system instead of a parallel product (PR-11):

Offline runs the same pipeline with BIND and SECURE precompiled.

SERVER (at sync-package build) DEVICE (at read time)
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ BIND + SECURE run here: │ signed │ ROUTE + EXECUTE only: │
│ working-set scope compiled │──package──▶│ SQLite base tables │
│ policy → manifest │ │ + device-branch overlay │
│ signed as a capability │ │ + FTS5 (text) + R-Tree (geo)│
│ (Biscuit: offline-verifiable)│ │ local IR subset (below) │
└──────────────────────────────┘ └──────────────────────────────┘
  • The device never evaluates policy; it holds policy’s output. The working-set manifest — entity types, traits, regions, materialized slices, permitted actions — is a compiled plan scope, signed as an offline-verifiable capability. Locally-authored policy does not exist.
  • Local reads compose base + overlay. Every local read composes the synced base with the device branch’s pending Acts/Asserts — §4.6’s composition rule applied locally. The engineer sees her own pending work instantly, flagged as pending.
  • The local IR subset is closed: Select · Project · Traverse(bounded) · Trace(working-set-scoped) · Spatial(bbox | near) · TextMatch · Temporal(latest). A Trace that reaches the edge of the synced working set reports its frontier honestly (frontier: truncated-at-working-set) rather than pretending completeness. No Aggregate beyond shipped materialized slices; no Similar (v0); no Compare; no as-of (the device holds one base). Offline analytics means shipped slices, declared like any materialization.
  • An offline package is a registry artifact (§4.5): built by a batch job, validated, promoted to SERVING, expiring. Expiry is enforced locally — past manifest expiry, reads degrade read-only, then lock (grace window declared per package); a server-side policy change invalidates the manifest at next contact.
  • Not owned here: the sync protocol, conflict adjudication, and the write-upload path (LLD §22, §15). This section owns only what a local read means.

The Edge profile’s data plane (§3.3) is this section.

4.8 The two side doors — tiles and standing queries

Two read paths that do not go through the per-request pipeline, and the rules that keep them from becoming unpoliced back doors.

Tiles are Serving-plane projections — derived, rebuildable, authoritative for nothing. The rule: policy compiles at publish time, not per request.

  • Baked layers (PMTiles pyramids) are batch outputs, partitioned by policy marking — a tile file exists per coarse partition, and a request selects a partition by capability. It never evaluates policy.
  • Dynamic layers (Martin) are generated functions whose SQL is produced by the same SECURE rewrite — one artifact per policy partition, republished on policy change (policy log → materializer → republish).
  • featureId == entityId in every tile: a click on any rendered feature is one governed point-read away from its truth. Live trait overlays (speeds, alarm states) ride the realtime gateway from the LVC — carrying ResolvedValues, so a CONTESTED value renders visibly differently.
  • Tile layers are registry artifacts; a re-bake promotes behind the layer alias (§4.5).

Standing queries are Subscribe over an IR-defined view: registered through the full pipeline once (BIND → SECURE → ROUTE — so a standing query is policy-compiled like any read), then evaluated incrementally by the materializer against the statement flow, delivered as diffs via the realtime gateway. Lean: re-evaluation on affected keys. Scale: warm-tier continuous views feeding the same contract.

The registration vocabulary is the closed pattern grammar of LLD §24.1 — event(type, predicate) · window(source, condition, duration) · sequence(p1 then p2 within t) · absence(p1 not-followed-by p2 within t), plus debounce / throttle / latch. The validation queue, the drift monitor, the alarm-flap suppressor, and the agent watchlist are all this one mechanism.

Plan-vs-evidence divergence is a composition, not a feature. A plan package declares its commitment traits and tolerances; registration compiles a standing Compare(plan-branch, canon) over exactly those traits; divergence beyond tolerance emits a statement that opens a WorkItem. Change-order adjudication, built from Compare + Subscribe + Aggregate with zero new machinery.

4.9 The dataset world — the altitude rule

There are two planners at two altitudes, permanently — and that is correct, not transitional (PR-12):

  • Above the seam: this document’s pipeline — entities, traits, planes, policy-compiled, branch-aware. Rust.
  • Below the seam: the dataset world — the existing NAP/DMP stack (Calcite planner, Trino federation, Spark/Sedona transforms, Cube semantics) serving datasets to analysts and pipelines. It keeps its SQL surface and its JVM; both are fine below the line.

The leak to refuse has a predictable shape: “the dataset stack already does X — expose it through the seam.” That is an IR bypass; the moment the dataset world’s SQL defines what the seam can express, the one-door model is dead. Traffic crosses the altitude line in exactly three governed ways:

  1. Mirrors up: dataset outputs enter as typed Asserts through driver manifests (the medallion staging → typed evidence path).
  2. Shared bytes: batch transforms read the same lakehouse the kernel writes — one copy of the data, two planners over it.
  3. Views down: governed SQL schemas project entity views down for BI tools — generated, filtered views, never raw tables.

The NAP engine fleet (PostGIS, ClickHouse, Trino, DuckDB, Iceberg/MinIO, Martin) is the adapter inventory both altitudes share, and its production bug ledger (~15% SQL-text correctness defects from regex-SQL surgery; a policy-blind result cache) is the standing empirical argument for compiling policy into plans (§4.4) and banning ad-hoc result caches (§7.2).


5. Read species catalog

The complete set of questions the data plane answers. This table is the contract; the engine columns are per-profile engineering (§6). Gates from §3.2 where they exist.

#SpeciesIR shapeTierLean servesScale servesGate
1Point read / read-your-writeSelect(ref) · ProjectoperationalPostgres (P1/P23)FDB Record Layer<50ms
2Resolved entity (entity-360)Select · Traverse(*, in, 1) · ProjectoperationalPostgres + inverse link indexFDB + inverse index<50ms
3List / filter (queues, worklists)Select(type, predicate)operationalPostgresFDB secondary idx + Quickwit<200ms
4Full-text / faceted searchSelect · TextMatchderivedPostgres tsvector/pg_trgmQuickwit
5SimilaritySimilar(embedding, k)derivedpgvectorpgvector / dedicated ANN (gated — §10)
6Aggregate / metricAggregate(groupBy, measures)warmPostgres materialized viewsClickHouse
7Traverse (bounded hops)Traverse(link, dir, depth)operationalPostgres recursive CTE (spike S2)FDB adjacency keyspace
8Trace (circuit / dependency)`Trace(updownpath)`operationalCTE / closure table (S2 decides)
9Spatial (within/near/intersects)Spatial(op, geom)operationalPostGIS (GiST + H3)H3/S2 cells in FDB + PostGIS read-model
10Latest high-volume valueSelect · Temporal(latest)hotLVC in-processLVC (FDB versionstamped / Redis Cluster)ms
11Window / historyTemporal(window)warmTimescaleDB (or ClickHouse — gated)ClickHouse
12As-of / canonicalTemporal(asOf)coldIceberg via DuckDBIceberg via DuckDB/Trino
13Branch read / diffTemporal(branch) · Compareanalyticalsnapshot clone + branch overlay + DuckDBsame
14Multi-branch compareCompare(branches[], plan, keys)analyticalDuckDB over clones + overlayssame
15Raw evidenceProject(raw) — elevated capabilityevidence storesTimescale + log archiveClickHouse + lakehouse archive
16Standing querySubscribe(view)streamingmaterializer re-eval → pushwarm continuous views → push
17Offline / local readprecompiled subset (§4.7)device branchSQLite base + overlay (FTS5, R-Tree)samelocal-instant
18Tilenot per-request IR (§4.8)servingMartin (dynamic) + PMTiles (baked)same, policy-partitioned<150ms

Notes the table needs:

  • Barriers (§4.6) may ride any species where coherent; BIND rejects them on #12, #13, #14, #18.
  • #13 covers plan branches too. A months-lived planning branch is a logical branch — base pointer + act log, no snapshot clone — so its reads compose an overlay exactly like device branches; periodic rebase is what surfaces its staleness; its budget comes from its branch class.
  • #14 is fan-out plus alignment, nothing more. Compare returns a table keyed by branch; ranking and statistics are ordinary Aggregate over it — judgment stays in userland.
  • Structure of the table itself: the operational tier carries half the species (which is why Lean’s answer is one Postgres); graph and spatial species are index-shape questions, not engine questions (adjacency and cells are keyspace layouts); and only four species ever touch the lakehouse interactively (#12–14, #15-archive) — everything else reads projections.

6. Engine mapping and scaling

6.1 What Lean collapses to

One Postgres instance is the operational store, the world store, the time-series store, the search index, and the tile source; the LVC is in-process memory; DuckDB reads Iceberg for cold reads. One engine serves species 1–11 and 18. No JVM exists in the read path. This is not a compromise configuration — it is the honest first build, and the pilot ships on it.

6.2 The Scale columns

Each species graduates independently to its Scale adapter (the §5 table’s sixth column): the operational store to FoundationDB, search to Quickwit, warm history to ClickHouse, the LVC to a shared tier, spatial to cell-keyed layouts, traces to precomputed reachability projections. ACCOUNT’s workload classes map to separate engine pools, so a dashboard stampede cannot starve field operations.

6.3 Graduation triggers — evidence, never aspiration

GraduationTrigger
Recent-write overlay arrivessustained barrier waits breaching the list gate (<200ms p99) at Scale materializer lag
P1 Postgres → FDBproven write contention / sharding need, not predicted
Search Postgres → Quickwitindex size / query latency evidence
Warm tier Timescale → ClickHouseassert feed volume (pilot: only if the inbound-feed stretch lands)
EXECUTE ID-set algebra → DataFusionresidual relational work volume; aggregate-pushdown limits (ADR due at M3 with Query IR v0)
Greedy anchor ordering → plan memocross-store ordering complexity observed in real plans
pgvector → dedicated ANNa real embedding corpus with recall/latency targets

6.4 Air-gap and Device

Air-gap is the Scale shape with self-hosted adapters, PMTiles-heavy serving, and offline-verifiable capabilities — nothing in the read path assumes an internet. Device is §4.7. The profile ladder is monotone: nothing a smaller profile promises is withdrawn by a larger one.


7. Cross-cutting concerns

7.1 Explain — three levels, two from the kernel

Every response carries provenance; explain expands it.

  • Execution explain (kernel): which tier, snapshot, ontology + policy version, engines, indexes, and materialization versions (aliases resolve to versions; the version served is recorded) answered each plan leaf. The audit answer to “why does it say 42.”
  • Data explain (kernel): per-trait derivedFrom (the evidence chain), settledness, freshness, quality warnings (§7.3), the watermark pair materialized_through / visible_through that makes lag a number, and the barrier verdict when one was requested (satisfied | timedOut, served).
  • Business explain (userland): why the answer matters — composed by Surfaces and agents from the first two. The kernel supplies evidence, never meaning.

7.2 Caching — semantic or nothing

CacheVerdictWhy
Plan cacheyeskeyed (IR shape · ontology v · policy v); invalidated by schema/policy log events; pure win
LVCyesis a tier, not a bolt-on
Recent-write overlayyes (Scale)a freshness layer, not a cache — entries retire on watermark, never invalidated by guess (§4.6)
Materializationsyesthe governed form of result caching: declared, lineage-invalidated, lifecycle-gated (§4.5)
Tile cacheyespolicy-partitioned by construction (§4.8)
Ad-hoc result cacheno (v0)a result is capability-scoped data; cache keys that omit the policy context are leak vectors, and keys that include it barely hit. The predecessor stack’s policy-blind Redis query cache is the cautionary precedent

Invalidation is semantic everywhere (PR-10): a statement lands → the materializer knows which materializations, LVC entries, standing queries, and tiles derive from it, because lineage is queryable.

7.3 Metrics and quality

Metrics. Dashboards must not each define “SLA breach” differently. A Metric is a P0-declared derived measure — formula (an Aggregate plan), grain, dimensions, owner, freshness contract — projected into the SDK as a typed read (metrics.towerHealth(region)), materialized per §4.5, versioned behind its alias. The kernel knows a metric’s shape; userland owns the formula.

Quality. Quality contracts ride source manifests: schema, freshness, null/range thresholds, spatial validity, duplicate policy. Violations become quality statements — evidence about evidence — surfacing in the same channel as settledness: a ResolvedValue can carry warnings: [source stale 42m · confidence low · schema drift], visible in the data explain. An agent checks the warnings before acting; that check is what makes autonomous action honest. There is no separate data-quality subsystem, because quality is a property of truth and lives where truth lives.

7.4 Attestations — how builds earn SERVING

A materialization job’s output is an attestation, not just a table: row counts, input versions and watermarks, quality/policy/lineage check results. That attestation is the evidence the lifecycle promotes on (§4.5): build → validate → promote. A failed check lands the version in FAILED/QUARANTINED and the alias never moves — the previous SERVING version keeps answering.


8. Worked examples

Four reads traced end to end. All run against the NetworkAccess pilot realm.

8.1 Read-your-write on a worklist (Lean)

Kavya completes work order WO-4711 on her phone, online. Her queue must not show it again.

  1. The completeWorkOrder Act commits; P1 returns OrderToken {cell: c1, seq: 88412}. The SDK stores it in her session.
  2. Her app refreshes the worklist: Select(WorkOrder, assignee = kavya ∧ status ∈ {open, in_progress}) with barrier: {afterToken: 0x…88412}, onTimeout: serve.
  3. BIND: valid plan; barrier coherent for a list read.
  4. SECURE: her region marking becomes an indexable predicate; no masked columns for her role.
  5. ROUTE: operational tier — the world store’s list tables.
  6. OPTIMIZE: registry stats say the assignee index is far more selective than the region predicate; anchor = assignee index.
  7. EXECUTE: the world store’s watermark is already past 88412 (Lean materializer lag is ~ms), so the barrier is satisfied without waiting; index scan → ID set → resolve traits.
  8. ACCOUNT: interactive-UI budget; wait time ≈ 0 charged.
  9. Answer: the list without WO-4711. Data explain: visible_through: 88415, barrier: satisfied.

At Scale, step 7 differs: if the world store’s watermark were behind, the recent-write overlay would compose WO-4711’s status patch (removing it from the result) instead of waiting.

8.2 A trace with policy, after a write (Scale)

The ops agent asks: “everything downstream of port P-2291, restricted to my capability’s region,” right after a splice Act reconnected it.

  1. Plan: Select(Port, id = P-2291) · Trace(downstream, path: fiberPath) · Project(status, geometry) with barrier: {afterToken: splice-token}, onTimeout: fail — an agent loop needs the post-splice topology, or it must not act.
  2. SECURE: region scope predicate injected; the trace is bounded to entities carrying the agent’s marking.
  3. ROUTE: operational tier; the trace runs on the derived adjacency index (declared hot path → maintained strand-to-strand keys).
  4. Barrier: the adjacency index’s watermark is 2 s behind the splice token. The router waits (coverage law: traces never compose the overlay). Watermark passes at +1.4 s — within the agent’s wait cap.
  5. EXECUTE: one range scan per logical hop; ID set → resolved traits. Degree counters (registry stats) had already confirmed no supernode on the path.
  6. Answer: the downstream set including the newly-spliced segment; explain records the wait. Had it timed out: a typed failure, and the agent retries or escalates — it never acts on a stale trace.

8.3 An as-of audit read (any profile)

Post-incident review: “the network around joint J-88 as of 02:14 last Tuesday — and why did the system believe the joint was at that location?”

  1. Plan: Select(area around J-88) · Temporal(asOf: 2026-06-23T02:14) · Project(resolved), then explain(J-88.location).
  2. BIND would reject any barrier here (as-of reads take none).
  3. ROUTE: cold tier — lakehouse snapshots, read through DuckDB; bitemporal, so the answer reflects what was believed at 02:14, not what later evidence revised.
  4. explain returns the evidence chain: the original survey statement, the as-built assertion that contested it, the fusion policy and trust weights as of that time (fusion configuration is itself bitemporal — LLD §23.2), and the settledness the value carried.
  5. Audit is a query, not a subsystem.

8.4 An offline read (Device)

Kavya, underground, no signal: “my assigned work orders within 5 km,” having just marked one complete locally.

  1. Her device holds a synced working set (signed manifest: her region, her entity types, 14-day expiry) and a device branch containing her pending completion Act.
  2. Local plan: Select(WorkOrder, assignee = me) · Spatial(near(gps, 5km)) · Temporal(latest) — inside the precompiled scope, so no policy evaluation happens on the device; the manifest is the compiled SECURE output.
  3. Local ROUTE + EXECUTE: SQLite base tables + R-Tree spatial index; the device-branch overlay composes her pending completion — that work order shows status: complete (pending sync).
  4. A local Trace from one work order’s cable stops at the working-set edge and reports frontier: truncated-at-working-set — honest, not silently complete.
  5. On reconnect, the device branch merges (write side, LLD §22); her local reads never claimed to be more than her base + her intent.

9. Decision log

Format: context → decision → rejected alternatives → status. PR-n references are to §2.2.

IDDecisionRejectedStatus
DP-1One Query IR, many surfaces, many engines (PR-2). Every read surface compiles to one closed logical plan; policy is enforced there, once.Per-surface planners; raw SQL as a first-class surface (an IR bypass); Calcite as the kernel’s planner (JVM in the hot path — it stays below the seam, §4.9)Locked
DP-2Semantics-first routing (PR-3). Truth tier = Temporal node × freshness class; cost optimization only within a tier.Cost-based tier selection (“the cache is faster than the log”)Locked
DP-3Two source classes — owned and mirrored; “virtual” dissolves into push / scheduled / on_read materialization policies; on_read traits are projectable, never filterable (PR-5, PR-6).A federated “virtual” class answering reads live from external systems (breaks policy compilation, as-of honesty, and branch isolation)Locked
DP-4Policy compiles into the plan (PR-4): markings → indexable predicates, denials → column masks, before routing; residual predicates must have proven evaluation sites; two-tier authz (markings then ReBAC on survivors).Per-store policy features; post-hoc result filtering as primary enforcement; per-row authorization checks on listsLocked
DP-5EXECUTE v0 = ID-set algebra in Rust; entity ID is the join currency; engines never join across each other. DataFusion embed is the named graduation.Distributed joins across engines; a federated executor on day oneLocked (graduation gated, §6.3)
DP-6Read barriers — `{afterTokenfreshness}withonTimeout: servefail(default serve-flagged), waits charged per workload class; satisfied against per-store watermarks; per-cell promises only (PR-8). Supersedes the earlier point-read-only token echo; absorbs the formerawait_freshness` fork.
DP-7Recent-write overlay = Scale accelerator behind the barrier contract — an Act-side LVC: global, policy-filtered, entity-shaped reads only, retiring on watermark. Lean ships without it.The overlay as the read contract (always-on composition — a second correctness path in every profile); session-scoped overlays (two-users-two-truths); patching search/trace/aggregate results from the overlayLocked (arrival gated, §6.3)
DP-8The registry is a system Realm (PR-1): catalog, statistics, lineage as kernel-owned entities governed by the ordinary verbs; bootstrap ladder file → tables → realm at M2.A parallel metadata store with its own security model (the “second catalog” anti-pattern)Locked
DP-9Artifact lifecycle: DRAFT…SERVING…ARCHIVED as registry state; only SERVING is visible to the planner; promotion is an Act (auto for package-declared artifacts passing contracts; steward Act otherwise; agents never self-certify); names are aliases repointed atomically (PR-9).A “queryability gate” as a pipeline stage; the L0–L5 certification ladder (its levels dissolve into existing boundaries: staging is outside the governed universe; “governed” is automatic per DP-4; “audited” is the evidentiary admission class); physical artifact names in plansLocked
DP-10Materialization discipline (PR-10): every precomputed answer declared, versioned, lineage-tracked, contract-monitored; builds produce attestations; target-lag declared, mode chosen by the system.Ad-hoc pipelines; schedule-first configurationLocked
DP-11No ad-hoc result cache (v0). Plan cache, LVC, overlay, materializations, tile cache — all semantic; nothing else.Policy-blind result caching (the predecessor’s incident ledger is the evidence)Locked
DP-12Metrics are P0-declared measures behind aliases; the kernel knows shape, never meaning.Per-dashboard metric definitions; a separate semantic-layer product above the seamLocked
DP-13Quality is evidence about evidence: violations are statements carried as warnings on ResolvedValues, in the same channel as settledness.A parallel data-quality dashboard subsystemLocked
DP-14Tiles compile policy at publish time: baked layers policy-partitioned, dynamic layers generated by the SECURE rewrite, featureId == entityId, layers as registry artifacts.Per-request policy evaluation on tiles; unpartitioned baked pyramids (an authorization bypass)Locked
DP-15Standing queries register through the full pipeline once, evaluate incrementally, deliver diffs; vocabulary = the closed LLD §24.1 grammar; plan-vs-evidence divergence = declared standing Compare composition.A general CEP engine in the kernel; a bespoke divergence-detection subsystemLocked
DP-16Two planners, two altitudes (PR-12): the dataset world keeps Calcite/Trino/Spark below the seam; three governed crossings (mirrors up, shared lakehouse bytes, governed views down).Exposing dataset-world capabilities through the seam; freezing analysts out entirelyLocked
DP-17One tier per plan (PR-7). Mixed-tier reads are two userland queries, visibly.Silent cross-tier joins by the plannerLocked (revisit gated on a named journey)
DP-18Offline = same pipeline, BIND+SECURE precompiled (PR-11): signed manifest as compiled scope, closed local IR subset, honest frontiers, packages as expiring registry artifacts.A local policy engine; CRDT-everything and last-write-wins sync (the branch-merge model stands); arbitrary local analyticsLocked
DP-19Barriers, watermarks, and freshness promises are per-cell; scatter-gather states per-cell freshness in the explain.Pretending a global consistent cut across cellsLocked (cell mechanics in the Cells doc)
DP-20QuerySpec is the wire surface; the QueryIR (LLD taxonomy: IR-2) never serializes publicly — BIND is the QuerySpec→IR compiler; the where condition tree (all/any/not + closed leaf set incl. settledness/geo/text/similar leaves) is the shared grammar organ. Detail: Query Surface, ADR-0028.Serialized IR on the wire (freezes the QueryIR early); GraphQL as the core API (resolver model = post-hoc policy; open composition vs the closed grammar — stays a generated projection per §4.1); dual endpointsLocked (2026-07-03)
DP-21Four typed doorways, one pipeline/query · /aggregate · /series · /subscribe, split by answer kind (PR-7 at the wire); all enter the same six stages; read_series and the trace syscall absorb behind the seam, leaving zero reads outside it.A single kitchen-sink endpoint discriminated by body kind; per-doorway pipelinesLocked (2026-07-03)
DP-22Streaming delivery matrix over one O(row) execution — JSON page (keyset cursor, never OFFSET) · NDJSON with in-band terminal summary (a stream without its summary is known-incomplete) · Arrow IPC with ontology-derived schemas · export-as-job to blob with attestation; ACCOUNT meters during via a stream combinator.Buffering results kernel-side; OFFSET pagination; completion status only at the HTTP layerLocked (2026-07-03)
DP-23Subscribe v0 = snapshot + ordered view-membership diffs (enter/update/leave, OrderToken-stamped, resume ring buffer + resync, heartbeats carry visibleThrough) over POST-SSE at Lean; WebSocket is the P26 graduation; Lean evaluation = re-eval on affected keys (§4.8 unchanged).Raw entity CDC for clients to re-filter (client-side filtering, banned); WebSocket-first at Lean; full incremental view maintenance before evidenceLocked (2026-07-03)
DP-24The strict mirror is a machine check: one fixture realm, language-neutral golden (spec, result) pairs covering executable blocks and BIND rejections, run by both the kernel CI (seeded Postgres) and the TS FixtureKernelClient CI.Doctrine-only compatibility; per-implementer test suites that driftLocked (2026-07-03)

9.1 External material adjudicated

The “Query Fabric” draft and its parent conversation (preserved under docs/_archive/adjudicated-inputs/) were adjudicated into this design in two passes (2026-07-02). Everything they proposed appears above either as an adopted-and-recast decision (the registry/catalog gap → DP-8; materialization discipline → DP-10; the write-visibility overlay → DP-7; the queryability gate → DP-9; offline package safety and local-store mechanics → DP-18; metric/quality/explain layers → DP-12/13 and §7.1; hybrid-retrieval governance → the Similar/search execution contract, §3.1 J-4), as something the kernel already had under another name (its “Semantic IR” = the Query IR; “actions vs assertions” = the Act/Assert verbs, which additionally carry fusion, settledness, and bitemporality; “attribute derivation modes” = source classes + fusion; its “Result Composer” = EXECUTE + SECURE’s masking, with redaction moved into the plan), or as an explicitly rejected alternative in the table above (Calcite in the kernel path, graph databases as masters, the L0–L5 ladder, safety-mode vocabulary — those are workload classes plus barrier options — session overlays, ad-hoc caches, CRDT sync).


10. Open questions

Each with the event that gates its resolution. Deciding earlier would be guessing.

#QuestionGate
1EXECUTE graduation — what evidence volume of residual relational work triggers the DataFusion embed; Substrait’s role in DuckDB hand-offADR at M3, with Query IR v0 running
2Overlay arrival — design fixed (DP-7); build when sustained barrier waits breach the list gate at ScaleScale-profile materializer lag observed
3Trace serving at pilot — recursive CTE vs closure table vs pgroutingspike S2 on real connectivity data
4Warm-store engine at pilot — TimescaleDB vs ClickHouseinbound-feed stretch goal lands (feed volume decides)
5Vector at Scale — pgvector-partitioned vs dedicated ANN; on-device Similara real embedding corpus with recall/latency targets; a field journey needing local similarity
6Mixed-tier reads — does any real journey need one plan across tiers?a named journey; the answer must produce snapshot semantics, not a convenience
7**on_read hydration under freshness barriers** — cap policy, partial-hydration answers, herd behaviorthe first real on_read source (IT-ops is the natural candidate)
8Divergence-tolerance vocabulary — what forms the plan-commitment declaration supports (absolute, %, temporal slack)the first plan package (SmartBuild)
9Governed SQL view depth — how much IR the per-principal schemas expose (Traverse in SQL is where it gets contested)BI-tool demand from a real deployment

The one-line summary for the hallway: every read walks through one door, gets policy sewn into it, is routed to the kind of truth it asked for, and comes back with its evidence, its freshness, and its bill.