Data Plane
Vinxi Kernel — Data Plane: Architecture & Design
Status: Full design, 2026-07-02. Open questions in §10.
Owner: Kernel team.
Supersedes: the 2026-07-02 morning draft of this document (dense ledger form — see git history) and the external “Query Fabric” material (adjudicated; preserved under docs/_archive/adjudicated-inputs/).
1. Introduction
1.1 Purpose and audience
This document describes the data plane of the Vinxi Kernel: everything that happens to a read between an SDK call and its answer — what kinds of questions the system answers, what data those answers are made of, how a query is planned and executed, how fresh an answer is guaranteed to be, and how the same design scales from a single Postgres node to a Jio-scale deployment to a phone in a trench.
Primary audience: engineers building or extending the kernel, including someone joining the team with no prior context. Secondary audience: technical reviewers evaluating the design. The document is self-contained: every kernel term it uses is defined in §1.4, and each section states its “why,” not just its “what.” Decisions are collected in an explicit decision log (§9) rather than woven into prose.
1.2 Scope
In scope: the read side — read species, the query pipeline, truth tiers and serving engines, the registry (the read planner’s control plane), consistency and read-your-write, the offline read path, tiles and standing queries, caching, and how everything scales up and down.
Out of scope, covered elsewhere: the write path (LLD §6, §9 — it appears here only where it manufactures read structures or defines read visibility), branch merge semantics (LLD §15), the sync protocol and conflict adjudication for offline devices (LLD §22), authorization internals (LLD §13.2), and the ontology/type system (LLD §19–20).
1.3 Related documents
| Document | What it fixes |
|---|---|
the-seam.md | The six verbs and the governed seam — the system’s physics |
lld.md | Port-by-port contracts (P0–P26), planes, type system, branches |
ports-and-adapters.md | Deployment profiles and the adapter matrix |
docs/superpowers/specs/2026-07-02-networkaccess-pilot-program-design.md | Pilot scope, entity census, performance gates |
docs/superpowers/plans/2026-07-02-m1-walking-skeleton.md | The first build increment (M1) |
docs/_archive/adjudicated-inputs/query-fabric-external-draft-2026-07.md, .../lepton-data-fabric.md | External material this design adjudicated (§9.4) |
1.4 Glossary
Read this once; every later section assumes it.
| Term | Meaning |
|---|---|
| Realm | A bounded slice of reality (one customer’s world, one city). Every read is scoped to a realm. |
| Branch | A fork of a realm’s reality for what-if simulation, planning, or offline work. main is canon. |
| Entity / Trait | The ontology’s nouns. An entity is a typed thing (Tower, WorkOrder); a trait is a named, typed property or link on it (tower.height, cable.splicedTo). |
| Verbs | The six operations the governed seam admits: Query (read), Act (a governed decision that changes state), Assert (a declaration of observed evidence), Call (a crossing to an external system), Subscribe, Schedule. |
| Statement | One piece of evidence: (subject entity, trait, value, valid-time, provenance, confidence). Asserts carry them; the system stores them, never overwrites them. |
| Fusion | The declared policy that combines multiple sources’ statements about one trait into a single resolved answer (e.g. trust-weighted mean). |
| ResolvedValue | What a read returns for a trait: value + uncertainty + settledness + the statements it derives from. The World plane never stores a naked scalar. |
| Settledness | Per-trait truth status: SETTLED (sources agree), CONTESTED (they disagree), STALE (evidence too old for the trait’s freshness class), BLIND (no evidence). Queryable, never hidden. |
| Planes | The five layers of the kernel: Process (transactional state, port P1), Truth (append-only logs, P2), World (resolved current interpretation, P23), Serving (indexes/caches — rebuildable, authoritative for nothing), Materialization (P8 stream / P7 batch — builds the others). |
| Ports (P-numbers) | The kernel’s stable interfaces. Relevant here: P0 ontology registry · P1 state store · P2 truth logs · P3 lakehouse · P4 time-series store · P5 search · P7 batch transform · P8 stream materializer · P12–P15 authorization stack · P16/P17 external boundary/drivers · P23 world store · P24 tiles · P26 realtime gateway. Adapters (Postgres, FDB, ClickHouse…) implement ports per profile. |
| Truth tier | Which copy of truth answers a read: operational (P1/P23 current state), hot (LVC latest values), warm (windows/recent history), cold (as-of/canonical lakehouse), analytical (branch reads), derived (search/vector indexes), serving (tiles). |
| LVC | Last-Value Cache — the hot tier: the latest value per (entity, trait) for high-volume streams, milliseconds fresh. |
| OrderToken | {cell, seq} minted when an Act commits. The total order of decisions within a cell; every downstream store applies the same order. |
| Watermark | How far a downstream store has applied that order: “applied through token T.” Every serving structure has one, because all consume the same ordered stream. |
| Cell | A shard/scale unit. Ordering and freshness promises are per-cell (see the Cells doc for placement). |
| Capability | A signed, scoped token proving policy was asked (P14). Every request carries one. |
| Workload class | The budget class of a caller: interactive UI · dashboard · agent-exploratory (tightest) · standing query · materialization job · export. |
| Freshness class | Declared per trait in P0: hot(ms) / warm(s) / cold(min). Determines which tier serves “latest” and when a trait is marked STALE. |
| Query IR | The single logical plan language every read compiles to, regardless of surface. Node set in §4.1. |
| Profiles | Deployment shapes: Lean (one node, small team), Scale (Jio-scale), Air-gap (self-hosted offline site), Edge/Device (phone or field laptop). Same contracts, different adapters. |
| Working set / sync manifest | The compiled, signed scope of data, traits, and actions a device may hold and use offline. |
| Registry | The data plane’s control plane: the catalog of what serving artifacts exist, their statistics, lineage, freshness, and lifecycle state (§4.5). |
| Seam | The single governed doorway between userland (apps, agents, dashboards) and the kernel. Nothing reads a store directly. |
2. Overview
2.1 What the data plane is
Userland — apps, agents, dashboards, BI tools — asks questions through the seam. Every question, whatever surface it arrives on, compiles to one logical plan language (the Query IR), gets policy woven into it exactly once, is routed to the tier of truth it semantically asked for, and is executed by engines that only compute. The write path appears at the top of the picture only because it manufactures what reads consume: every committed Act and admitted Assert flows down one ordered stream into the stores the router chooses between.
USERLAND: apps · agents · dashboards · BI │ SDK · GraphQL · MCP · governed SQL · standing queries │ ═══════════════════════════ GOVERNED SEAM ═══════════════════════════ │ QUERY IR │ ┌──────────────────────────────┴──────────────────────────────┐ │ THE PIPELINE (§4.4) │ │ 1 BIND → 2 SECURE → 3 ROUTE → 4 OPTIMIZE → 5 EXECUTE → 6 ACCOUNT └──────┬────────┬──────────┬──────────┬───────────┬───────────┘ │ │ │ │ │ ┌───▼───┐┌───▼────┐┌────▼─────┐┌───▼────┐┌─────▼──────┐ │ HOT ││ OPERA- ││ WARM ││ COLD ││ DERIVED / │ │ LVC ││ TIONAL ││ windows/ ││ as-of, ││ SERVING │ │latest ││ P1/P23 ││ history ││ canon ││ search·ANN │ │values ││ current││ (P4) ││ (P3) ││ graph·tiles│ └───▲───┘└───▲────┘└────▲─────┘└───▲────┘└─────▲──────┘ │ │ │ │ │ └────────┴────── one ordered stream ───────┘ ▲ P8 materializer (fusion, projections, indexes) ▲ WRITES: Acts (P1 + outbox) · Asserts (P4 gate)Three facts orient everything else:
- One door. Every read enters as Query IR. That is the only place policy can be enforced once, and the reason no engine, index, or cache can become an authorization bypass.
- Tiers are semantics. “Latest speed” (hot), “speed history last week” (warm), “state as of March 3” (cold), and “state in my what-if branch” (analytical) are different questions, not different performance options. The router reads the question type off the plan; it never chooses a tier because it’s cheaper.
- One ordered stream. All tiers consume the same commit order. They can lag; they cannot diverge. Lag is measured by watermarks and is visible in the answer, never hidden.
2.2 Design principles
Numbered so later sections and the decision log can cite them.
- PR-1 · The engines only compute. Meaning lives in the ontology (P0); trust lives in the registry; policy lives in the compiled plan. Any engine that accumulates one of those is a leak, and a bug.
- PR-2 · One IR, policy injected once. All surfaces compile to the same Query IR; SECURE rewrites policy into the plan before any engine sees it. There is no second read path.
- PR-3 · Routing is semantics first, cost second. The truth tier is read off the plan. Cost-based choice exists only inside a tier (anchor ordering, index choice) — never between tiers.
- PR-4 · Policy compiles into the plan, or the plan is invalid. A policy predicate that no backend can evaluate and the kernel cannot retain post-retrieval makes the plan invalid — not slow, invalid.
- PR-5 · A Query never leaves the owned planes. External systems feed the planes (through the Assert path); the planes answer queries. No read fans out to an external API.
- PR-6 · Projectable is not filterable. A lazily-mirrored (
on_read) trait can be read on entities you already found; it can never be used to find entities, because the landed evidence is an incomplete set. Promotion to streaming buys the filter. - PR-7 · One plan, one tier. A single query never silently joins two tiers (now-data with March-data). Callers who want that compose two queries in userland, visibly.
- PR-8 · Barriers wait on watermarks; the overlay only accelerates. Freshness guarantees are expressed as waits on watermarks (§4.6). The recent-write overlay is machinery that makes those waits ~zero at Scale — it never becomes the semantic contract.
- PR-9 · SERVING or invisible; promotion is an Act. A materialization, index, or package the planner may use is one in lifecycle state
SERVING(§4.5). Everything else does not exist to the planner. Moving an artifact into service is a governed, audited Act. - PR-10 · Declared materializations, semantic invalidation — or no cache at all. Every precomputed answer is declared, versioned, and lineage-tracked, so invalidation is a graph walk, not a guess. Ad-hoc result caching is banned (§7.2).
- PR-11 · Offline is the same pipeline with the first two stages precompiled. A device runs no policy engine; it holds policy’s output, signed. Local reads are ROUTE+EXECUTE over a synced base plus the device’s own pending writes.
- PR-12 · Two planners, two altitudes, one door. The dataset world (SQL over datasets, for analysts and pipelines) keeps its own planner below the seam. Entity reads never route through it; it never defines what the seam can express.
- PR-13 · Lag and staleness are queryable, never aspirational.
STALEis a property an agent checks before acting; watermarks are numbers in the response, not a dashboard someone watches.
2.3 A read’s life at a glance
ctx.query( workOrders.assignedTo(me).near(here, 5km) ) ── SDK builder │ [1] BIND parse → IR; resolve ontology/realm/branch; reject bad plans [2] SECURE policy → plan: region predicate injected, columns masked [3] ROUTE "current operational state" → operational tier (P23 tables) [4] OPTIMIZE spatial index vs assignee index? registry stats say assignee [5] EXECUTE index scan → ID set → resolve traits (ResolvedValues) [6] ACCOUNT cost within interactive-UI budget; usage attributed │ answer: entities + per-trait { value, settledness, freshness } + explainSix stages, one of which (SECURE) carries the correctness burden, one of which (OPTIMIZE) carries the cleverness, and four of which are deliberately boring. Details in §4.4; four fully-worked traces in §8.
3. Requirements
3.1 The journeys the data plane must serve
These journeys (from the acceptance scenarios and the pilot program) are the demand side of this design. Each is a compressed statement of what it reads.
J-1 · NetworkAccess field operations (the pilot). A NOC operator watches a wallboard of alarms (standing query with diff-push); an ops agent traces a dead circuit upstream to a candidate cut (trace), checks the segment’s as-built location and finds it CONTESTED, so it refuses to dispatch blind (point read + settledness); a field engineer sees her assigned work orders near her (list + spatial), completes one, and it leaves her queue immediately (read-your-write on a list); a post-incident review reconstructs the network as of 02:14 and asks why the system believed the joint was where it said (as-of + explain). Volume: ~1M entities in the pilot cell, growing to 5–10M amplified.
J-2 · Trafficure corridor management. Speed feeds from three providers disagree; the map renders the corridor hatched (CONTESTED on tiles); latest speeds are milliseconds fresh (LVC); an engineer forks a what-if branch, an agent forks forty (branch reads), and the forty are ranked in one comparative read (multi-branch compare); the winning intervention renders as a diff-overlay on the live map (branch-diff tiles).
J-3 · SmartBuild construction. A months-lived plan branch is reviewed against reality as a diff (plan-branch read); trench progress is read per chainage (linear-referenced traits); claimed progress and photo-derived progress diverge and the span flips CONTESTED, opening a dispute (plan-vs-evidence standing query); crews work offline for days and their devices answer “my jobs near me” instantly from local state (offline reads).
J-4 · Agents everywhere. Every agent read is budget-capped (tightest workload class), policy-compiled like any read, and answers carry settledness + quality warnings the agent must check before acting. Retrieval for agents (search, similarity) is governed: policy restricts the universe before retrieval, and every hit resolves to a governed entity reference.
3.2 Performance gates
From the pilot program design; p99, measured against the pilot cell (~1M entities, 300k+ connectivity ports, amplified to 5–10M by M3).
| Read | Gate (p99) |
|---|---|
| Point read / read-your-write | < 50 ms |
| List / worklist (validation queue) | < 200 ms |
| Circuit / dependency trace | < 500 ms |
| Map tile | < 150 ms |
| Latest high-volume value (LVC) | milliseconds |
| Others (search, aggregate, as-of, branch) | no pilot gate; engineered per §5 |
Write-side context that shapes the read design: 500 Acts/s sustained per cell, 0.5–2k Asserts/s, 1M-row bulk migration under 4 hours.
3.3 Deployment profiles
The same design must run in four shapes. This is a hard requirement, not an aspiration — half the market is on-prem or air-gapped, and the pilot itself deploys from signed bundles on customer infrastructure.
| Profile | Shape | Data-plane consequence |
|---|---|---|
| Lean | one node, one team, moderate scale | One Postgres serves seven read species (§6.1); DuckDB reads the lakehouse; LVC is in-process; no JVM anywhere in the read path |
| Scale | Jio-scale, many cells | Engines split per workload class; per-species Scale adapters (§6.2); graduations are evidence-gated (§6.3) |
| Air-gap | self-hosted, no internet | Identical shape to Scale with self-hosted adapters; baked tiles as offline base; offline-verifiable capabilities |
| Edge / Device | phone, field laptop | §4.7 — SQLite + precompiled scope; the smallest honest deployment of the same pipeline |
A customer graduates column-by-column (swap one adapter at a time); no plan shape, Surface, or SDK changes. The species table (§5) is the contract; the engine columns are engineering.
4. Architecture
4.1 Surfaces and the Query IR
Every read surface compiles to one closed, kernel-owned logical plan language (PR-2). The surfaces:
- the typed SDK query builder (primary — generated per-realm from the ontology),
- GraphQL, generated per-realm,
- MCP tools for agents (the tool surface is a projection of the ontology — a hallucinated tool fails to exist, not at runtime),
- governed SQL views — per-principal schemas of filtered entity views for BI tools; never raw tables, never a raw SQL door (raw SQL is an IR bypass),
- standing-query registration (§4.8).
The IR node set (closed; growing it is a design event, LLD §10):
| Node | Question it expresses |
|---|---|
Select(entityType, predicate) | find entities |
| `Project(traits, resolved | raw)` |
| `Traverse(link | family |
| `Trace(upstream | downstream |
| `Spatial(within | near |
| `Temporal(latest | window |
Aggregate(groupBy, measures) | grouped measures |
Similar(embedding, k) | vector similarity |
TextMatch(query) | full-text search |
Compare(branches[], plan, alignKeys) | run one plan across N branches, aligned by key (LLD §24.5) |
Subscribe(view) | standing query over an IR-defined view |
4.2 Truth tiers and serving stores
A read’s Temporal node names the kind of truth it wants; each kind has a home:
| You asked for | Tier | Served from | Freshness |
|---|---|---|---|
| Current state of an entity | operational | P1 (just-committed) / P23 (resolved world) | ms behind commit |
| Latest value of a fast-changing trait | hot | LVC | milliseconds |
| A window or history of values | warm | P4 time-series store | seconds |
| State as of a past time | cold | P3 lakehouse snapshots (bitemporal) | minutes (by design) |
| State in a branch | analytical | branch overlay + base snapshot | branch-local |
| Text/similarity hits | derived | P5 / ANN indexes | index watermark |
| Map tiles | serving | P24 tile artifacts | publish watermark |
Why tiers can never disagree: every one of them consumes the same ordered stream of committed facts (§2.1). A tier can only be behind, and “how far behind” is its watermark — a number the response can carry (PR-13). This is also what makes read barriers (§4.6) implementable everywhere: every store knows exactly which token it has applied through.
The hot path in one line (LLD §9): driver Asserts → cheap authority check → one ordered statement stream → three consumers at three speeds — LVC (ms), warm store (s), lakehouse (min). Only traits declared hot earn LVC entries; fusion policies used in the LVC must be incrementally computable.
4.3 Source classes — where answers can possibly come from
Every trait’s evidence has exactly one of two origins, declared in its P0 manifest (locked decision DP-3):
- Owned — truth born inside, written by Acts. Full tier routing, replayable, branchable.
- Mirrored — external truth entering as Asserts under a source manifest. Lands in the time-series store, fuses into the World plane, and its staleness is visible (
STALE/BLINDare queryable settledness states).
There is no third class. “Virtual” or “federated live” traits dissolve into a materialization policy on the mirror manifest:
| Policy | Mechanism | Typical source |
|---|---|---|
push | CDC / streaming driver emits continuously | ERP, OSS, sensor feeds |
scheduled | periodic sweep emits diffs | slow-changing registries |
on_read | a read finding the trait BLIND/STALE beyond its freshness class triggers a driver fetch; the fetched value enters through the ordinary Assert path; the read is answered from the plane | long-tail attributes not worth streaming |
The structural rule that keeps every invariant intact: a Query never leaves the owned planes (PR-5). on_read hydration is maintenance of the mirror, not an answer path. Consequences, all deliberate: policy still compiles into indexes we own; as-of stays honest (“we were BLIND then” is a true answer); branches never hydrate (a simulation never phones the outside world); and external non-determinism stays quarantined in the Assert path where it already lives.
Two promise deltas for on_read traits, and only two:
- Freshness — they spend more of their life
STALE; a caller may attach afreshnessbarrier (§4.6) and pay for the wait. - Filterability — PR-6. A filter over an
on_readtrait would scan only landed evidence: every value honest, the set silently incomplete. Soon_readtraits are projectable, not filterable, statically enforced at BIND. Filterability is what promotion topush/scheduledbuys.
Operational guard: hydration is batched, rate-limited through the egress port, and capped per query — a 10k-entity projection must not become a thundering herd against a customer’s ERP.
What correctly remains outside the Query surface:
- *Read-
Call*— when a workflow needs the external system’s own answer at time T as a recorded, correlated fact. A hydrated Query returns plane truth; a read-Call returns a crossing. Two questions, two verbs. - Dataset federation — analysts querying external sources in place happens below the seam (§4.9), on datasets, not entities.
4.4 The query pipeline — six stages
Query IR → 1 BIND → 2 SECURE → 3 ROUTE → 4 OPTIMIZE → 5 EXECUTE → 6 ACCOUNT → answerThe complexity budget is deliberately lopsided: SECURE carries correctness, OPTIMIZE carries the only cost-based search, and the other four are simple by design.
Stage 1 · BIND — validate everything statically checkable, reject early
Parse the surface into IR. Resolve ontology version, realm, branch, and as-of time against P0. Validate every trait reference and the kind-compatibility of every predicate. Enforce the static rules: PR-6 (no filters on on_read traits) and barrier coherence (§4.6 — a barrier on an as-of, branch, or tile read is invalid, not slow). Stamp the plan-cache key (IR shape · ontology version · policy version).
Technology: Rust, against an in-memory ontology snapshot. Cost: microseconds. Design intent: everything rejectable without touching data is rejected here, so nothing downstream needs defensive checks.
Stage 2 · SECURE — compile policy into the plan, once
The correctness-critical stage; everything downstream is allowed to be dumb because this stage is not.
- Coarse markings (realm / region / classification band) and row-level rules become indexable predicates on
Select— cheap, applied to every plan, evaluated by ordinary indexes. - Trait-level denials become column masking on
Project. Project(raw)demands its elevated capability now, not at fetch time.- Fine-grained relationship checks (ReBAC) run only on the candidate set that survives the marking predicates — the two-tier model of LLD §13.2. List queries are never N per-row checks.
Two invariants:
- Residual-predicate discipline (PR-4): every injected predicate must have a proven evaluation site — pushed to a backend that can evaluate it, or retained kernel-side for post-retrieval filtering. A plan where a policy predicate has no site is invalid.
- SECURE precedes ROUTE and OPTIMIZE, so policy predicates participate in index selection rather than being bolted on after.
Because predicates are compiled into the plan, they apply identically to composed rows — a row patched by the recent-write overlay (§4.6) passes the same predicates as a base row. The overlay therefore adds no policy surface.
Stage 3 · ROUTE — deterministic tier and plane selection
A lookup, deliberately not clever (PR-3). Truth tier = the plan’s Temporal node × the trait’s declared freshness class (latest→LVC · window→warm · asOf→cold · point→operational). Species→plane per the §5 table. A cost model choosing between tiers would be a bug: the tier is what kind of truth you asked for.
A read barrier never changes routing; it gates execution of the routed subplan on that plane’s watermark (§4.6).
Output: subplans pinned to planes.
Stage 4 · OPTIMIZE — the one genuinely hard problem, kept small
Within the pinned backends only:
- (a) Anchor selection. A multi-modal plan — “entities in this polygon, with CONTESTED speed, similar to E, two hops downstream of X” — can start at the spatial index, the settledness scan, the ANN index, or the traversal; the wrong anchor is orders of magnitude slower. The search space is 2–5 candidates, so this is System-R-lite: greedy (or tiny-DP) ordering over per-kind selectivity estimates from the registry (§4.5). No plan memo, no Cascades, no join-order explosion.
- (b) Materialized-view rewrite. If the registry holds a
SERVINGmaterialization that subsumes a subplan and is fresh enough for the caller’s workload class, substitute it. This is the reason common questions are fast (§4.5). Artifacts in any other lifecycle state are invisible to the planner (PR-9). - (c) Pushdown and pruning. Predicates and projections pushed into each backend leaf; partition, H3-cell, and snapshot pruning for cold-tier leaves.
Graduation path if cross-store ordering ever outgrows greedy: an optd-style plan memo — evidence-gated like every adapter (§6.3).
Stage 5 · EXECUTE — mechanical by design
v0: backend leaves return entity-ID sets plus trait columns; the kernel composes them with ID-set algebra (intersect / union / semi-join) in Rust; Project resolves the survivors against the World store, composing the recent-write overlay where one exists (§4.6). The entity ID is the join currency — engines never join across each other.
Graduation (evidence-gated, target M3+): embed DataFusion as the federated executor — each plane a TableProvider leaf, Traverse/Trace as custom plan nodes producing ID-set streams, Arrow end-to-end. Calcite never enters this path — no JVM in the read hot path (the dataset world keeps its Calcite, below the seam, §4.9).
Stage 6 · ACCOUNT — governance wearing an optimizer’s numbers
Before execution: the estimated cost (OPTIMIZE’s selectivity numbers doing double duty) is checked against the caller’s workload-class budget. Over budget → reject with the estimate, offer async execution (a batch job), or degrade to approximate where the surface declared tolerance. During: row / time / hydration caps; barrier waits are charged and capped per workload class; branch reads bill against the branch’s class envelope (a sweep’s forty branches meter as one aggregate budget — LLD §23.1). After: actual cost attributed to tenant / actor / workload — the pricing meter and the abuse detector are the same instrument.
4.5 The registry — the planner’s control plane
Without a control plane that knows what exists, how fresh it is, how it derives, and what it costs, the router is blind. The registry is that control plane — and it is built as a system Realm: the kernel’s own reality, populated with kernel-owned entity types, governed by the same verbs as everything else.
Dataset · Projection · Index · Materialization · EmbeddingIndex ·Metric · SourceManifest · FreshnessContract · QualityContract ·EngineProfile · StatisticsSnapshot · LineageEdge · OfflinePackageRegistering an index is an Act. Lineage is a link traversal. “Which materializations are stale?” is an ordinary Query. The catalog UI is a Surface. There is no parallel metadata store and no second security model.
Three load-bearing roles:
- Access-path inventory, per entity type — OPTIMIZE’s input. Example for
Tower: primary key → Postgres · geometry → PostGIS GiST · H3 → partition key · text → search index · embedding → pgvector (model@version) · adjacency → link index · latest → LVC · history → lakehouse. Anchor selection is a read over this inventory plus statistics. - Statistics as a materializer projection — cardinalities, per-trait distinct counts and null rates, settledness distribution, H3-cell density, link-degree distribution (supernodes are quantified before anyone traverses them), partition layouts, stream rates. Declared, versioned, rebuildable — never a side-channel
ANALYZEwith its own freshness semantics. The materializer sees every statement anyway, so these stats are fresher and finer than any generic optimizer’s. - Lineage drives semantic invalidation — the caching rule (§7.2) and re-materialization both walk the same
LineageEdgegraph. “Which caches does a work-order change touch?” is aTrace, not tribal knowledge.
The artifact lifecycle (decision DP-9). Every registry artifact — materialization, index, embedding index, tile layer, offline package — carries a lifecycle state as an ordinary trait:
DRAFT → BUILDING → VALIDATING → SERVING → DEPRECATED → ARCHIVED failure states: FAILED · QUARANTINEDThree rules do all the work (no seventh pipeline stage):
- Only
SERVINGartifacts exist to the planner (PR-9). MV-rewrite, projections, and tile serving consult the state; everything else is never a candidate, so it never needs “blocking.” - Promotion is an Act. Package-declared materializations auto-promote when their declared contracts pass — the attestation a build produces (schema, quality, freshness, lineage checks — §7.4) is the evidence. Human- and agent-authored artifacts require a steward’s promotion Act. Agents never self-certify: an agent’s draft artifact serves only inside its own branch/project scope, under its quota, until promoted.
- Names are aliases. Plans reference the registry name; the name resolves to the current
SERVINGversion; promotion repoints atomically; rollback is a repoint. Physical artifact identifiers never appear in a plan — which makes silent metric drift structurally impossible: a definition change is a new version behind the same name, and the version served is recorded in the execution explain (§7.1).
Bootstrap ladder (resolved): a file-backed manifest (M1, pre-kernel — the planner reads a file) → the same content as plain Postgres tables → system-Realm entities at M2, absorbed via an ordinary bulk load. Each rung answers the planner identically; only the storage moves.
Materialization discipline (DP-10): platforms feel fast because common questions are already answered. The ladder — raw statements → fused world state → derived projections → metric tables → serving projections (search, tiles, LVC) — is the materializer’s job, and every rung is declared (a registry entry: definition as Query IR, target, freshness contract, owner), versioned, lineage-tracked, and monitored against its contract. Never an ad-hoc pipeline. Four execution modes: streaming (statement-driven), incremental batch, full rebuild (also the disaster-recovery story), and on-demand (first read materializes, registry caches). The operator declares target lag; the system chooses the mode — freshness contracts are the API, schedules are the adapter’s problem.
4.6 Consistency — barriers, the overlay, and read-your-write
The questions this section answers: when I write something, when do my reads see it? What may I rely on? What does it cost?
Baseline: eventual, with visible lag. All serving stores apply the same ordered stream (§4.2); a read with no freshness request is served from wherever ROUTE sends it, and the response’s data-explain carries the store’s watermark. Lag is a number, not a surprise (PR-13).
The read barrier — the freshness contract (decision DP-6). Any coherent read may carry:
barrier: { afterToken: OrderToken } -- "after my write" | { freshness: Duration } -- "no staler than"onTimeout: serve | fail -- default: serveSemantics: the router holds the affected subplan until the routed store’s watermark satisfies the condition. Every serving structure has a watermark — the world store’s applied token, the search index’s, the adjacency index’s, the LVC’s — because all consume the same ordered stream. On timeout under serve (the default), the read executes anyway and the data-explain carries visible_through < requested: lag explicit, UI responsive. Under fail, the caller gets a typed error — for agent loops and workflow preconditions whose correctness needs the wait. Waits are charged by ACCOUNT and capped per workload class.
One primitive, two former problems: read-your-write (afterToken) and awaited freshness on lazily-mirrored traits (freshness) are the same mechanism. Barriers wait on watermarks (PR-8).
Coverage law. Where barriers apply, and how they are satisfied:
| Species | Barrier? | Satisfied by |
|---|---|---|
| Point / resolved entity / list | yes | overlay composition (Scale) or watermark wait (Lean) |
| Search, traverse, trace | yes | wait on that index’s watermark (a post-splice Trace(afterToken) waits for the adjacency index to pass the token — “is my splice live?” gets an honest yes) |
| Aggregate / window | yes | wait on the warm store’s watermark |
| As-of, branch reads, tiles | no — rejected at BIND | you cannot be “after your write” in March; tiles are publish-time artifacts |
The overlay never patches a text-match, a traversal frontier, or an aggregate: the moment the accelerator owns per-species correctness, it has become a second planner.
The recent-write overlay — machinery, not contract (decision DP-7). At Scale, the materializer lags seconds, and pure waiting would breach the list gate. The overlay keeps barrier waits at ~zero for entity-shaped reads. It is the Act-side analogue of the LVC: the latest committed trait patches keyed by entity, fed from the same outbox stream the materializer consumes, each entry retiring as the materializer’s watermark passes its token.
- Global and policy-filtered, never session-scoped. Everyone composes the same overlay; SECURE’s compiled predicates evaluate on composed rows exactly as on base rows, so there is no two-users-two-truths anomaly and no new policy surface.
- Lean ships without it. Materializer lag there is milliseconds; waits are already ~zero. The overlay graduates in on evidence: sustained barrier waits breaching the list gate at Scale (§6.3).
- The overlay accelerates; it never defines truth. Truth is the log and the planes; the overlay is fresh visibility over them.
Read-your-write, end to end. An Act returns its OrderToken; the SDK carries it. A point read presenting it is served from the Process plane (P1), which committed it — guaranteed by construction. An entity-shaped or index read presenting it gets barrier semantics: the field engineer’s completed work order leaves her queue because her next list read carries her token, and either the overlay composes the patch (Scale) or the watermark passes in milliseconds (Lean). Reads with no token remain eventually consistent with visible lag — strong exactly where the loop needs it, receipts everywhere else.
Per-cell honesty. OrderTokens are {cell, seq}; a barrier binds the minting cell’s watermark only. A scatter-gather read across cells promises per-cell freshness, never a global cut — stated in the explain, not hidden. (Cell mechanics: the Cells doc.)
One tier per plan (PR-7, unchanged): a single query never joins hot-now against cold-March. The SDK may sugar a two-query composition; the seam sees two plans, and the incoherence — if wanted — is userland’s explicit, visible choice.
4.7 The offline / edge read path
Offline is a branch (LLD §22): a disconnected device is an implicit device branch — base = its last-synced replica, branch log = its queued writes — and reconnection is an ordinary branch merge. This section fixes what its reads are. One rule keeps offline part of the same system instead of a parallel product (PR-11):
Offline runs the same pipeline with BIND and SECURE precompiled.
SERVER (at sync-package build) DEVICE (at read time) ┌──────────────────────────────┐ ┌──────────────────────────────┐ │ BIND + SECURE run here: │ signed │ ROUTE + EXECUTE only: │ │ working-set scope compiled │──package──▶│ SQLite base tables │ │ policy → manifest │ │ + device-branch overlay │ │ signed as a capability │ │ + FTS5 (text) + R-Tree (geo)│ │ (Biscuit: offline-verifiable)│ │ local IR subset (below) │ └──────────────────────────────┘ └──────────────────────────────┘- The device never evaluates policy; it holds policy’s output. The working-set manifest — entity types, traits, regions, materialized slices, permitted actions — is a compiled plan scope, signed as an offline-verifiable capability. Locally-authored policy does not exist.
- Local reads compose base + overlay. Every local read composes the synced base with the device branch’s pending Acts/Asserts — §4.6’s composition rule applied locally. The engineer sees her own pending work instantly, flagged as pending.
- The local IR subset is closed:
Select · Project · Traverse(bounded) · Trace(working-set-scoped) · Spatial(bbox | near) · TextMatch · Temporal(latest). ATracethat reaches the edge of the synced working set reports its frontier honestly (frontier: truncated-at-working-set) rather than pretending completeness. NoAggregatebeyond shipped materialized slices; noSimilar(v0); noCompare; no as-of (the device holds one base). Offline analytics means shipped slices, declared like any materialization. - An offline package is a registry artifact (§4.5): built by a batch job, validated, promoted to
SERVING, expiring. Expiry is enforced locally — past manifest expiry, reads degrade read-only, then lock (grace window declared per package); a server-side policy change invalidates the manifest at next contact. - Not owned here: the sync protocol, conflict adjudication, and the write-upload path (LLD §22, §15). This section owns only what a local read means.
The Edge profile’s data plane (§3.3) is this section.
4.8 The two side doors — tiles and standing queries
Two read paths that do not go through the per-request pipeline, and the rules that keep them from becoming unpoliced back doors.
Tiles are Serving-plane projections — derived, rebuildable, authoritative for nothing. The rule: policy compiles at publish time, not per request.
- Baked layers (PMTiles pyramids) are batch outputs, partitioned by policy marking — a tile file exists per coarse partition, and a request selects a partition by capability. It never evaluates policy.
- Dynamic layers (Martin) are generated functions whose SQL is produced by the same SECURE rewrite — one artifact per policy partition, republished on policy change (policy log → materializer → republish).
featureId == entityIdin every tile: a click on any rendered feature is one governed point-read away from its truth. Live trait overlays (speeds, alarm states) ride the realtime gateway from the LVC — carrying ResolvedValues, so aCONTESTEDvalue renders visibly differently.- Tile layers are registry artifacts; a re-bake promotes behind the layer alias (§4.5).
Standing queries are Subscribe over an IR-defined view: registered through the full pipeline once (BIND → SECURE → ROUTE — so a standing query is policy-compiled like any read), then evaluated incrementally by the materializer against the statement flow, delivered as diffs via the realtime gateway. Lean: re-evaluation on affected keys. Scale: warm-tier continuous views feeding the same contract.
The registration vocabulary is the closed pattern grammar of LLD §24.1 — event(type, predicate) · window(source, condition, duration) · sequence(p1 then p2 within t) · absence(p1 not-followed-by p2 within t), plus debounce / throttle / latch. The validation queue, the drift monitor, the alarm-flap suppressor, and the agent watchlist are all this one mechanism.
Plan-vs-evidence divergence is a composition, not a feature. A plan package declares its commitment traits and tolerances; registration compiles a standing Compare(plan-branch, canon) over exactly those traits; divergence beyond tolerance emits a statement that opens a WorkItem. Change-order adjudication, built from Compare + Subscribe + Aggregate with zero new machinery.
4.9 The dataset world — the altitude rule
There are two planners at two altitudes, permanently — and that is correct, not transitional (PR-12):
- Above the seam: this document’s pipeline — entities, traits, planes, policy-compiled, branch-aware. Rust.
- Below the seam: the dataset world — the existing NAP/DMP stack (Calcite planner, Trino federation, Spark/Sedona transforms, Cube semantics) serving datasets to analysts and pipelines. It keeps its SQL surface and its JVM; both are fine below the line.
The leak to refuse has a predictable shape: “the dataset stack already does X — expose it through the seam.” That is an IR bypass; the moment the dataset world’s SQL defines what the seam can express, the one-door model is dead. Traffic crosses the altitude line in exactly three governed ways:
- Mirrors up: dataset outputs enter as typed Asserts through driver manifests (the medallion staging → typed evidence path).
- Shared bytes: batch transforms read the same lakehouse the kernel writes — one copy of the data, two planners over it.
- Views down: governed SQL schemas project entity views down for BI tools — generated, filtered views, never raw tables.
The NAP engine fleet (PostGIS, ClickHouse, Trino, DuckDB, Iceberg/MinIO, Martin) is the adapter inventory both altitudes share, and its production bug ledger (~15% SQL-text correctness defects from regex-SQL surgery; a policy-blind result cache) is the standing empirical argument for compiling policy into plans (§4.4) and banning ad-hoc result caches (§7.2).
5. Read species catalog
The complete set of questions the data plane answers. This table is the contract; the engine columns are per-profile engineering (§6). Gates from §3.2 where they exist.
| # | Species | IR shape | Tier | Lean serves | Scale serves | Gate |
|---|---|---|---|---|---|---|
| 1 | Point read / read-your-write | Select(ref) · Project | operational | Postgres (P1/P23) | FDB Record Layer | <50ms |
| 2 | Resolved entity (entity-360) | Select · Traverse(*, in, 1) · Project | operational | Postgres + inverse link index | FDB + inverse index | <50ms |
| 3 | List / filter (queues, worklists) | Select(type, predicate) | operational | Postgres | FDB secondary idx + Quickwit | <200ms |
| 4 | Full-text / faceted search | Select · TextMatch | derived | Postgres tsvector/pg_trgm | Quickwit | — |
| 5 | Similarity | Similar(embedding, k) | derived | pgvector | pgvector / dedicated ANN (gated — §10) | — |
| 6 | Aggregate / metric | Aggregate(groupBy, measures) | warm | Postgres materialized views | ClickHouse | — |
| 7 | Traverse (bounded hops) | Traverse(link, dir, depth) | operational | Postgres recursive CTE (spike S2) | FDB adjacency keyspace | — |
| 8 | Trace (circuit / dependency) | `Trace(up | down | path)` | operational | CTE / closure table (S2 decides) |
| 9 | Spatial (within/near/intersects) | Spatial(op, geom) | operational | PostGIS (GiST + H3) | H3/S2 cells in FDB + PostGIS read-model | — |
| 10 | Latest high-volume value | Select · Temporal(latest) | hot | LVC in-process | LVC (FDB versionstamped / Redis Cluster) | ms |
| 11 | Window / history | Temporal(window) | warm | TimescaleDB (or ClickHouse — gated) | ClickHouse | — |
| 12 | As-of / canonical | Temporal(asOf) | cold | Iceberg via DuckDB | Iceberg via DuckDB/Trino | — |
| 13 | Branch read / diff | Temporal(branch) · Compare | analytical | snapshot clone + branch overlay + DuckDB | same | — |
| 14 | Multi-branch compare | Compare(branches[], plan, keys) | analytical | DuckDB over clones + overlays | same | — |
| 15 | Raw evidence | Project(raw) — elevated capability | evidence stores | Timescale + log archive | ClickHouse + lakehouse archive | — |
| 16 | Standing query | Subscribe(view) | streaming | materializer re-eval → push | warm continuous views → push | — |
| 17 | Offline / local read | precompiled subset (§4.7) | device branch | SQLite base + overlay (FTS5, R-Tree) | same | local-instant |
| 18 | Tile | not per-request IR (§4.8) | serving | Martin (dynamic) + PMTiles (baked) | same, policy-partitioned | <150ms |
Notes the table needs:
- Barriers (§4.6) may ride any species where coherent; BIND rejects them on #12, #13, #14, #18.
- #13 covers plan branches too. A months-lived planning branch is a logical branch — base pointer + act log, no snapshot clone — so its reads compose an overlay exactly like device branches; periodic rebase is what surfaces its staleness; its budget comes from its branch class.
- #14 is fan-out plus alignment, nothing more.
Comparereturns a table keyed by branch; ranking and statistics are ordinaryAggregateover it — judgment stays in userland. - Structure of the table itself: the operational tier carries half the species (which is why Lean’s answer is one Postgres); graph and spatial species are index-shape questions, not engine questions (adjacency and cells are keyspace layouts); and only four species ever touch the lakehouse interactively (#12–14, #15-archive) — everything else reads projections.
6. Engine mapping and scaling
6.1 What Lean collapses to
One Postgres instance is the operational store, the world store, the time-series store, the search index, and the tile source; the LVC is in-process memory; DuckDB reads Iceberg for cold reads. One engine serves species 1–11 and 18. No JVM exists in the read path. This is not a compromise configuration — it is the honest first build, and the pilot ships on it.
6.2 The Scale columns
Each species graduates independently to its Scale adapter (the §5 table’s sixth column): the operational store to FoundationDB, search to Quickwit, warm history to ClickHouse, the LVC to a shared tier, spatial to cell-keyed layouts, traces to precomputed reachability projections. ACCOUNT’s workload classes map to separate engine pools, so a dashboard stampede cannot starve field operations.
6.3 Graduation triggers — evidence, never aspiration
| Graduation | Trigger |
|---|---|
| Recent-write overlay arrives | sustained barrier waits breaching the list gate (<200ms p99) at Scale materializer lag |
| P1 Postgres → FDB | proven write contention / sharding need, not predicted |
| Search Postgres → Quickwit | index size / query latency evidence |
| Warm tier Timescale → ClickHouse | assert feed volume (pilot: only if the inbound-feed stretch lands) |
| EXECUTE ID-set algebra → DataFusion | residual relational work volume; aggregate-pushdown limits (ADR due at M3 with Query IR v0) |
| Greedy anchor ordering → plan memo | cross-store ordering complexity observed in real plans |
| pgvector → dedicated ANN | a real embedding corpus with recall/latency targets |
6.4 Air-gap and Device
Air-gap is the Scale shape with self-hosted adapters, PMTiles-heavy serving, and offline-verifiable capabilities — nothing in the read path assumes an internet. Device is §4.7. The profile ladder is monotone: nothing a smaller profile promises is withdrawn by a larger one.
7. Cross-cutting concerns
7.1 Explain — three levels, two from the kernel
Every response carries provenance; explain expands it.
- Execution explain (kernel): which tier, snapshot, ontology + policy version, engines, indexes, and materialization versions (aliases resolve to versions; the version served is recorded) answered each plan leaf. The audit answer to “why does it say 42.”
- Data explain (kernel): per-trait
derivedFrom(the evidence chain), settledness, freshness, quality warnings (§7.3), the watermark pairmaterialized_through/visible_throughthat makes lag a number, and the barrier verdict when one was requested (satisfied|timedOut, served). - Business explain (userland): why the answer matters — composed by Surfaces and agents from the first two. The kernel supplies evidence, never meaning.
7.2 Caching — semantic or nothing
| Cache | Verdict | Why |
|---|---|---|
| Plan cache | yes | keyed (IR shape · ontology v · policy v); invalidated by schema/policy log events; pure win |
| LVC | yes | is a tier, not a bolt-on |
| Recent-write overlay | yes (Scale) | a freshness layer, not a cache — entries retire on watermark, never invalidated by guess (§4.6) |
| Materializations | yes | the governed form of result caching: declared, lineage-invalidated, lifecycle-gated (§4.5) |
| Tile cache | yes | policy-partitioned by construction (§4.8) |
| Ad-hoc result cache | no (v0) | a result is capability-scoped data; cache keys that omit the policy context are leak vectors, and keys that include it barely hit. The predecessor stack’s policy-blind Redis query cache is the cautionary precedent |
Invalidation is semantic everywhere (PR-10): a statement lands → the materializer knows which materializations, LVC entries, standing queries, and tiles derive from it, because lineage is queryable.
7.3 Metrics and quality
Metrics. Dashboards must not each define “SLA breach” differently. A Metric is a P0-declared derived measure — formula (an Aggregate plan), grain, dimensions, owner, freshness contract — projected into the SDK as a typed read (metrics.towerHealth(region)), materialized per §4.5, versioned behind its alias. The kernel knows a metric’s shape; userland owns the formula.
Quality. Quality contracts ride source manifests: schema, freshness, null/range thresholds, spatial validity, duplicate policy. Violations become quality statements — evidence about evidence — surfacing in the same channel as settledness: a ResolvedValue can carry warnings: [source stale 42m · confidence low · schema drift], visible in the data explain. An agent checks the warnings before acting; that check is what makes autonomous action honest. There is no separate data-quality subsystem, because quality is a property of truth and lives where truth lives.
7.4 Attestations — how builds earn SERVING
A materialization job’s output is an attestation, not just a table: row counts, input versions and watermarks, quality/policy/lineage check results. That attestation is the evidence the lifecycle promotes on (§4.5): build → validate → promote. A failed check lands the version in FAILED/QUARANTINED and the alias never moves — the previous SERVING version keeps answering.
8. Worked examples
Four reads traced end to end. All run against the NetworkAccess pilot realm.
8.1 Read-your-write on a worklist (Lean)
Kavya completes work order WO-4711 on her phone, online. Her queue must not show it again.
- The
completeWorkOrderAct commits; P1 returnsOrderToken {cell: c1, seq: 88412}. The SDK stores it in her session. - Her app refreshes the worklist:
Select(WorkOrder, assignee = kavya ∧ status ∈ {open, in_progress})withbarrier: {afterToken: 0x…88412}, onTimeout: serve. - BIND: valid plan; barrier coherent for a list read.
- SECURE: her region marking becomes an indexable predicate; no masked columns for her role.
- ROUTE: operational tier — the world store’s list tables.
- OPTIMIZE: registry stats say the assignee index is far more selective than the region predicate; anchor = assignee index.
- EXECUTE: the world store’s watermark is already past 88412 (Lean materializer lag is ~ms), so the barrier is satisfied without waiting; index scan → ID set → resolve traits.
- ACCOUNT: interactive-UI budget; wait time ≈ 0 charged.
- Answer: the list without WO-4711. Data explain:
visible_through: 88415, barrier: satisfied.
At Scale, step 7 differs: if the world store’s watermark were behind, the recent-write overlay would compose WO-4711’s status patch (removing it from the result) instead of waiting.
8.2 A trace with policy, after a write (Scale)
The ops agent asks: “everything downstream of port P-2291, restricted to my capability’s region,” right after a splice Act reconnected it.
- Plan:
Select(Port, id = P-2291) · Trace(downstream, path: fiberPath) · Project(status, geometry)withbarrier: {afterToken: splice-token}, onTimeout: fail— an agent loop needs the post-splice topology, or it must not act. - SECURE: region scope predicate injected; the trace is bounded to entities carrying the agent’s marking.
- ROUTE: operational tier; the trace runs on the derived adjacency index (declared
hotpath → maintained strand-to-strand keys). - Barrier: the adjacency index’s watermark is 2 s behind the splice token. The router waits (coverage law: traces never compose the overlay). Watermark passes at +1.4 s — within the agent’s wait cap.
- EXECUTE: one range scan per logical hop; ID set → resolved traits. Degree counters (registry stats) had already confirmed no supernode on the path.
- Answer: the downstream set including the newly-spliced segment; explain records the wait. Had it timed out: a typed failure, and the agent retries or escalates — it never acts on a stale trace.
8.3 An as-of audit read (any profile)
Post-incident review: “the network around joint J-88 as of 02:14 last Tuesday — and why did the system believe the joint was at that location?”
- Plan:
Select(area around J-88) · Temporal(asOf: 2026-06-23T02:14) · Project(resolved), thenexplain(J-88.location). - BIND would reject any barrier here (as-of reads take none).
- ROUTE: cold tier — lakehouse snapshots, read through DuckDB; bitemporal, so the answer reflects what was believed at 02:14, not what later evidence revised.
explainreturns the evidence chain: the original survey statement, the as-built assertion that contested it, the fusion policy and trust weights as of that time (fusion configuration is itself bitemporal — LLD §23.2), and the settledness the value carried.- Audit is a query, not a subsystem.
8.4 An offline read (Device)
Kavya, underground, no signal: “my assigned work orders within 5 km,” having just marked one complete locally.
- Her device holds a synced working set (signed manifest: her region, her entity types, 14-day expiry) and a device branch containing her pending completion Act.
- Local plan:
Select(WorkOrder, assignee = me) · Spatial(near(gps, 5km)) · Temporal(latest)— inside the precompiled scope, so no policy evaluation happens on the device; the manifest is the compiled SECURE output. - Local ROUTE + EXECUTE: SQLite base tables + R-Tree spatial index; the device-branch overlay composes her pending completion — that work order shows
status: complete (pending sync). - A local
Tracefrom one work order’s cable stops at the working-set edge and reportsfrontier: truncated-at-working-set— honest, not silently complete. - On reconnect, the device branch merges (write side, LLD §22); her local reads never claimed to be more than her base + her intent.
9. Decision log
Format: context → decision → rejected alternatives → status. PR-n references are to §2.2.
| ID | Decision | Rejected | Status |
|---|---|---|---|
| DP-1 | One Query IR, many surfaces, many engines (PR-2). Every read surface compiles to one closed logical plan; policy is enforced there, once. | Per-surface planners; raw SQL as a first-class surface (an IR bypass); Calcite as the kernel’s planner (JVM in the hot path — it stays below the seam, §4.9) | Locked |
| DP-2 | Semantics-first routing (PR-3). Truth tier = Temporal node × freshness class; cost optimization only within a tier. | Cost-based tier selection (“the cache is faster than the log”) | Locked |
| DP-3 | Two source classes — owned and mirrored; “virtual” dissolves into push / scheduled / on_read materialization policies; on_read traits are projectable, never filterable (PR-5, PR-6). | A federated “virtual” class answering reads live from external systems (breaks policy compilation, as-of honesty, and branch isolation) | Locked |
| DP-4 | Policy compiles into the plan (PR-4): markings → indexable predicates, denials → column masks, before routing; residual predicates must have proven evaluation sites; two-tier authz (markings then ReBAC on survivors). | Per-store policy features; post-hoc result filtering as primary enforcement; per-row authorization checks on lists | Locked |
| DP-5 | EXECUTE v0 = ID-set algebra in Rust; entity ID is the join currency; engines never join across each other. DataFusion embed is the named graduation. | Distributed joins across engines; a federated executor on day one | Locked (graduation gated, §6.3) |
| DP-6 | Read barriers — `{afterToken | freshness}withonTimeout: serve | fail(default serve-flagged), waits charged per workload class; satisfied against per-store watermarks; per-cell promises only (PR-8). Supersedes the earlier point-read-only token echo; absorbs the formerawait_freshness` fork. |
| DP-7 | Recent-write overlay = Scale accelerator behind the barrier contract — an Act-side LVC: global, policy-filtered, entity-shaped reads only, retiring on watermark. Lean ships without it. | The overlay as the read contract (always-on composition — a second correctness path in every profile); session-scoped overlays (two-users-two-truths); patching search/trace/aggregate results from the overlay | Locked (arrival gated, §6.3) |
| DP-8 | The registry is a system Realm (PR-1): catalog, statistics, lineage as kernel-owned entities governed by the ordinary verbs; bootstrap ladder file → tables → realm at M2. | A parallel metadata store with its own security model (the “second catalog” anti-pattern) | Locked |
| DP-9 | Artifact lifecycle: DRAFT…SERVING…ARCHIVED as registry state; only SERVING is visible to the planner; promotion is an Act (auto for package-declared artifacts passing contracts; steward Act otherwise; agents never self-certify); names are aliases repointed atomically (PR-9). | A “queryability gate” as a pipeline stage; the L0–L5 certification ladder (its levels dissolve into existing boundaries: staging is outside the governed universe; “governed” is automatic per DP-4; “audited” is the evidentiary admission class); physical artifact names in plans | Locked |
| DP-10 | Materialization discipline (PR-10): every precomputed answer declared, versioned, lineage-tracked, contract-monitored; builds produce attestations; target-lag declared, mode chosen by the system. | Ad-hoc pipelines; schedule-first configuration | Locked |
| DP-11 | No ad-hoc result cache (v0). Plan cache, LVC, overlay, materializations, tile cache — all semantic; nothing else. | Policy-blind result caching (the predecessor’s incident ledger is the evidence) | Locked |
| DP-12 | Metrics are P0-declared measures behind aliases; the kernel knows shape, never meaning. | Per-dashboard metric definitions; a separate semantic-layer product above the seam | Locked |
| DP-13 | Quality is evidence about evidence: violations are statements carried as warnings on ResolvedValues, in the same channel as settledness. | A parallel data-quality dashboard subsystem | Locked |
| DP-14 | Tiles compile policy at publish time: baked layers policy-partitioned, dynamic layers generated by the SECURE rewrite, featureId == entityId, layers as registry artifacts. | Per-request policy evaluation on tiles; unpartitioned baked pyramids (an authorization bypass) | Locked |
| DP-15 | Standing queries register through the full pipeline once, evaluate incrementally, deliver diffs; vocabulary = the closed LLD §24.1 grammar; plan-vs-evidence divergence = declared standing Compare composition. | A general CEP engine in the kernel; a bespoke divergence-detection subsystem | Locked |
| DP-16 | Two planners, two altitudes (PR-12): the dataset world keeps Calcite/Trino/Spark below the seam; three governed crossings (mirrors up, shared lakehouse bytes, governed views down). | Exposing dataset-world capabilities through the seam; freezing analysts out entirely | Locked |
| DP-17 | One tier per plan (PR-7). Mixed-tier reads are two userland queries, visibly. | Silent cross-tier joins by the planner | Locked (revisit gated on a named journey) |
| DP-18 | Offline = same pipeline, BIND+SECURE precompiled (PR-11): signed manifest as compiled scope, closed local IR subset, honest frontiers, packages as expiring registry artifacts. | A local policy engine; CRDT-everything and last-write-wins sync (the branch-merge model stands); arbitrary local analytics | Locked |
| DP-19 | Barriers, watermarks, and freshness promises are per-cell; scatter-gather states per-cell freshness in the explain. | Pretending a global consistent cut across cells | Locked (cell mechanics in the Cells doc) |
| DP-20 | QuerySpec is the wire surface; the QueryIR (LLD taxonomy: IR-2) never serializes publicly — BIND is the QuerySpec→IR compiler; the where condition tree (all/any/not + closed leaf set incl. settledness/geo/text/similar leaves) is the shared grammar organ. Detail: Query Surface, ADR-0028. | Serialized IR on the wire (freezes the QueryIR early); GraphQL as the core API (resolver model = post-hoc policy; open composition vs the closed grammar — stays a generated projection per §4.1); dual endpoints | Locked (2026-07-03) |
| DP-21 | Four typed doorways, one pipeline — /query · /aggregate · /series · /subscribe, split by answer kind (PR-7 at the wire); all enter the same six stages; read_series and the trace syscall absorb behind the seam, leaving zero reads outside it. | A single kitchen-sink endpoint discriminated by body kind; per-doorway pipelines | Locked (2026-07-03) |
| DP-22 | Streaming delivery matrix over one O(row) execution — JSON page (keyset cursor, never OFFSET) · NDJSON with in-band terminal summary (a stream without its summary is known-incomplete) · Arrow IPC with ontology-derived schemas · export-as-job to blob with attestation; ACCOUNT meters during via a stream combinator. | Buffering results kernel-side; OFFSET pagination; completion status only at the HTTP layer | Locked (2026-07-03) |
| DP-23 | Subscribe v0 = snapshot + ordered view-membership diffs (enter/update/leave, OrderToken-stamped, resume ring buffer + resync, heartbeats carry visibleThrough) over POST-SSE at Lean; WebSocket is the P26 graduation; Lean evaluation = re-eval on affected keys (§4.8 unchanged). | Raw entity CDC for clients to re-filter (client-side filtering, banned); WebSocket-first at Lean; full incremental view maintenance before evidence | Locked (2026-07-03) |
| DP-24 | The strict mirror is a machine check: one fixture realm, language-neutral golden (spec, result) pairs covering executable blocks and BIND rejections, run by both the kernel CI (seeded Postgres) and the TS FixtureKernelClient CI. | Doctrine-only compatibility; per-implementer test suites that drift | Locked (2026-07-03) |
9.1 External material adjudicated
The “Query Fabric” draft and its parent conversation (preserved under docs/_archive/adjudicated-inputs/) were adjudicated into this design in two passes (2026-07-02). Everything they proposed appears above either as an adopted-and-recast decision (the registry/catalog gap → DP-8; materialization discipline → DP-10; the write-visibility overlay → DP-7; the queryability gate → DP-9; offline package safety and local-store mechanics → DP-18; metric/quality/explain layers → DP-12/13 and §7.1; hybrid-retrieval governance → the Similar/search execution contract, §3.1 J-4), as something the kernel already had under another name (its “Semantic IR” = the Query IR; “actions vs assertions” = the Act/Assert verbs, which additionally carry fusion, settledness, and bitemporality; “attribute derivation modes” = source classes + fusion; its “Result Composer” = EXECUTE + SECURE’s masking, with redaction moved into the plan), or as an explicitly rejected alternative in the table above (Calcite in the kernel path, graph databases as masters, the L0–L5 ladder, safety-mode vocabulary — those are workload classes plus barrier options — session overlays, ad-hoc caches, CRDT sync).
10. Open questions
Each with the event that gates its resolution. Deciding earlier would be guessing.
| # | Question | Gate |
|---|---|---|
| 1 | EXECUTE graduation — what evidence volume of residual relational work triggers the DataFusion embed; Substrait’s role in DuckDB hand-off | ADR at M3, with Query IR v0 running |
| 2 | Overlay arrival — design fixed (DP-7); build when sustained barrier waits breach the list gate at Scale | Scale-profile materializer lag observed |
| 3 | Trace serving at pilot — recursive CTE vs closure table vs pgrouting | spike S2 on real connectivity data |
| 4 | Warm-store engine at pilot — TimescaleDB vs ClickHouse | inbound-feed stretch goal lands (feed volume decides) |
| 5 | Vector at Scale — pgvector-partitioned vs dedicated ANN; on-device Similar | a real embedding corpus with recall/latency targets; a field journey needing local similarity |
| 6 | Mixed-tier reads — does any real journey need one plan across tiers? | a named journey; the answer must produce snapshot semantics, not a convenience |
| 7 | **on_read hydration under freshness barriers** — cap policy, partial-hydration answers, herd behavior | the first real on_read source (IT-ops is the natural candidate) |
| 8 | Divergence-tolerance vocabulary — what forms the plan-commitment declaration supports (absolute, %, temporal slack) | the first plan package (SmartBuild) |
| 9 | Governed SQL view depth — how much IR the per-principal schemas expose (Traverse in SQL is where it gets contested) | BI-tool demand from a real deployment |
The one-line summary for the hallway: every read walks through one door, gets policy sewn into it, is routed to the kind of truth it asked for, and comes back with its evidence, its freshness, and its bill.