HLD (Historical)
Vinxi Kernel — High-Level Design
STATUS: HISTORICAL. Superseded by
ports-and-adapters.md(architecture contract) andlld.md(full depth). Preserved for the Palantir-lessons analysis and as the original candidate stack, which now lives on as the Scale reference profile. Do not implement from this document.
Concrete technology choices for the architecture in the-seam.md. Fresh stack, chosen from first principles and anchored to Palantir’s actual infrastructure primitives — Conjure, AtlasDB, Object Storage V2 / Funnel, Apollo, Rubix — not just its Ontology. Every choice names the invariant it serves and the Palantir lesson it steals. Status: HLD, forks flagged in §7.
1. What the HLD must honor
The conceptual design fixes eight invariants the physical stack cannot violate:
- The seam is the only door. Every
Query / Act / Assert / Callpasses one mediated gateway; no component reaches a store directly. - Three planes. Operational (rebuildable cache), log (ordered spine + audit), analytical (canonical state truth).
- Six verbs. Triggers
Subscribe · Schedule; effectsQuery · Act · Assert · Call. - Capabilities are tokens. Authority is acquired at the seam and carried as a signed, scoped, typed token.
- Branching is copy-on-write. A simulation is a cheap clone of an analytical snapshot plus a branch-local log.
- Provenance selects verb and storage. Owned decisions →
Act→ log + state; declarations/mirrored →Assert→ time-series; hypothetical → branch; external effect →Call→ recorded fact. - Thin kernel, meaning in userland. The kernel knows kinds and shape; the customer owns entity types and semantics.
- One binary, many modes. SaaS, dedicated, on-prem, air-gapped, edge — the same stack, deployed differently.
2. The Palantir lessons, distilled
Palantir’s real edge is a set of infrastructure primitives, each of which answers a question our design also asks. These are the lessons the stack is built on.
| Palantir primitive | What it actually is | The lesson we steal |
|---|---|---|
| Conjure | An IDL: one API definition (YAML) → a JSON intermediate representation → generators emit typed clients in Java, TypeScript, Python | Compile a schema to a language-neutral IR first, then fan out to per-language SDKs. This is exactly “compile reality into a typed SDK” — except our IR is generated from the live Ontology, not hand-written. |
| Dialogue | Client-side RPC for Conjure APIs — client-side load balancing, AIMD concurrency limiting, retries, queuing | Push resilience into the generated client; the SDK is smart, not just typed. |
| AtlasDB | MVCC transactions layered over commodity KV stores (Cassandra, Postgres); the backend for every Palantir product | Don’t marry a database; build transactions and snapshot-isolation over a scalable ordered KV, and keep the store swappable. |
| Object Storage V2 + Funnel | Funnel indexes datasets and user edits into serving object databases; V2 split indexing from querying so each scales alone | The operational store is a derived index built by a materializer, and the indexer and the query path are separate services. |
| Actions + action log | Actions apply governed edits and record a historical log of decisions | The decision log is a first-class artifact; audit is a byproduct of the write path. |
| Spark on Kubernetes | Co-developed by Palantir, merged upstream; their batch/transform engine | Use Spark-on-K8s for heavy batch, ML, and reindex/backfill — but only there. |
| Federation | Foundry federates queries to external systems instead of ingesting everything | Mirror what you must; federate what you can. Not all truth needs to land. |
| Apollo | Declarative pull-based CD: a Hub computes desired state, per-environment spokes reconcile; Release Channels; cryptographically signed bundles cross the air gap | Air-gapped delivery is a pull model with signed bundles and per-site controllers — not a push pipeline. |
| Rubix | Hardened zero-trust Kubernetes; compute nodes drained and replaced every ~48h | Ephemeral, zero-trust nodes make persistence hard for an intruder; bake it into the runtime. |
Two divergences from Palantir are deliberate and justified below: Rust for the kernel’s hot path where they use Java, and a fully open lakehouse (Iceberg) where their datasets are a closed format. Both are chosen for the security-chokepoint and air-gap-portability requirements that are sharper for us than for them.
3. Component architecture
Organized by layer. Each row: the choice, its role, and the Palantir primitive it echoes.
A. Contract & SDK layer — the Conjure lesson
The schema is the contract; the SDK is its projection.
| Component | Choice | Role |
|---|---|---|
| Ontology / schema registry | Rust service over the transactional substrate (§C) | Holds the declared reality — entity types, traits, kinds, actions, policies, capabilities. Introspectable (reflection = a Query against schema). Versioned; a schema change is an Act. |
| Reality IR | Custom, modeled on Conjure’s JSON IR | The live schema compiled to a language-neutral intermediate representation — the pivot every generator reads. |
| SDK projector | Codegen (Rust host running per-language generators) | Reads the Reality IR → emits branded TypeScript / Python / Rust SDKs per reality: value types as branded types, Actions as typed functions, Events as typed payloads, capabilities as typed tokens. Re-emits on schema change; publishes versioned packages to a per-tenant registry. |
| Base kernel API | Protobuf + Connect/gRPC (Buf) | The stable, universal verb surface (Query/Act/Assert/Call/Subscribe/Schedule). Connect gives one contract for browser and backend. |
| Client runtime | Generated client + Dialogue-style resilience | Client-side load balancing, concurrency limiting, retries — resilience lives in the SDK. |
Divergence: Palantir hand-writes Conjure YAML; our IR is generated from the live Ontology, so schema evolution is a runtime
Actwhose consequence is an automatic SDK re-projection. Conjure is the machine; the dynamic front-end is ours.
B. The seam — the kernel runtime
The one door. Stateless, horizontally scaled, on the hot path.
| Component | Choice | Role |
|---|---|---|
| Verb gateway | Rust (Tokio, Tonic/Connect) behind Envoy | Terminates every verb call. Authenticates the Actor, resolves the capability, routes to the right plane, writes through the commit path. The syscall interface as a service. |
| Capability service | Rust; issues Biscuit tokens | Implements acquire() — evaluates full policy (§H), returns a signed, attenuable, offline-verifiable capability token. Re-validated cheaply at invoke(). Biscuit’s offline verification is what makes air-gapped capability checks possible. |
| Query router | Rust | Routes reads by shape: point / read-your-write → operational; aggregate / historical / as-of → analytical; search → search index. Userland sees one model; the router hides the plane. |
Divergence: Rust, not Java. The seam is a security chokepoint and the WASM host (§F) is Rust-native (Wasmtime). Memory safety plus predictable latency at the one place everything funnels through is worth leaving Palantir’s JVM ecosystem.
C. Operational plane — the AtlasDB + Object Storage V2 lesson
A rebuildable, low-latency, transactional serving index.
| Component | Choice | Role |
|---|---|---|
| Transactional substrate | FoundationDB | The modern AtlasDB: an ordered KV with serializable ACID, MVCC snapshot reads, and versionstamps. Its ordered keyspace serves graph adjacency and geo (via H3/S2 cell keys) natively. Runs the same everywhere, including air-gapped. |
| Typed object layer | FDB Record Layer | Typed (protobuf) records + secondary indexes over FDB — the direct analogue of Object Storage V2’s typed, indexed object store. |
| Search / full-text / complex filter | Quickwit | Object-storage-native, Rust, decoupled compute/storage full-text + faceted search — the Phonograph-Elasticsearch role, but cheaper and air-gap-friendly. Mature alternative: OpenSearch. |
| Geo serving | H3 / S2 cell indexes in FDB | Point, proximity, and spatial-join serving. The kernel indexes a trait spatially because its kind is geo — mechanism, not meaning. |
The commit trick: an Act’s state write and its outbox record land in one FDB transaction, and FDB’s versionstamp is the global order — so the operational commit itself stamps the effect log’s ordering. No separate ordering authority needed.
Divergence & fork: FoundationDB + Record Layer is a real build investment. The pragmatic on-ramp is PostgreSQL + Citus (with PostGIS for rich geo) behind the same seam interface, migrating to FDB when scale demands. Because the seam closes over the store, this is a swap, not a rewrite. (§7)
D. Log / spine — the Actions-log lesson
| Component | Choice | Role |
|---|---|---|
| Effect log | Redpanda | Kafka API, single C++ binary (no JVM/ZooKeeper), tiered storage to object store. Carries Act events and Call facts, ordered by FDB versionstamp. The audit truth, the CDC backbone, and the branch-replay source. |
| Long-term archive | Redpanda tiered storage → Iceberg on MinIO | Infinite retention for audit and replay-past-the-snapshot-horizon. |
Redpanda over Kafka for operational simplicity in on-prem and edge; over NATS for the Kafka ecosystem, tiered storage, and throughput the spine needs.
E. Analytical plane — the Spark + federation lesson
The canonical state truth, snapshots, aggregates, and history.
| Component | Choice | Role |
|---|---|---|
| Table format | Apache Iceberg | Canonical current-state and snapshots (as-of queries for free). Open and engine-agnostic — the air-gap-portability requirement rules out closed formats. |
| Catalog | Nessie | Git-like catalog branching — reality-branches map onto catalog branches, above Iceberg’s own snapshot/branch. (Governance-forward alternative: Apache Polaris.) |
| Object storage | MinIO on-prem/air-gap · S3 in SaaS | The lake. |
| Batch / transform / ML | Spark on Kubernetes | Heavy pipelines, feature computation, reindex/backfill. The Palantir lesson, used only where batch belongs. |
| Streaming materializer | Apache Flink | The Funnel analogue: consumes the log, writes Iceberg canonical state, updates the operational index, feeds the time-series store. Exactly-once, stateful. |
| Assert time-series | ClickHouse | High-volume declarations: billions of rows, windows, last-value, bitemporal. Also the OTel backend (§L). |
| Federated analytical SQL | Trino | Cross-entity, historical, ad-hoc SQL over Iceberg and external sources in place — federate what you needn’t mirror. |
| Embedded / edge / in-branch | DuckDB | In-process reads of Iceberg for branch diffs and edge deployments. |
The V1→V2 lesson is structural here: the materializer (Flink/Spark) and the query path (Trino/serving) are separate services that scale independently.
F. Compute & workflow — the Functions lesson
Sandboxed user code, born inside the rings with no ambient authority.
| Component | Choice | Role |
|---|---|---|
| Function sandbox | WASM — Wasmtime + Component Model | Short logic, effectful value-type constraints, agent-authored functions. Deny-by-default: the only imports are the kernel-injected Query/Act/Assert/Call host functions. Compile TS, Rust, Python to components. This is “no ambient authority,” mechanized. |
| Heavy / long-running / drivers | Firecracker microVMs (gVisor alt) | Isolation for driver code and long jobs; no network except the egress gateway (§G). |
| Workflow engine | Temporal | Durable, replayable workflow execution — a workflow is the first userland program, composed of the verbs, and Temporal makes it resumable and observable. Every activity’s effects route through the seam. |
G. Drivers & connectors — the boundary (Call)
| Component | Choice | Role |
|---|---|---|
| Egress gateway | Rust proxy (or Envoy) | The Call chokepoint: enforces which driver holds which external capability, applies rate limits, audits every crossing. You can’t govern Stripe; you govern the crossing. |
| Bulk / CDC ingestion | Debezium + driver SDK | Mirrored data in: external changes become Asserts (declarations), never decision Events — this is what keeps the log lean. |
H. Authorization — the dynamic-security lesson
| Component | Choice | Role |
|---|---|---|
| ReBAC / RBAC | SpiceDB | Zanzibar-style relationship graph — the instance and relationship checks (does this actor own this work order). |
| ABAC | Cedar | Context predicates (amount thresholds, business hours, road-open-ness) with formal analyzability. |
| Capability tokens | Biscuit | Attenuable, offline-verifiable tokens. acquire() fans out to SpiceDB + Cedar, mints a scoped Biscuit; invoke() re-checks it at the seam. |
The policy split from the design doc lands exactly here: only the capability vocabulary is compiled into SDK types; who-holds-it / on-which-instance / under-what-condition all stay in SpiceDB + Cedar, changeable by an admin Act with no SDK regeneration.
I. Identity — Actors, human and agent
| Component | Choice | Role |
|---|---|---|
| Human identity | Zitadel (OIDC) | Users, SSO, MFA. Self-hostable for air-gap. |
| Workload / agent identity | SPIFFE / SPIRE | Every service and agent gets an attestable SVID. The seam verifies identity while staying author-blind about intent — an agent is an Actor with a verifiable name. |
J. UI — the Blueprint lesson
| Component | Choice | Role |
|---|---|---|
| Component kit | React + TypeScript (Blueprint or Radix base) | Consumes the generated per-reality SDK, so dashboards are type-safe against the live reality. |
| App / dashboard builder | Low-code builder over the SDK | Binds components to Entity/Trait/Action through the typed contract — the SDK is what makes on-the-fly apps safe. |
K. Deployment & multi-mode — the Apollo + Rubix lesson
The crux for SaaS-through-air-gapped from one codebase.
| Component | Choice | Role |
|---|---|---|
| Runtime substrate | Kubernetes everywhere | One target for every mode. |
| Hardened profile | Rubix-inspired | Zero-trust, ephemeral nodes cycled on a fixed interval; intruder persistence is bounded by the cycle. |
| Desired-state CD | Argo CD / Flux base, wrapped in an Apollo-style hub-and-spoke pull model | A central Hub computes desired state and emits signed release bundles (Sigstore/cosign); each environment runs a spoke controller that pulls — or in true air-gap, receives via physical media — the signed bundle and reconciles locally. Release Channels (STABLE / CANARY / RELEASE) per environment. |
| Air-gap transfer | Signed, integrity-validated bundles | The Apollo “Airgapped SaaS” pattern: push pipelines break across the gap; a pull model with signed bundles and a local reconciler does not. This is the concrete form of the Factory-Hub / Site-Spoke split. |
L. Observability & audit
| Component | Choice | Role |
|---|---|---|
| Telemetry | OpenTelemetry → ClickHouse → Grafana | Traces, metrics, logs. |
| Audit | The effect log (Redpanda → Iceberg), queried via Trino / DuckDB | Free: audit is the log’s byproduct, not a separate system. |
4. Request lifecycles
The architecture is clearest as the four effects flowing through it.
Query (read). Gateway authenticates the Actor → capability check → Query router picks the plane: point / read-your-write → FDB Record Layer; search → Quickwit; aggregate / historical / as-of → Trino over Iceberg (snapshot for as-of); high-volume time-series → ClickHouse. One model in, one plane answers, the seam hides which.
Act (owned decision). Gateway → acquire() evaluates SpiceDB + Cedar → Biscuit token → the Action runs in the WASM sandbox, emitting a declared mutation intent → the seam commits it in one FDB transaction (state + outbox), versionstamp-ordered, immediately readable (read-after-write from the operational plane) → the outbox row ships to Redpanda (now the audited, ordered fact) → Flink materializes it into Iceberg (canonical state, snapshotted) and refreshes the operational index and ClickHouse. Analytical catches up and becomes final. Reconciliation rule stands: analytical wins on state, log wins on history, operational is rebuildable.
Assert (declaration / mirrored). Debezium or a driver ingests the external value → gated cheaply on is this source authorized over this trait → written bitemporally to ClickHouse, bypassing the outbox and the decision log → only the current hot value is cached in FDB for point reads. High volume never touches the decision spine.
Call (boundary crossing). A driver in a Firecracker microVM holds one external capability → its request exits only through the egress gateway, which enforces the capability, rate-limits, and audits the crossing → the returned value is recorded as a fact on the log (never replayed). In a branch, this fact is stubbed or served from record — never re-issued.
Branch (simulation). Nessie branches the catalog + Iceberg snapshots the state (copy-on-write) → replay Acts into the branch (deterministic) → stub Calls, take Asserts as-of from ClickHouse → run the sim engine (e.g., a Spark or DuckDB job) against the branch → diff against canon. The operational plane is never branched.
5. Deployment topology across modes
One stack, five shapes, differing only in what’s managed where:
- SaaS — full stack in the vendor cloud; S3, managed MinIO/ClickHouse; Apollo Hub central.
- Dedicated cloud — the same, single-tenant, in the customer’s cloud account.
- Managed on-prem — Kubernetes on customer hardware (his Proxmox-class substrate); MinIO for the lake; Hub reaches the spoke over a connected link.
- Air-gapped — spoke controller reconciles from signed bundles carried across the gap; everything self-hosted (FDB, Redpanda, MinIO, ClickHouse, SpiceDB, Zitadel); Biscuit’s offline verification and Iceberg’s open format are load-bearing here.
- Edge — a thin slice: DuckDB + a local FDB + a cache, syncing to the nearest site when connected.
The single reason this works from one codebase is that store and mode are choices behind the verbs — userland never learns which engine answered, so the physical topology is swappable because the logical one is closed.
6. The stack at a glance
- Language: Rust (seam, data plane, WASM host) · TypeScript (SDKs, UI) · JVM (Spark only)
- Contract/SDK: Protobuf + Connect (base) · Conjure-style IR + custom projector (reality SDKs)
- Operational: FoundationDB + Record Layer · Quickwit · H3/S2 geo (on-ramp: Postgres + Citus + PostGIS)
- Log: Redpanda (tiered → Iceberg)
- Analytical: Iceberg + Nessie on MinIO/S3 · Spark-on-K8s · Flink · ClickHouse · Trino · DuckDB
- Compute: Wasmtime (Component Model) · Firecracker · Temporal
- Boundary: Egress proxy · Debezium
- Authz: SpiceDB · Cedar · Biscuit
- Identity: Zitadel (OIDC) · SPIFFE/SPIRE
- UI: React + generated SDK
- Deploy: Kubernetes · Argo/Flux + Apollo-style hub/spoke · Sigstore-signed bundles · Rubix-style hardening
- Observability: OpenTelemetry → ClickHouse → Grafana
7. Forks and divergences
Deliberate departures from Palantir, and the choices still genuinely open.
Chosen divergences.
- Rust over Java at the seam and data plane — security chokepoint + WASM-host nativeness beat JVM ecosystem maturity here.
- Open Iceberg lakehouse over closed datasets — air-gap portability and engine-agnosticism are sharper requirements for us than for a closed platform.
- Nessie catalog branching as a first-class simulation primitive — Palantir doesn’t foreground branch-of-reality the way our design does.
Open forks.
- FoundationDB vs Postgres+Citus for the operational substrate. FDB is the honest AtlasDB and the scale answer; Postgres is the fast on-ramp with better geo. The seam makes it a later swap, but picking the starting substrate shapes the first year of work.
- Redpanda vs Kafka vs NATS JetStream for the spine — throughput and tiered storage vs operational weight vs simplicity.
- WASM vs microVM as the default sandbox. WASM is lighter and more capability-clean; microVMs are more compatible with arbitrary code. The split proposed here (WASM default, microVM for drivers/heavy) is a hypothesis, not a settled line.
- How much geo lives in the kernel index vs a PostGIS read-model. H3/S2 cells serve point and proximity as pure mechanism; complex OGC operations may force a specialized read-model, which risks pulling geo semantics toward the kernel.
- The three design-doc forks still stand upstream of all of this: the Assert→Act promotion boundary, the atomicity boundary (single-aggregate outbox vs cross-aggregate saga over the log), and the trait-kinds closed set (whether
graph-edgecovers telecom, roads, and IT dependency graphs, or needs splitting).