LTAP log-sourced storage spike

Concluded 2026-07-05 · code: crates/experiments/vinxi-lakebase-spike (36 tests) · design: docs/superpowers/specs/2026-07-05-ltap-log-sourced-storage-design.md · pipeline semantics: Data Plane

The question

Databricks’ Lakebase/LTAP thesis — unify transactional and analytical processing at the storage layer (one durable columnar copy, engines specialized, recent tail merged at read time) — maps almost line-for-line onto Vinxi’s data-plane design, one altitude up: the statement log plays SafeKeeper, the materializer plays PageServer, the seam plays Unity Catalog. The spike asked: does the log-sourced version actually deliver the LTAP properties — cheap object storage, zero-ETL fresh analytics, performance isolation, branch-as-pointer scenarios — and where does it break?

How it was run (and why the arc matters)

The spike was built test-first against the real kernel seams (TruthStatement, StatementLogPort, BlobStorePort), benchmarked at 1M/5M/10M entities across {local fs, MinIO S3} stores and {memory, Postgres-fsync} commit tiers — and then three independent adversarial audits invalidated the first verdict. The initial “every gate met” claim was retracted: the isolation number was true by construction, the freshness path was unbuilt, the commit gate had failed as pre-registered, the 10M run was memory-thrashing, and the test oracle was partially circular. A same-day remediation round fixed every code blocker test-first and built the missing experiments. Everything below is post-audit evidence only; the pre-audit headline numbers should not be cited.

What is demonstrated

Property	Evidence
Open single copy (zero-ETL)	DuckDB — an external engine, no Vinxi code — queried the kernel’s own Parquet segment bytes directly: 30M rows aggregated in 0.17s.
Object-storage economics	26 B/statement (zstd Parquet) on the spike’s low-entropy corpus (realistic-entropy estimate 60–150 B — still cheap); segment-grain PUTs on a real S3 API amortize to ~zero per statement.
Analytical freshness (the LTAP tail-merge)	With time-based sealing + an analytical scan that unions sealed segments with the un-sealed commit tail, write→analytically-visible p99 ≈ the 250 ms poll interval — at 2,500 stmts/s and at 10 stmts/s (time-driven, so quiet cells stay fresh). Without the tail-merge, staleness = the seal cadence.
Performance isolation, under real contention	One Postgres served single-Act durable commits and SQL point reads while DataFusion scanned 7.4B rows in 30s (247M rows/s) on the same host: commit p99 degraded 1.02×, reads 0.86×.
Sustained pilot-shape load	500 stmts/s of paced single-Act Postgres commits for 5 minutes with sealing + checkpointing live: no latency or memory drift.
Branch = pointer, scenarios at scale	Fork is O(1) by construction; 1,000 fork + cable-cut-simulation + diff scenarios in ~1.1–1.3s at 5–10M entities; merge planning linear in delta with seq-based conflict detection (ABA-safe, links included).
Set queries on branches	Composed answers (indexed base @ cut ⊕ delta ⊕ cut-correction ring) verified equal to a naive oracle under two-sided drift, including branch-created entities; policy predicates evaluate identically on base and branch rows.
Branch lifecycle economics (rungs 1–3)	Composition cost is O(churn-since-fork): the 200 ms list gate breaks at churn ≈ 10⁶ on laptop hardware. Promoting the branch to its own projection (2.65s / 231 MB at 1.05M entities, answers verified equal) collapses ring-dependent queries to microseconds; ring retention then reclaims the memory, and stale cuts are refused, never silently wrong. Promotion must trigger on churn-since-fork, not delta size.
Recovery discipline	Restart from durable state only (manifest per seal, checkpoint + bounded tail replay); replay-idempotency by statement id; as-of below a restored checkpoint’s horizon is refused rather than answered wrongly.

What is not demonstrated (open, honestly)

Single-Act commit p99 gate (≤10 ms): failed at ~23 ms — present at baseline too, i.e. a dockerized-Postgres fsync tail, not contention. Needs a target-hardware run before the Lean commit tier is declared inside the gate.
Server-path point reads (the spike’s 10–130 ns reads are an in-process projection; the 50 ms gate was never exercised through a transport).
10M entities on adequate memory (the 24 GB laptop compressed/thrashed; treat 5M as the largest honest in-memory scale measured).
Realistic-entropy storage numbers; segment-size sweep; pagination for million-row answers (surfaced by the branch experiment: answer size, not mechanism, dominates past ~10⁶ rows).

Lessons beyond the numbers

Adversarial QA earned its keep: every one of the five headline claims it killed had looked plausible in good faith. Pre-registered gates only work if the outcomes table is actually filled against the registered shapes.
Sub-microsecond benchmark numbers on macOS are timer-quantum artifacts (41.67 ns tick); batch-time or don’t report.
The cut-correction ring is not the recent-write overlay — retention contracts differ by orders of magnitude (seconds vs months). They share a shape, not a lifecycle.
Kernel-relevant side-finding: vinxi-adapter-memory’s statement dedup is a linear scan per append — O(N²) at volume. Fine for unit tests; disqualifying for load work (tracked in the repo backlog).
Ecosystem coupling: DataFusion pins its own arrow/parquet/object_store versions — always consume its re-exports and match its object_store, or store registration type-mismatches.

Verdict

The log-sourced LTAP architecture survives adversarial scrutiny on every structural claim: one log fanned to row and columnar projections with nothing to reconcile, fresh zero-ETL analytics via tail-merge, isolation without a second truth, and a branch lifecycle whose costs are now measured rather than asserted. The remaining risk is quantitative (fsync tail on target hardware, server-path latency), not architectural.