Plugin authoring›Hosted & In-Memory Builds

Hosted & In-Memory Builds

refrakt's build pipeline can run without a local filesystem. The page corpus is a plain-object ContentTree, the core packages (transform, runes, svelte, behaviors) operate on strings and trees, and every ad-hoc file read — snippet sources, sandbox src directories, namespaced fileRoots partials, the plan plugin's scan — goes through one injectable seam: ProjectFiles.

This page is for two audiences:

  • Adapter / host authors building a renderer that feeds refrakt content from somewhere other than disk (a GitHub fetch, a database, an in-browser authoring sandbox).
  • Plugin authors whose pipeline hooks read files, so they read through the provider instead of node:fs and work in both fs and hosted modes.

If you only write self-hosted sites, you never touch this — the CLI and the framework adapters construct the provider for you.

The ProjectFiles contract

One synchronous interface, defined in @refrakt-md/types:

interface ProjectFiles {
  read(path: string): string | null;
  list(path: string): string[];
  exists(path: string): boolean;
}

The rules:

  • Keys, not paths. Every argument is a normalized POSIX, project-root-relative key — never absolute, never containing .. after normalization.
  • Containment is part of the contract. Implementations reject absolute paths and any path that escapes the root, treating a violation exactly like an absent file: read returns null, list returns [], exists returns false. Callers never re-implement path safety — a src="../../etc/passwd" is denied by the provider, not by each consumer.
  • list is readdir-shaped. It returns the immediate child entry names (basenames) directly under a directory key; the empty string lists the project root.
  • Synchronous. Markdoc's transform is synchronous and the sandbox rune reads its src at transform time, so the provider boundary sits in front of the pipeline (at materialization time), not inside it. There is deliberately no async reader threaded through the build.

Stock providers

The interface lives in the package root; the providers are a Node-only subpath so a browser bundle that only imports @refrakt-md/types never pulls in node:fs:

import {
  fsProjectFiles,
  memoryProjectFiles,
  recordingProjectFiles,
} from '@refrakt-md/types/project-files';
  • fsProjectFiles(rootDir) — the OSS/CLI default. Wraps node:fs, anchored at the project root, and enforces containment (absolute reject, traversal reject, symlink-escape reject via realpath). This is what loadContent constructs for a self-hosted build.
  • memoryProjectFiles(files: Map<string, string>) — the hosted/editor provider. Backed by a plain map of normalized keys; traversal is structurally impossible. list and exists are derived from key prefixes, and reads pass through to the live map, so a warm host can patch a single key between builds without reconstructing the provider.
  • recordingProjectFiles(inner, onRead) — a thin pass-through that reports every { op, key } access. Pure instrumentation, no behaviour change. It is the per-page read-set capture point that incremental rebuild builds on — see below.

When a consumer should read through it

A plugin pipeline hook that reads a project file should take the provider from the context rather than importing an fs-bound reader:

  • Preprocess hooks receive a ProjectFiles on the PreprocessContext (ctx.sandbox). This is the same provider the transform-time sandbox reads through — snippet, expand, and file-ref all resolve through it.
  • configure hooks receive projectFiles on PluginConfigureOptions. A plugin that scans the project at configure time (the way @refrakt-md/plan scans plan.dir) walks it through the provider so a hosted build stays fs-free, falling back to direct fs only when no provider is wired.

Reading through the provider is what makes your plugin work unchanged in a hosted deployment — and what makes its reads recordable for future incremental builds.

Fetch-then-build: the materialization pattern

For a remote source, the provider is materialized before the build, which keeps the whole pipeline synchronous. The hosted product (a GitHub app that renders a tenant's repo on push) follows this recipe:

  1. Fetch the repo tarball at the commit SHA. One API call, an atomic snapshot — no torn reads across a push, and rate-limit-friendly versus per-file contents calls.
  2. Unpack to a Map<string, string> of normalized keys. Binary assets are the host's concern — served from its own storage/CDN, not the pipeline.
  3. Assemble the ContentTree from the map by key-prefix filtering (the content directory). ContentTree.fromContentMap does exactly this.
  4. Build with memoryProjectFiles(map) and loadContentFromTree.
import { ContentTree, loadContentFromTree } from '@refrakt-md/content';
import { memoryProjectFiles } from '@refrakt-md/types/project-files';

// `files` is the materialized repo: normalized POSIX keys → file contents.
const files: Map<string, string> = await materializeTarball(repo, sha);

const tree = ContentTree.fromContentMap(files, { contentDir: 'site/content' });

const site = await loadContentFromTree(tree, {
  projectFiles: memoryProjectFiles(files),
  projectRoot: '/',               // the key anchor for the provider
  sandboxExamplesDir: 'examples', // project-relative key for sandbox `src`
  gitTimestamps,                  // optional: from the source's history API
  // …icons, plugins, xrefPatterns, repoUrl, etc. — the host owns config.
});

loadContentFromTree accepts projectFiles and gitTimestamps (a remote host supplies timestamps from its source's history API; omit for frontmatter-only dates). The GitHub specifics — App auth, the tarball endpoint, webhook handling — belong to the hosted product; steps 3–4 above are the entire integration surface: no fs, no traversal surface, no async plumbing.

One map, the tree derived from it

The materialized Map is the single surface the build reads from. The ContentTree is assembled from it (fromContentMap filters by the content-dir prefix: _layout.md becomes a directory's layout, _partials/… at the content root become partials, other .md files become pages), and memoryProjectFiles(map) serves every ad-hoc read — snippet, sandbox src, fileRoots, the plan scan — from that same map.

There is one materialization surface, not two. That is what keeps a refresh coherent: you never have a ContentTree built from one snapshot and a file provider reading from another.

Warm instances & incremental fetch

Nothing in the contract requires re-fetching the whole tarball per change. A long-lived host can keep the map resident and, on a single-file webhook, fetch just that key and re-run loadContentFromTree over the updated map — memoryProjectFiles reads through to the live map, so no provider reconstruction is needed.

The tradeoff is consistency:

  • Tarball-at-SHA is an atomic snapshot — every key is from the same commit.
  • Single-key patch from a "file changed" event is cheaper to fetch but can straddle two commits if the underlying push touched siblings (renames, multi-file commits).

Incremental fetch is a host-side materialization choice; the synchronous ProjectFiles boundary is indifferent to it. Note this buys cheaper fetch, not cheaper build — recompute stays whole-corpus regardless (see below).

Towards incremental rebuild

Every build today is a full build: loadContentFromTree re-parses and re-transforms every page, and the cross-page register → aggregate → post-process pipeline re-runs over the whole corpus. A provider refresh changes which bytes are read, never how much of the pipeline re-runs.

That is deliberate. The seam is a prerequisite for per-file incremental rebuild, not a delivery of it: by centralizing I/O through one choke point, wrapping it with recordingProjectFiles(inner, onRead) captures a page's dependency read-set for free. The target architecture is dependency-tracked invalidation with the registry as the firewall, and this recordable seam is what makes it possible. The rebuild engine itself does not exist yet.

Checklist for plugin authors

  • Read project files through the context's provider (ctx.sandbox in preprocess, projectFiles in configure), not node:fs.
  • Treat a null / [] / false result as "absent or denied" — don't try to distinguish containment failures; the provider owns that decision.
  • Keep an fs fallback only where a path can legitimately live outside the project root (e.g. plugin partial directories in node_modules).
  • If your hook needs whole-corpus data, that's the cross-page pipeline — ProjectFiles is for per-file reads.

Path rules for file-reading runes

snippet (path), data (src) and sandbox (src) read through the provider, so they inherit its containment. Paths resolve against the directory holding refrakt.config.json (a sandbox src against the examples directory inside it), and every one of these is refused:

PatternOutcome
path="/etc/passwd"Rejected: absolute paths are not allowed.
path="../../etc/passwd"Rejected: the path leaves the project root after normalization.
path="link-to-outside" (a symlink pointing outside the root)Rejected: fs.realpath is checked too.
path="src/" (a directory)Rejected: it must be a regular file.
path="missing.ts"Rejected: the build error names the resolved path and the page that referenced it.

That makes these runes safe on a site that accepts untrusted author content, such as a hosted authoring product or an external editor: the most a malicious author can do is quote a file inside the project root. data reports a refused path as an error callout in place of its table; sandbox shows an error in place of the preview.

Background: ·

ProjectFiles seam — virtual project filesystem for hosted and in-browser builds

SPEC-113shipped

ProjectFiles seam — virtual project filesystem for hosted and in-browser builds

The hosted refrakt product (a GitHub app that renders a tenant's repo on push) never touches a local filesystem — content arrives from the GitHub API. The pipeline is already most of the way there: loadContentFromTree() runs the full transform + cross-page pipeline from a caller-assembled ContentTree, and the core packages (transform, runes, svelte, behaviors) are fs-free. What remains is a handful of ad-hoc filesystem seams at the edges — sandbox example reads, the snippet/expand/file-ref readers, fileRoots scanning, the plan plugin's scan — each with its own injection story (or none). This spec consolidates them into one injectable ProjectFiles interface with filesystem and in-memory providers, defines the fetch-then-build materialization contract for remote sources, and closes the sandbox src path-traversal gap as part of the migration.

contentrunespipelinehostedsecurityarchitecture

Overview

What is already virtual (verified against the code)

  • loadContentFromTree(tree, options) (packages/content/src/site.ts) — the page corpus, partials, and layouts come from a plain-object ContentTree; the full transform + register/aggregate/post-process pipeline runs. Its docstring already names the use case: "a GitHub fetch, a database, an in-memory authoring sandbox."
  • Core packages are pure. transform/runes/svelte/behaviors operate on strings and trees; node:fs imports concentrate in content, the adapters, the editor's server side, and the CLI — none of which a hosted renderer needs (the host owns config parsing and brings its own app shell).

The remaining fs seams (the inventory this spec consolidates)

  1. Sandbox example reads — injectable today (SandboxHooks — read/list/ exists — on both the transform variables (__sandboxReadFile family) and the preprocess context), defaulting to null hooks in tree mode. The seam exists; it is one of four shapes.
  2. snippet / expand / file-ref reads — snippet-pipeline.ts, expand-pipeline.ts, and file-ref-resolve.ts import readSnippetFile / readWholeSandboxedFile from packages/runes/src/lib/read-file.ts, which calls node:fs directly — even though the preprocess context also exposes the injectable sandbox hook family (packages/types/src/pipeline.ts). The readers bypass the seam. This is the main gap.
  3. fileRoots scanning (packages/content/src/file-roots.ts) — namespace partial roots are scanned with direct fs at load time.
  4. Plan plugin configure scan — @refrakt-md/plan scans the plan directory with direct fs in its configure hook.
  5. Timestamps — processContentTree accepts an injectable gitTimestamps map, but LoadContentFromTreeOptions doesn't expose it (one-line plumb; a remote host supplies timestamps from its source's history API).
  6. Config (refract-loader.ts) — direct fs, deliberately out of scope: a hosted product owns config resolution (and its plugin allowlisting) itself.

The wrong abstraction, rejected up front

An async read(path): Promise<string> provider threaded through the pipeline is the tempting shape — and the codebase has already declined it: LoadContentFromTreeOptions.reader?: VirtualReader is documented as accepted but not plumbed. The precise reason matters. Markdoc's transform is synchronous, and the sandbox rune reads its src files at transform time (__sandboxReadFile, consumed inside Markdoc.transform) — that read genuinely cannot await. The preprocess phase, by contrast, is already async: hooks are awaited in processContentTree, so snippet / expand / file-ref / data resolve their files in an async context today. They are synchronous only because their reader (read-file.ts) calls node:fs synchronously — not because the phase forbids awaiting. So the irreducible sync surface is the transform-time sandbox read; the preprocess readers are kept sync for uniformity (one ProjectFiles shape for every consumer), not out of necessity. Either way, making the whole pipeline async-capable is a large refactor with no payoff: the provider boundary belongs in front of the pipeline, at materialization time — not inside it.

Design

1. The ProjectFiles interface

One synchronous interface, replacing the four ad-hoc shapes:

/** Synchronous read access to the project's files, rooted at the project
 *  root. Paths are normalized POSIX, project-root-relative keys — never
 *  absolute, never containing `..` after normalization. */
interface ProjectFiles {
	read(path: string): string | null;
	list(path: string): string[];
	exists(path: string): boolean;
	/** Optional: stat-level data where the provider has it (sizes for
	 *  budget caps; future use). */
}
  • Keys, not paths. The contract is normalized POSIX repo-relative keys. Containment is part of the interface contract: implementations reject absolute paths and any path that escapes the root after normalization — callers never re-implement containment.
  • Lives in @refrakt-md/types (the foundational package), so runes, content, and plugins can all consume it without new dependency edges.

2. Providers

  • fsProjectFiles(rootDir) — the OSS/CLI default. Wraps node:fs with the containment rules read-file.ts already implements for snippets (absolute-path reject, traversal reject, symlink-escape reject) — promoted from snippet-specific code to the provider, so every consumer gets it.
  • memoryProjectFiles(files: Map<string, string>) — the hosted/editor provider. Backed by a plain map of normalized keys; traversal is structurally impossible (lookups are dictionary keys, and the dictionary contains exactly one project). list is derived from key prefixes.

3. Consumer migration

All file-touching consumers read through an injected ProjectFiles:

  • Sandbox examples — the __sandboxReadFile family and the preprocess sandbox hooks become a ProjectFiles (the hook shape is already identical; this is a rename + containment upgrade). The src directory join in packages/runes/src/tags/sandbox.ts (examplesDir + '/' + src — currently unguarded string concat) goes through the provider and inherits containment, closing the path-traversal gap in fs mode.
  • snippet / expand / file-ref — read-file.ts keeps its line-slicing and diagnostics but delegates I/O to the provider instead of node:fs; its containment logic moves into fsProjectFiles.
  • fileRoots — scanned through the provider.
  • Plan plugin — the configure-time scan accepts a provider from the pipeline context (hosted deployments that don't build plan sites can simply not provide plan content).
  • loadContent (fs mode) constructs fsProjectFiles(projectRoot) and threads it — existing behaviour, now with uniform containment. loadContentFromTree accepts projectFiles?: ProjectFiles and exposes gitTimestamps (the missing plumb).

4. Fetch-then-build — the materialization contract for remote sources

For the hosted product (and any remote source), the provider is materialized before the build, keeping the pipeline synchronous:

  1. Webhook push → fetch the repo tarball at the commit SHA — one API call, an atomic snapshot (no torn reads across a push), rate-limit-friendly versus per-file contents calls.
  2. Unpack to a Map<string, string> of normalized keys (binary assets are the host's concern — served from its storage/CDN, not the pipeline's).
  3. Assemble the ContentTree from the map (content dir traversal is key-prefix filtering); supply timestamps from the source's history API if desired.
  4. memoryProjectFiles(map) + loadContentFromTree(tree, { projectFiles, … }).

The GitHub specifics (App auth, tarball endpoint, webhook handling) belong to the hosted product's repo; this spec's deliverable is that steps 3–4 are the entire integration surface — no fs, no traversal surface, no async plumbing.

One map, the tree derived from it. The materialized Map is the single surface the build reads from: the ContentTree is assembled from it by key-prefix filtering (the content dir), and memoryProjectFiles(map) serves every ad-hoc read (snippet / data / sandbox src / fileRoots) from the same map. There is one materialization surface, not two — which is what keeps an incremental refresh coherent.

Warm instances and incremental fetch. Nothing in the contract requires re-fetching the whole tarball per change. A long-lived host may keep the map resident and, on a single-file webhook, fetch just that key and re-run loadContentFromTree over the updated map — the provider may back a mutable/refreshable map. The tradeoff is consistency: tarball-at-SHA is an atomic snapshot, whereas patching one key from a "file changed" event can straddle two commits if the underlying push touched siblings (renames, multi-file commits). Incremental fetch is a host-side materialization choice; the sync ProjectFiles boundary is indifferent to it. Note this buys cheaper fetch, not cheaper build — recompute stays whole-corpus regardless (see Non-goals).

5. Non-goals

  • No async pipeline. VirtualReader stays reserved/unused; if a future consumer genuinely needs on-demand async resolution, that's its own spec.
  • No config abstraction. refract-loader stays fs-based for OSS; hosts own config.
  • No adapter changes. The sveltekit/eleventy/next adapters remain fs-bound — they serve the self-hosted path; a hosted renderer consumes @refrakt-md/content + the renderer directly.
  • No incremental rebuild. Recompute is whole-corpus today — the cross-page register → aggregate → post-process pipeline and the per-page parse/transform loop re-run over the entire site on every build, and this spec keeps it that way. A provider refresh changes which bytes are read, never how much of the pipeline re-runs. Per-file incremental rebuild (dependency tracking
    • invalidation) is a separate, future spec; the ProjectFiles seam is a prerequisite for it (it centralizes I/O so reads become recordable) but does not deliver it. See ADR-025 for the target architecture and the one piece of groundwork this seam should carry (a recordable read wrapper).

Acceptance Criteria

  • ProjectFiles (read/list/exists over normalized POSIX project-relative keys, containment as interface contract) is defined in @refrakt-md/types, with fsProjectFiles (containment: absolute reject, traversal reject, symlink-escape reject — promoted from read-file.ts) and memoryProjectFiles (map-backed) providers.
  • Sandbox example resolution (transform __sandboxReadFile family + preprocess hooks) consumes ProjectFiles; the sandbox src directory join inherits containment — {% sandbox src="../…" %} resolves to the in-band error message, with a regression test, in both providers.
  • snippet / expand / file-ref readers delegate I/O to the provider (line-slicing and diagnostics unchanged); their existing sandbox tests pass against both providers.
  • fileRoots scanning and the plan plugin's configure scan read through the provider.
  • loadContentFromTree accepts projectFiles and gitTimestamps; a full site build from a pure in-memory map (no fs access) is covered by an integration test — pages, partials, layouts, a src-directory sandbox, and a snippet all resolve.
  • loadContent (fs mode) behaviour is unchanged for well-formed projects (existing test suites green); the only behavioural change is containment on previously unguarded paths.
  • Docs: plugin-authoring / adapter docs describe the ProjectFiles contract and the fetch-then-build materialization pattern for remote hosts.

Work breakdown (provisional)

  1. Interface + providers — ProjectFiles in types; fsProjectFiles (containment promoted from read-file.ts) + memoryProjectFiles; unit tests incl. traversal/symlink cases.
  2. Sandbox consumer migration — hook family → provider; src join containment + regression test.
  3. snippet/expand/file-ref migration — read-file.ts delegates I/O; tests against both providers.
  4. fileRoots + plan scan + loadContentFromTree plumbs (projectFiles, gitTimestamps); the in-memory full-build integration test.
  5. Docs — contract + materialization pattern.

References

  • Virtual entry point + reserved async reader: packages/content/src/site.ts (loadContentFromTree, VirtualReader).
  • Existing seams: sandbox hooks (packages/content/src/site.ts, packages/types/src/pipeline.ts preprocess context); snippet containment model packages/runes/src/lib/read-file.ts; packages/content/src/file-roots.ts.
  • The unguarded join this fixes: packages/runes/src/tags/sandbox.ts (examplesDir + '/' + src).
  • Security context: SecurityPolicy tiers (packages/types/src/security.ts) govern client-side sandbox content; this spec governs build-time file access — the two are complementary halves of the hosted threat model.
  • Related: SPEC-093 (data-bound sandbox — build-time resolution that stays fs-free); SPEC-101 (the sandbox-backdrop work that surfaced the hosted questions).

Incremental rebuild: dependency-tracked invalidation with the registry as firewall

ADR-025proposed

Incremental rebuild: dependency-tracked invalidation with the registry as firewall

pipelineincrementalperformancearchitecturehostedregistry

Context

Today every build is a full build. processContentTree re-parses and re-transforms every page on every call (packages/content/src/site.ts, the for (const page of tree.pages()) loop), and runPipeline (packages/content/src/pipeline.ts) re-runs all three cross-page phases — register → aggregate → post-process — over the entire page array. There is no memoization, no dirty-tracking, and no dependency graph anywhere in the pipeline.

This is fine for CLI builds of a docs site, and SPEC-113 deliberately keeps it that way. But the hosted product (a GitHub app that re-renders a tenant's repo on push) makes the cost visible: a one-character edit to one file triggers a whole-site recompute. SPEC-113's warm-map / incremental-fetch path makes re-reading cheap, but it explicitly does not make re-building cheap — recompute stays whole-corpus. At some site size or rebuild-latency threshold we will want per-file incremental rebuild.

We are not building that engine now. This ADR records how to think about it so that the SPEC-113-era work is shaped to enable it cheaply, rather than forcing an expensive retrofit later.

Local dev HMR is the second beneficiary

The hosted product is not the only consumer. The Vite adapters' content HMR has the same whole-corpus shape today. On any .md save, setupContentHmr (packages/transform/src/content-hmr.ts) calls invalidateSite() — which nulls the loader's single memoized Promise<Site> (packages/content/src/loader.ts; the cache is binary, no per-page granularity) — and sends a blanket full-reload. The next SSR request re-reads the entire content dir, re-parses and re-transforms every page, and re-runs register → aggregate → post-process over the whole corpus. So a single keystroke-save costs a full rebuild; it is the local analog of the hosted single-file-webhook case. An incremental engine therefore pays off twice — hosted rebuild latency and local dev DX — and unlocks a second-order HMR win: knowing the dirty cut lets the adapter send a scoped reload (only the routes whose output actually changed) instead of the blanket full-reload it is forced into today precisely because any edit can, in principle, move any other page's output.

The dependency structure (read from the code)

The pipeline already is a DAG; its phase boundaries are the edges.

  • A page's renderable depends on: its own source bytes; its layout chain (the _layout.md cascade up the directory tree); every partial it references ({% partial %}, incl. namespaced fileRoots); every file it reads via snippet / data / sandbox src; site-wide config / icons / variables; and — the awkward one — for expand / collection, the rendered content of other entities it embeds.
  • The registry is a fold over all pages: each page contributes its entities independently.
  • Aggregated data (pageTree, breadcrumbPaths, headingIndex, …) depends on the whole registry.
  • postProcess(page) depends on the page's renderable plus the slices of aggregated data it consumes.

The key property: the registry is the firewall

When page X changes there are three escalating blast radii, and the registry is what distinguishes them:

  • Tier 0 — content-only, no registry delta (typo, prose edit). Re-transform X, diff the entities X contributes against the prior build. Identical → stop. Blast radius: one page. The common case.
  • Tier 1 — registry changed, aggregate stable. X added / renamed / reordered an entity, but the aggregated outputs are unchanged or move in a slice few pages read. Re-run aggregate, diff its outputs, re-run postProcess only for pages whose consumed slice moved. Blast radius: bounded.
  • Tier 2 — structural (new / deleted / moved page, layout edit). A layout edit invalidates every page under that directory subtree; a new page can shift pageTree / breadcrumbs globally. Blast radius: wide, but scoped by a reverse index.

If a change does not perturb the registry, aggregate and every other page are untouched. That firewall is what makes incrementality tractable.

The one edge below the registry

expand / collection pull other entities' rendered content into a page at preprocess / transform time. So page A's output depends on page B's source, directly — not via the registry summary. A naive "registry is the firewall" model misses this: editing B's body (Tier 0 for B) must still invalidate A. The per-page read-set therefore has to include "entities embedded via expand / collection," not just files on disk.

Options Considered

  1. Dependency-tracked invalidation, registry-as-firewall (chosen as the target architecture). Record each page's read-set, hash outputs, and recompute the minimal dirty cut across the existing phase DAG, using the registry as the coarse firewall and per-slice aggregate edges as the fine one. Matches the pipeline's actual shape; degrades gracefully (a missing edge over-invalidates, never under-invalidates).

  2. Fine-grained query memoization (salsa / rustc-style) from day one. Rejected as the starting point. The theoretically clean model (pure functions keyed by inputs, invalidated by recorded reads) is the right shape but the wrong amount of machinery to introduce now — it would require threading a tracking context through Markdoc transform internals we don't own. We adopt its shape (recorded reads + hashed outputs), not its framework.

  3. Output-level caching only (skip re-emitting unchanged HTML; recompute the pipeline regardless). Rejected as insufficient. Saves downstream render/upload but not the pipeline CPU that actually dominates at scale. Useful as an orthogonal add-on, not as the answer.

  4. Do nothing; rely on whole-corpus recompute + incremental fetch. Rejected as the long-term answer, accepted as the current state. Adequate until rebuild latency hurts — which is exactly the threshold this ADR exists to be ready for.

Decision

Adopt dependency-tracked invalidation with the registry as firewall as the target architecture for incremental rebuild, and defer building the engine. Commit now only to the groundwork that is expensive to retrofit:

  1. Route every read through the ProjectFiles seam (SPEC-113), then make reads recordable. A recordingProjectFiles(inner, onRead) wrapper captures the per-page dependency set for free — this is the load-bearing reason the seam enables incrementality. It is not about async; it is about centralizing I/O so reads can be tracked at all. While I/O is scattered across node:fs in read-file.ts, the sandbox hooks, and file-roots.ts, a page's read-set cannot be captured.

  2. Capture the AST-derived dependencies the recording provider can't see: referenced partials, the layout chain, and expand / collection embed targets. Emit the full per-page read-set as a side-output of a normal build. This gives us the dependency graph as data before anything consumes it.

  3. Make outputs hashable. Post-serialize renderables are already plain {$$mdtype} objects (JSON-hashable); registry entries likewise. A per-page content-hash manifest is the substrate for "did this actually change."

  4. Thread an optional { priorBuild, dirtySet } through processContentTree / runPipeline as an additive signature, with full build as dirtySet = ALL. No behaviour change yet — just the capability seam.

The actual optimizations — the registry-diff firewall (Tier 0/1), then fine-grained aggregate→postProcess slice edges (Tier 3) — land later, in their own spec, on top of data we have been collecting all along.

Rationale

  • The seam pays for itself twice. SPEC-113 already centralizes I/O for hosting and security; making that one choke point recordable is the whole prerequisite for incrementality, at near-zero marginal cost. Retrofitting read-tracking onto scattered node:fs calls later would be far more expensive.
  • The pipeline's existing phases are the dependency graph. We are not inventing structure; we are refusing to recompute nodes whose inputs are unchanged. The registry firewall falls straight out of the register→aggregate boundary that already exists.
  • Graceful degradation. A missing or coarse dependency edge causes over-invalidation (a needless recompute), never under-invalidation (a stale page). So the system is correct from the first coarse cut and only gets faster as edges sharpen.
  • Proven shape. Gatsby's incremental builds are the direct analog — its GraphQL data layer is our registry, its page queries are our postProcess consumers, and it recomputes the minimal set by tracking which pages touch which nodes. The query-memoization literature (salsa / rustc) is the theory underneath; we borrow the shape (recorded reads + hashed outputs + a dirty cut), not the framework.

Consequences

  • Shapes SPEC-113 work, doesn't expand its scope. The only concrete near-term ask is that the seam expose a recording wrapper point and that a build can emit per-page read-sets. SPEC-113's "no incremental rebuild" non-goal links here for the deferred remainder.
  • A future incremental-rebuild spec owns the engine: the dirty-cut algorithm, the registry diff, the reverse indexes (partial→pages, layout-dir→pages, embed-target→pages), and the priorBuild cache format.
  • The expand / collection below-the-registry edge is the known sharp corner; the future spec must treat embed targets as first-class dependencies, not rely on the registry summary alone.
  • Orthogonal output-level caching (skip re-emitting unchanged HTML) can be added independently for the hosted render/upload path without waiting on the engine.