A 'structured' file kind beside text and blob #44

New issue

Closed

opened 2026-09-21 18:03:44 +01:00 by cruelacid · 1 comment

cruelacid commented

2026-09-21 18:03:44 +01:00

Owner

Part of #37. Prerequisite for the rest of Phase 17.

The finding this rests on

The server needs no change. packages/server/src/doc-store.ts states it in
its own header — "Nothing in this file interprets payload" — and the code
matches: appendUpdate, getUpdatesSince and putSnapshot treat the payload as
a BLOB and the name as opaque past the folder-id prefix. The only Y.Doc in
ws-server.ts is an empty shell for awareness; there is no Y.applyUpdate
outside tests.

This is not a hopeful reading. The folder listing is already a Y.Map
(file-sync.ts:109, ydoc.getMap('files')) travelling the identical encrypted
pipeline. A Map-shaped document is a shape the server already carries in
production.

What we do today

packages/plugin/src/blob-policy.ts:28 returns exactly three kinds, and only
.md is text:

if (path.endsWith('.md')) return 'text';
return 'blob';

The file's own comment explains why .canvas and .json are blobs — JSON merged
character-wise under concurrent edit produces something syntactically invalid
that looks fine until opened. That reasoning is sound and this issue does not
overturn it: the answer is not to treat canvas as text, but to add a third
kind that merges structurally.

What changes

A structured member of FileKind.
structured-sync.ts beside content-sync.ts, binding a file to Y.Map /
Y.Array rather than Y.Text, serialising to disk on change.
A third root in the __meta__ listing document, following the precedent Phase 8
set with Y.Map('blobs'): older clients never call getMap on it, so they
replicate entries faithfully and cannot act on them.
Dispatch in vault-watcher.ts and file-sync.ts.

Migration

.canvas files already synced are blobs. They must convert. Cheap while the user
base is small; it will not get cheaper.

Risk

risk:none at the server. Client-only.

Verification

Falsify the load-bearing claim: sync a structured document end to end against
an unmodified server. If that needs a server change, this issue's risk rating
is wrong and Phase 17's sequencing changes.

Part of #37. **Prerequisite for the rest of Phase 17.** ## The finding this rests on **The server needs no change.** `packages/server/src/doc-store.ts` states it in its own header — *"Nothing in this file interprets `payload`"* — and the code matches: `appendUpdate`, `getUpdatesSince` and `putSnapshot` treat the payload as a BLOB and the name as opaque past the folder-id prefix. The only `Y.Doc` in `ws-server.ts` is an empty shell for awareness; there is no `Y.applyUpdate` outside tests. This is not a hopeful reading. The folder listing is **already** a `Y.Map` (`file-sync.ts:109`, `ydoc.getMap('files')`) travelling the identical encrypted pipeline. A Map-shaped document is a shape the server already carries in production. ## What we do today `packages/plugin/src/blob-policy.ts:28` returns exactly three kinds, and only `.md` is text: ```js if (path.endsWith('.md')) return 'text'; return 'blob'; ``` The file's own comment explains why `.canvas` and `.json` are blobs — JSON merged character-wise under concurrent edit produces something syntactically invalid that looks fine until opened. That reasoning is sound and this issue does not overturn it: the answer is not to treat canvas as text, but to add a **third kind** that merges structurally. ## What changes - A `structured` member of `FileKind`. - `structured-sync.ts` beside `content-sync.ts`, binding a file to `Y.Map` / `Y.Array` rather than `Y.Text`, serialising to disk on change. - A third root in the `__meta__` listing document, following the precedent Phase 8 set with `Y.Map('blobs')`: older clients never call `getMap` on it, so they replicate entries faithfully and cannot act on them. - Dispatch in `vault-watcher.ts` and `file-sync.ts`. ## Migration `.canvas` files already synced are blobs. They must convert. Cheap while the user base is small; it will not get cheaper. ## Risk `risk:none` at the server. Client-only. ## Verification **Falsify the load-bearing claim**: sync a structured document end to end against an **unmodified** server. If that needs a server change, this issue's risk rating is wrong and Phase 17's sequencing changes.

cruelacid added this to the Phase 17 - Structured documents milestone

2026-09-21 18:03:44 +01:00

cruelacid added the

labels

2026-09-21 18:03:44 +01:00

cruelacid referenced this issue

2026-09-21 18:08:51 +01:00

Relay feature parity, with end-to-end encryption (tracking) #37