Nectenda/nectenda

Watch

Fork 0

NEC-21: A 'structured' file kind beside text and blob #214

Merged

nectenda-agent merged 5 commits from worktree-nec-21-a-structured-file-kind-beside-text-and-bl into main

2026-09-29 18:35:53 +01:00

nectenda-agent commented

2026-09-29 18:22:46 +01:00

Collaborator

Task: NEC-21 — https://projectron.nerchure.com/tasks/21

Adds a third kind of synced file beside notes (text) and attachments: 'structured', merged key by key through Y.Map/Y.Array in its own document and written back over disk. This is the machinery only — no format is registered, so nothing a user has changes kind until the canvas codec lands (NEC-22).

What it adds:

A codec contract (structured-formats.ts), StructuredSync (structured-sync.ts), a third listing root structured that older clients neither read nor act on, and routing in FileSync and VaultWatcher.
Five data-safety rules, SAFE-A13 to A17, each defended by tests and mutation-checked both ways: unreadable files are never read as empty and are backed up before being replaced; a write lost under a concurrent edit is kept as a conflict copy (judged by the winning entry's Yjs origin, so it holds for reconnect deltas and snapshots); no rewrite loops; documents from a newer version are left alone; migrating from attachment keeps the attachment entry, and an older client's later version is kept beside the file.
The load-bearing claim is falsified against the unmodified server (packages/server/src/structured-doc-relay.test.ts): live relay, catch-up, and a newcomer served from a compacted snapshot all pass with no server change.
The first-sync backup and conflict-copy naming move to local-backup.ts, shared by both engines.

Two review rounds found four silent-loss bugs in earlier drafts; all are fixed with tests, recorded in the spec. One case is left open by the user's decision: an edit that had reached the server, inside a record deleted by a vault that had not yet seen it. Each codec owns that deletion policy (NEC-22), documented in sync-limitations and the codec contract.

Spec: docs/changes/NEC-21-structured-file-kind/spec.md

Changelog

NONE

Task: NEC-21 — https://projectron.nerchure.com/tasks/21 Adds a third kind of synced file beside notes (text) and attachments: 'structured', merged key by key through Y.Map/Y.Array in its own document and written back over disk. This is the machinery only — no format is registered, so nothing a user has changes kind until the canvas codec lands (NEC-22). What it adds: - A codec contract (structured-formats.ts), StructuredSync (structured-sync.ts), a third listing root `structured` that older clients neither read nor act on, and routing in FileSync and VaultWatcher. - Five data-safety rules, SAFE-A13 to A17, each defended by tests and mutation-checked both ways: unreadable files are never read as empty and are backed up before being replaced; a write lost under a concurrent edit is kept as a conflict copy (judged by the winning entry's Yjs origin, so it holds for reconnect deltas and snapshots); no rewrite loops; documents from a newer version are left alone; migrating from attachment keeps the attachment entry, and an older client's later version is kept beside the file. - The load-bearing claim is falsified against the unmodified server (packages/server/src/structured-doc-relay.test.ts): live relay, catch-up, and a newcomer served from a compacted snapshot all pass with no server change. - The first-sync backup and conflict-copy naming move to local-backup.ts, shared by both engines. Two review rounds found four silent-loss bugs in earlier drafts; all are fixed with tests, recorded in the spec. One case is left open by the user's decision: an edit that had reached the server, inside a record deleted by a vault that had not yet seen it. Each codec owns that deletion policy (NEC-22), documented in sync-limitations and the codec contract. Spec: `docs/changes/NEC-21-structured-file-kind/spec.md` ## Changelog NONE

nectenda-agent added the

area:plugin

label

2026-09-29 18:22:46 +01:00

nectenda-agent added 5 commits

2026-09-29 18:22:46 +01:00

Add a structured file kind beside text and attachments f8661a2924

Files merged key by key through Y.Map/Y.Array in their own document, written
back over disk. The machinery only: no format is registered, so nothing a user
has changes kind until a codec lands (the canvas one is NEC-22).

The piece Yjs does not give a structured document is loss reporting. Two
writes to one map key keep one value, and a map deleted under a concurrent edit
takes the edit, both silently. The provider now shows StructuredSync each
remote update's own delete set before it lands; an entry this vault wrote that
vanishes without being in it was lost unseen, and is kept as a conflict copy.

A third listing root, `structured`, which older clients never read. Moving a
path from attachment to structured keeps its attachment entry, and an older
client's later attachment version is kept beside the file, once per version.

The first-sync backup and the conflict-copy naming move to local-backup.ts,
shared by both engines; the conflict-copy path tests now exercise the real
function rather than a copy of it written inside the test.

Task: NEC-21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011J6bDvvjhm2suCNMWUtvSx

Specify structured files: SAFE-A13 to A17, and what they leave open d23919d331

The five rules a document of maps needs and text does not, each naming the
tests that defend it: unreadable files, concurrent writes lost silently,
rewrite loops, documents from a newer version, and members who have not
updated. ADR-0011 records the opportunity it left open as taken up;
sync-limitations gains the two cases that are mitigated rather than closed.

Task: NEC-21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011J6bDvvjhm2suCNMWUtvSx

Judge a lost structured write by its origin, not the update's delete set 9743a58b5a

Review found the delete-set rule blind to the case it exists for. On
reconnect the provider sends one merged delta, which carries the sender's
whole delete set - including the other vault's entry, discarded unseen by the
sender's own merge. The online vault that lost to an offline winner kept no
copy, and its value existed nowhere.

A same-key loss is now decided by what the winning entry was written on top
of, which travels with the entry however it arrives. An entry removed with a
deleted container is kept while it was not yet known to be on the server,
tracked from the provider's own account and persisted. The harness now
reconnects as the provider does - catch up, then one merged delta - which is
what hid the bug.

Also from review: an edit read before the first write is no longer diffed
against the document (it reverted remote changes); a vault receiving a rename
no longer fills the new document too; a migration download keeps its carried
hash when the watcher lists the file first; a rename's echo no longer purges
the old document; the disk is re-read after a backup before writing; and a
format with no codec gets no placeholder.

Task: NEC-21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011J6bDvvjhm2suCNMWUtvSx

Restate SAFE-A14 as built, and record the case left open 06168965fd

The rule now says how a loss is judged and why the update's delete set must
not be the signal. An edit inside a record deleted by a vault that was offline,
after the edit had reached the server, cannot be told from a deliberate delete;
decided with the user to leave it to each codec's deletion policy (NEC-22),
recorded in sync-limitations and the codec contract.

Task: NEC-21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011J6bDvvjhm2suCNMWUtvSx

Keep a save made during a structured file's first-write backup

Release note / release-note (pull_request) Successful in 12s

Details

CI / build (pull_request) Successful in 4m50s

Details

CI / e2e (pull_request) Successful in 5m23s

Details

CI / promote (pull_request) Has been skipped

Details

Deploy site / deploy (push) Successful in 54s

Details

CI / e2e (push) Successful in 5m43s

Details

CI / build (push) Successful in 5m56s

Details

CI / promote (push) Successful in 33s

Details

05eaf91ee0

Second review round. Two of the first round's fixes met: the retried write is
no longer the first, and the no-base fix leaves no agreed text, so the one
remaining backup branch - which required agreed text - kept nothing, and the
save was overwritten. A disk that differs from the last agreed text is now
always kept, with no agreed text counting as different.

The harness now pushes a reconnect delta on the provider's own test (over two
bytes); it skipped deltas that only delete, so an offline deletion never left
the vault. And the open A14 case is any deleter that had not yet seen the
edit, typically but not only an offline one; the docs said "offline".

Task: NEC-21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011J6bDvvjhm2suCNMWUtvSx

nectenda-agent merged commit 05eaf91ee0 into main

2026-09-29 18:35:53 +01:00

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Nectenda/nectenda!214

No description provided.

Rows
Columns