Nectenda/nectenda

Watch

Fork 0

Property-based testing does not reach the merge path: the convergence harness is outstanding #92

New issue

Closed

opened 2026-09-22 17:19:40 +01:00 by cruelacid · 1 comment

cruelacid commented

2026-09-22 17:19:40 +01:00

Owner

Corrected 22 September 2026. This issue was filed claiming we had no
property-based testing at all. That was wrong — it was written against a stale
local checkout. 5a4d1b7 (21 September) added fast-check 4.10.2 to
packages/plugin and packages/shared, with 23 properties across
vault-path.property.test.ts (10), blob-cipher.property.test.ts (9) and
doc-id.property.test.ts (4). It found a real thing immediately: a blob
envelope byte-flip accepted after fifteen runs, shrunk to bytes 7–10 — the
header's chunkSize, absent from the AAD — now documented as CRYPTO-087.
The issue is rescoped below to what that commit explicitly left out.

Why

The library, the harness conventions and the CI path all now exist. What does
not exist is the one target 5a4d1b7 named and deferred, in its own words:

"Three of the four targets in the plan. Convergence across simulated vaults
is the fourth and is a harness rather than a test file; it is not here."

That fourth target is the merge path, and it is the one that matters most here.
The three that landed cover pure functions with clean oracles — a path
normaliser, a document id, an AEAD envelope. Convergence is different in kind: it
needs generated operation sequences across simulated replicas, and the oracle
is an invariant rather than a return value.

Every one of our remaining test cases over that path is a hand-written example:
we assert the cases we thought of.
The bug classes this repo actually suffers from are the ones nobody thought of —
docs/sync-limitations.md is largely a list of them, and CLAUDE.md records
that several survived rounds of testing because the file on disk looked correct
right up until it was overwritten.

Concurrent CRDT merge under adversarial interleaving is the canonical case for
generated tests rather than enumerated ones. It is also where our central
principle bites hardest: a merge that loses a paragraph is the failure mode the
whole codebase is subordinate to.

What Relay does, as far as it can be read

Caveat, stated first: their test bodies are encrypted and I have not read a
single line. .gitattributes in No-Instructions/Relay is one line —
__tests__/** filter=git-crypt diff=git-crypt — and all 193 files under it are
git-crypt encrypted. Nothing else in that repo is. Their CONTRIBUTING.md says
so plainly:

"Relay has a proprietary development harness and test suite that is not
publicly available. The unit tests included in this repository are encrypted."

Their CI agrees, erroring with "git-crypt unlock did not expose the private unit
tests" and unlocking from a RELAY_GIT_CRYPT_KEY_B64 secret.

So everything below is inferred from filenames and directory structure only.
Treat it as a sketch of an approach, not as knowledge of their implementation.

The shape that is visible

Their merge engine is a hierarchical state machine, and the fuzzing is
model-based over that machine rather than generic input fuzzing. 58 of their 193
test files sit in __tests__/merge-hsm/, with a sibling hsm/interpreter.test.ts
and parallel canvas-hsm/ and folder-hsm/ suites.

The support harness — __tests__/merge-hsm/testing/, excluded from the run by
testPathIgnorePatterns — has 12 modules whose names describe the method:

Module	What the name implies
`random.ts`	Their own randomness source
`syntheticDoc.ts`	Generated documents as input
`createTestHSM.ts`	State-machine factory under test
`createCrossVaultTest.ts`	Multi-replica scenarios as a first-class fixture
`MockYjsProvider.ts`	Deterministic provider, so interleaving is controllable
`fakeEditorHarness.ts`	Editor stub — our `FakeVault` in `vault-adapter.ts` is the same idea
`assertions.ts`	Shared invariant checks rather than per-test asserts
`replay-fixtures.ts`	Failing generated cases captured and replayed as fixtures
`fixtures.ts`, `events.ts`, `integration.ts`, `index.ts`	Supporting

Test names in the same directory that corroborate it: born-attached-fuzz,
invariants, merge-path-adversarial, crdt-integrity,
machine-edit-convergence, machine-edit-rewind, machine-definition,
recording, E2ERecorded, network-resilience, merge-large-doc-perf.

The four ideas worth taking

Model-based, not byte-fuzzing. Generate sequences of operations against
an explicit state machine, not random bytes. machine-definition plus
machine-edit-convergence and machine-edit-rewind reads as: define the
machine, generate edit sequences, assert convergence and that rewind returns
where it started.
Invariants as the oracle. A separate invariants test and a shared
assertions.ts say the property being checked is "these things are always
true", not "output equals expected". For us the invariants are already
written down in prose in CLAUDE.md — no silent loss, conflicts keep both
sides, an empty document never blanks a file that has held content.
replay-fixtures.ts is the part that makes fuzzing pay. A random failure
that cannot be reproduced is a flake; captured as a fixture it becomes a
permanent regression test. Any fuzzing we add needs a seed printed on failure
and a one-command path from that seed to a committed fixture, or it will be
deleted within a month for being noisy.
No fuzzing library. Their package.json declares no fast-check,
jsverify or similar, and random.ts suggests they hand-rolled it. Worth
weighing against fast-check, which gives shrinking for free — shrinking is
the expensive half to build and the half that makes a failure legible.

Where they do not fuzz

relay-server has none. 46 readable .rs files, 245 test functions, and
no proptest, quickcheck, arbitrary or cargo-fuzz anywhere.
Their CI does not run fuzzing. No fuzz, seed or iteration-count
references in any of their 9 workflows. Their nightly e2e-burnin is
repetition of scripted e2e plans with flake rates accumulated in an S3 ledger —
a different technique, and separately worth copying.

So their fuzzing appears to be unit-level, developer-run, over the merge state
machine. Not a nightly fuzzing campaign.

What we would do

Unknown until scoped; the likely shape:

Property tests over applyMinimalDiff and the CRDT merge path with generated
edit sequences across two or more replicas.
Oracle = the invariants in CLAUDE.md, asserted as code rather than prose.
Seed printed on every failure; a documented path from seed to committed
fixture, mirroring replay-fixtures.ts.
fast-check is already the answer — settled by 5a4d1b7, and its
shrinker is what named the chunkSize bytes. The open question is narrower:
whether it can drive MockProvider-style replicas deterministically, or
whether the sequence generator has to be hand-rolled over it.
Mutation-check it (CLAUDE.md): invert a real invariant and confirm the
generator finds it. A fuzzer that has never failed proves nothing — and this
is exactly the rule that has already caught tests here that could not fail.

packages/shared/src/crypto.ts is already partly covered by
blob-cipher.property.test.ts; merge convergence is the gap.

Ties into the v1 conformance work — 5a4d1b7 moved docs/v1/conformance.md and
docs/v1/protocol/crypto.md alongside the tests, so a convergence harness
should state which conformance clauses it exercises rather than standing alone.

Not in scope

Fuzzing the server. It stores opaque ciphertext and never parses a document
(doc-store.ts: "Nothing in this file interprets payload"), so there is
little for a fuzzer to find. Malformed-frame handling at the WebSocket boundary
could be argued separately.

Relay repositories read 22 September 2026 at plugin main. Filenames and
directory structure only — the contents are encrypted and were not read.

> **Corrected 22 September 2026.** This issue was filed claiming we had no > property-based testing at all. That was wrong — it was written against a stale > local checkout. `5a4d1b7` (21 September) added **fast-check 4.10.2** to > `packages/plugin` and `packages/shared`, with 23 properties across > `vault-path.property.test.ts` (10), `blob-cipher.property.test.ts` (9) and > `doc-id.property.test.ts` (4). It found a real thing immediately: a blob > envelope byte-flip accepted after fifteen runs, shrunk to bytes 7–10 — the > header's `chunkSize`, absent from the AAD — now documented as CRYPTO-087. > The issue is rescoped below to what that commit explicitly left out. ## Why The library, the harness conventions and the CI path all now exist. What does **not** exist is the one target `5a4d1b7` named and deferred, in its own words: > *"Three of the four targets in the plan. **Convergence across simulated vaults > is the fourth and is a harness rather than a test file; it is not here.**"* That fourth target is the merge path, and it is the one that matters most here. The three that landed cover pure functions with clean oracles — a path normaliser, a document id, an AEAD envelope. Convergence is different in kind: it needs generated *operation sequences* across simulated replicas, and the oracle is an invariant rather than a return value. Every one of our remaining test cases over that path is a hand-written example: we assert the cases we thought of. The bug classes this repo actually suffers from are the ones nobody thought of — `docs/sync-limitations.md` is largely a list of them, and `CLAUDE.md` records that several survived rounds of testing because the file on disk looked correct right up until it was overwritten. Concurrent CRDT merge under adversarial interleaving is the canonical case for generated tests rather than enumerated ones. It is also where our central principle bites hardest: a merge that loses a paragraph is the failure mode the whole codebase is subordinate to. ## What Relay does, as far as it can be read **Caveat, stated first: their test bodies are encrypted and I have not read a single line.** `.gitattributes` in `No-Instructions/Relay` is one line — `__tests__/** filter=git-crypt diff=git-crypt` — and all 193 files under it are git-crypt encrypted. Nothing else in that repo is. Their `CONTRIBUTING.md` says so plainly: > *"Relay has a proprietary development harness and test suite that is not > publicly available. The unit tests included in this repository are encrypted."* Their CI agrees, erroring with *"git-crypt unlock did not expose the private unit tests"* and unlocking from a `RELAY_GIT_CRYPT_KEY_B64` secret. So **everything below is inferred from filenames and directory structure only.** Treat it as a sketch of an approach, not as knowledge of their implementation. ### The shape that is visible Their merge engine is a **hierarchical state machine**, and the fuzzing is model-based over that machine rather than generic input fuzzing. 58 of their 193 test files sit in `__tests__/merge-hsm/`, with a sibling `hsm/interpreter.test.ts` and parallel `canvas-hsm/` and `folder-hsm/` suites. The support harness — `__tests__/merge-hsm/testing/`, excluded from the run by `testPathIgnorePatterns` — has 12 modules whose names describe the method: | Module | What the name implies | |---|---| | `random.ts` | Their own randomness source | | `syntheticDoc.ts` | Generated documents as input | | `createTestHSM.ts` | State-machine factory under test | | `createCrossVaultTest.ts` | Multi-replica scenarios as a first-class fixture | | `MockYjsProvider.ts` | Deterministic provider, so interleaving is controllable | | `fakeEditorHarness.ts` | Editor stub — our `FakeVault` in `vault-adapter.ts` is the same idea | | `assertions.ts` | Shared invariant checks rather than per-test asserts | | **`replay-fixtures.ts`** | **Failing generated cases captured and replayed as fixtures** | | `fixtures.ts`, `events.ts`, `integration.ts`, `index.ts` | Supporting | Test names in the same directory that corroborate it: `born-attached-fuzz`, `invariants`, `merge-path-adversarial`, `crdt-integrity`, `machine-edit-convergence`, `machine-edit-rewind`, `machine-definition`, `recording`, `E2ERecorded`, `network-resilience`, `merge-large-doc-perf`. ### The four ideas worth taking 1. **Model-based, not byte-fuzzing.** Generate *sequences of operations* against an explicit state machine, not random bytes. `machine-definition` plus `machine-edit-convergence` and `machine-edit-rewind` reads as: define the machine, generate edit sequences, assert convergence and that rewind returns where it started. 2. **Invariants as the oracle.** A separate `invariants` test and a shared `assertions.ts` say the property being checked is "these things are always true", not "output equals expected". For us the invariants are already written down in prose in `CLAUDE.md` — no silent loss, conflicts keep both sides, an empty document never blanks a file that has held content. 3. **`replay-fixtures.ts` is the part that makes fuzzing pay.** A random failure that cannot be reproduced is a flake; captured as a fixture it becomes a permanent regression test. Any fuzzing we add needs a seed printed on failure and a one-command path from that seed to a committed fixture, or it will be deleted within a month for being noisy. 4. **No fuzzing library.** Their `package.json` declares no `fast-check`, `jsverify` or similar, and `random.ts` suggests they hand-rolled it. Worth weighing against `fast-check`, which gives shrinking for free — shrinking is the expensive half to build and the half that makes a failure legible. ### Where they do *not* fuzz - **`relay-server` has none.** 46 readable `.rs` files, 245 test functions, and no `proptest`, `quickcheck`, `arbitrary` or `cargo-fuzz` anywhere. - **Their CI does not run fuzzing.** No `fuzz`, seed or iteration-count references in any of their 9 workflows. Their nightly `e2e-burnin` is repetition of scripted e2e plans with flake rates accumulated in an S3 ledger — a different technique, and separately worth copying. So their fuzzing appears to be **unit-level, developer-run, over the merge state machine**. Not a nightly fuzzing campaign. ## What we would do Unknown until scoped; the likely shape: - Property tests over `applyMinimalDiff` and the CRDT merge path with generated edit sequences across two or more replicas. - Oracle = the invariants in `CLAUDE.md`, asserted as code rather than prose. - Seed printed on every failure; a documented path from seed to committed fixture, mirroring `replay-fixtures.ts`. - **`fast-check` is already the answer** — settled by `5a4d1b7`, and its shrinker is what named the `chunkSize` bytes. The open question is narrower: whether it can drive `MockProvider`-style replicas deterministically, or whether the sequence generator has to be hand-rolled over it. - **Mutation-check it** (`CLAUDE.md`): invert a real invariant and confirm the generator finds it. A fuzzer that has never failed proves nothing — and this is exactly the rule that has already caught tests here that could not fail. `packages/shared/src/crypto.ts` is already partly covered by `blob-cipher.property.test.ts`; merge convergence is the gap. Ties into the v1 conformance work — `5a4d1b7` moved `docs/v1/conformance.md` and `docs/v1/protocol/crypto.md` alongside the tests, so a convergence harness should state which conformance clauses it exercises rather than standing alone. ## Not in scope Fuzzing the server. It stores opaque ciphertext and never parses a document (`doc-store.ts`: *"Nothing in this file interprets `payload`"*), so there is little for a fuzzer to find. Malformed-frame handling at the WebSocket boundary could be argued separately. --- *Relay repositories read 22 September 2026 at plugin `main`. Filenames and directory structure only — the contents are encrypted and were not read.*

cruelacid added the

risk:none

area:plugin

labels

2026-09-22 17:19:40 +01:00

cruelacid added this to the System Stability and Testing project

2026-09-22 17:20:15 +01:00

cruelacid referenced this issue

2026-09-22 17:38:16 +01:00

No flake ledger: per-scenario e2e history does not survive a run #93

cruelacid changed title from ~~No fuzz or property-based testing over the merge path~~ to Property-based testing does not reach the merge path: the convergence harness is outstanding

2026-09-22 17:51:41 +01:00

cruelacid commented

2026-09-23 18:55:52 +01:00

Author

Owner

Moved to the Vikunja board as NEC-68: https://projectron.nerchure.com/tasks/68

Moved to the Vikunja board as **NEC-68**: https://projectron.nerchure.com/tasks/68

cruelacid closed this issue

2026-09-23 18:55:52 +01:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

Nectenda/nectenda#92

No description provided.

Rows
Columns