NEC-69: No flake ledger: per-scenario e2e history does not survive a run

nectenda-agent commented

2026-09-28 11:58:44 +01:00

Collaborator

Task: NEC-69 — https://projectron.nerchure.com/tasks/69

A flake ledger for the e2e suite, in two stores. Every CI run of the multi-vault suite records one row per scenario (verdict, duration, first error line, run, commit, Obsidian version) in Forgejo's generic package registry, one version per run, so history outlives logs and artifacts; scripts/e2e-ledger.mjs report gives per-scenario pass rates (infra excluded both ways) and flags duration drift. Known flakes are catalogued in a new Vikunja project, Flakes (child of Nectenda, FLAKE-1..7 seeded), with match rules; scripts/agent/flakes.mjs triage names the signature a failure matches, or NEW. card.mjs show now prints a card's comments and related cards (a related flake signature in full), which is how this card's own added requirement was first missed; create --relate links a new card to a signature. packages/e2e/README.md gains the red-run policy (stability items 2, 3, 5). Review found and this fixes a describe-scoped hook failure that would have recorded a red run as all-pass. Verified locally: 99 script tests, 22 mutations all caught, gate green with 96 e2e scenarios, and record/triage run on that real report.

Spec: docs/changes/NEC-69-flake-ledger/spec.md

Changelog

NONE

Task: NEC-69 — https://projectron.nerchure.com/tasks/69 A flake ledger for the e2e suite, in two stores. Every CI run of the multi-vault suite records one row per scenario (verdict, duration, first error line, run, commit, Obsidian version) in Forgejo's generic package registry, one version per run, so history outlives logs and artifacts; scripts/e2e-ledger.mjs report gives per-scenario pass rates (infra excluded both ways) and flags duration drift. Known flakes are catalogued in a new Vikunja project, Flakes (child of Nectenda, FLAKE-1..7 seeded), with match rules; scripts/agent/flakes.mjs triage names the signature a failure matches, or NEW. card.mjs show now prints a card's comments and related cards (a related flake signature in full), which is how this card's own added requirement was first missed; create --relate links a new card to a signature. packages/e2e/README.md gains the red-run policy (stability items 2, 3, 5). Review found and this fixes a describe-scoped hook failure that would have recorded a red run as all-pass. Verified locally: 99 script tests, 22 mutations all caught, gate green with 96 e2e scenarios, and record/triage run on that real report. Spec: `docs/changes/NEC-69-flake-ledger/spec.md` ## Changelog NONE

nectenda-agent added the

area:ops

label

2026-09-28 11:58:44 +01:00

nectenda-agent added 5 commits

2026-09-28 11:58:44 +01:00

Record every e2e scenario's verdict and duration in a ledger outside CI e010fef8bc

Before this a run's per-scenario result existed only inside a zstd job
log, so "has repro-runaway ever failed, and how often" meant paging logs
by hand. The multi-vault suite now writes a vitest JSON report, and the
e2e action turns it into one row per scenario (verdict, duration, first
error line, run, commit, ref, event, Obsidian version) and puts it in
Forgejo's generic package registry, one version per run. A version per
run rather than one file appended to, because the gate and the nightly
finish together and a pull-append-put loses one of them.

infra is only "the suite never reached the scenario" and is left out of
the rate both ways; a hook failure is a fail on (setup). Writing the
ledger never changes the job's result. `report` builds the rolling
table, with a duration-drift flag, from the registry alone.

Task: NEC-69
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpwTCLAtXqwC5vo1om4Rob

Catalogue flake signatures on the board, and show cards with their comments 20c41d67fd

A red run can now be looked up: the Flakes project (a child of Nectenda,
FLAKE-n) holds one task per known flake, status as its bucket, each with
a flake-match rule. flakes.mjs triage matches a run's failed rows against
them and prints the signature or NEW; --record adds one occurrence
comment per run. Triage only annotates, and excludes nothing from a rate
or the gate. Seeded with seven signatures from the README, specs and
commits; the bodies are in the change folder.

card.mjs show printed only a card's description, which is how NEC-69's
own added requirement, left as a comment, was first missed. It now
prints every comment and the related cards, and a related signature in
full, so a card filed with create --relate FLAKE-n starts with its
evidence. The skill says to read comments and triages an e2e failure
before anything else.

Task: NEC-69
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpwTCLAtXqwC5vo1om4Rob

Write down the red-run policy, and mark stability items 2, 3 and 5 done 29acee6412

packages/e2e/README.md now says how to read the ledger and the
catalogue, and what to do when the suite goes red: triage before any
re-run, a match on an open signature is evidence not permission, a
match on a Fixed one is a regression. The change spec records where the
build departs from the plan: the owner created the Flakes project, and
NEC-87 is deliberately not catalogued.

Task: NEC-69
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpwTCLAtXqwC5vo1om4Rob

Build the ledger's ANSI pattern from a string, as no-control-regex asks 5e614e7abe

Task: NEC-69
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpwTCLAtXqwC5vo1om4Rob

Close four ways the ledger could misreport, found in review

CI / e2e (pull_request) Failing after 3s

Details

Release note / release-note (pull_request) Successful in 12s

Details

CI / promote (pull_request) Has been cancelled

Details

CI / build (pull_request) Has been cancelled

Details

e910d8780e

- A beforeAll or afterAll inside a describe fails the file with an
  empty message and marks its scenarios skipped (vitest 4.1.11), so the
  run was recorded as all-pass. A failed file with no failed scenario is
  now a (setup) fail whatever its message says.
- A report is moved aside once recorded, so a later local run that dies
  before vitest writes one records "no report" instead of the old result.
- triage says so when a run has infra rows, rather than "no failed
  scenarios".
- A malformed match rule matches nothing instead of throwing, and the
  most open signature wins whatever order the buckets sit in, with the
  others named.

Task: NEC-69
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpwTCLAtXqwC5vo1om4Rob

cruelacid referenced this pull request from a commit

2026-09-28 12:00:31 +01:00

Quote the ledger step's run line as a block, which YAML can parse

cruelacid added 1 commit

2026-09-28 12:00:31 +01:00

Quote the ledger step's run line as a block, which YAML can parse

Release note / release-note (pull_request) Successful in 15s

Details

e2e / multi (push) Successful in 4m48s

Details

CI / build (pull_request) Successful in 5m3s

Details

CI / e2e (pull_request) Successful in 5m36s

Details

CI / promote (pull_request) Has been skipped

Details

a4e9323aaa

":⚠️📒 record exited" has ": " in a plain scalar, which
YAML reads as a mapping; the runner rejected the whole obsidian-e2e
action for it (PR #200, run 940, task 1591), so the suite never ran.

Task: NEC-69
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LpwTCLAtXqwC5vo1om4Rob

nectenda-agent referenced this pull request

2026-09-28 12:20:42 +01:00

NEC-144: Lint parses workflow and action YAML before CI does #201

cruelacid referenced this pull request from a commit

2026-09-28 14:51:51 +01:00

Quote the ledger step's run line as a block, which YAML can parse

cruelacid force-pushed worktree-nec-69-no-flake-ledger-per-scenario-e2e-history from a4e9323aaa

Release note / release-note (pull_request) Successful in 15s

Details

e2e / multi (push) Successful in 4m48s

Details

CI / build (pull_request) Successful in 5m3s