docs(testing): prioritize real workflows and distinct regressions

This commit is contained in:
morluto
2026-10-09 02:58:03 +08:00
parent 0e0b386dc3
commit 9588482bfe
2 changed files with 19 additions and 2 deletions
+2
View File
@@ -90,6 +90,8 @@ See [docs/tool-design.md](docs/tool-design.md) for the design checklist. When us
## Testing Guidelines
Prioritize real end-to-end workflows, boundary integration, and golden producer data. Keep a focused module test when it covers a distinct failure or semantic case absent from those workflows; test paths do not establish depth. See [docs/testing.md](docs/testing.md) for pruning and classification rules.
Name tests `*.test.ts`. Use Vitest and production seams (`tests/fixtures/`) rather than module mocks. Domain tests assert pure behavior; adapter tests use fake launcher/socket seams; MCP tests connect with the client SDK version pinned in `package.json`. Preserve the canonical tool inventory defined by `TOOL_CONTRACTS` and verified through `CATALOG_IDENTITY` and generated product metadata. Cover malformed input, cancellation, lifecycle cleanup, and actual format, protocol, host-permission, and target-identity boundaries. Do not add tests that merely freeze arbitrary caps or prescribed call sequences. Real Hopper, Ghidra, browser, managed conformance, and any real managed-tool claims cannot be replaced by mocks; use the corresponding `verify:*` command.
Keep tool catalogs complete and self-describing; prefer capability- and session-scoped availability over schema truncation. Serialized bytes alone do not measure agent usability or model context cost.
+17 -2
View File
@@ -7,8 +7,10 @@ semantic cases that these workflows cannot reliably reproduce. A test's path
or suite name does not establish its behavioral depth.
Avoid tests that mirror getters, enum mappings, fixture helpers, or prescribed
call sequences. Prefer one representative workflow over a Cartesian matrix
when every row crosses the same boundary. Keep distinct command handlers,
call sequences, or assert cache object identity without a caller-visible contract.
Prefer representative workflows over a Cartesian matrix when rows cross the
same boundary. Retain combinations that exercise a distinct interaction, such as
filtering within a full-output envelope. Keep distinct command handlers,
producer representations, failure reasons, and lifecycle states covered.
Consolidate their shared setup without hiding the inputs or expected evidence.
@@ -21,6 +23,19 @@ tests consume a production module, trace CLI/MCP and verifier imports,
including imports of compiled files, and remove abandoned scaffolding with its
tests when it has no runtime consumer.
When a test expects a `Result` to succeed, throw its error before asserting the
returned value. An `.ok` assertion immediately before the same throwing guard
adds no coverage and hides the typed failure. A removed parser matrix can leave
one golden malformed-input case and a real filesystem lifecycle that proves
configure, update, readback, and removal; an add-only journey does not cover the
whole lifecycle.
Keep in-memory MCP sessions in the MCP boundary lane and direct filesystem
workflows in the filesystem boundary lane. Reserve acceptance paths for journeys
through the compiled public CLI or MCP runtime. Calling a compiled private
workflow or injecting a provider still establishes its narrower integration
claim, even if a directory name calls it acceptance.
Measure slow files before removing capacity regressions. Optimize repeated
process startup, fixture construction, and production algorithms when they
dominate runtime. Keep at least one input beyond each formerly failing size or