docs(testing): prioritize real workflows and distinct regressions
This commit is contained in:
@@ -90,6 +90,8 @@ See [docs/tool-design.md](docs/tool-design.md) for the design checklist. When us
|
||||
|
||||
## Testing Guidelines
|
||||
|
||||
Prioritize real end-to-end workflows, boundary integration, and golden producer data. Keep a focused module test when it covers a distinct failure or semantic case absent from those workflows; test paths do not establish depth. See [docs/testing.md](docs/testing.md) for pruning and classification rules.
|
||||
|
||||
Name tests `*.test.ts`. Use Vitest and production seams (`tests/fixtures/`) rather than module mocks. Domain tests assert pure behavior; adapter tests use fake launcher/socket seams; MCP tests connect with the client SDK version pinned in `package.json`. Preserve the canonical tool inventory defined by `TOOL_CONTRACTS` and verified through `CATALOG_IDENTITY` and generated product metadata. Cover malformed input, cancellation, lifecycle cleanup, and actual format, protocol, host-permission, and target-identity boundaries. Do not add tests that merely freeze arbitrary caps or prescribed call sequences. Real Hopper, Ghidra, browser, managed conformance, and any real managed-tool claims cannot be replaced by mocks; use the corresponding `verify:*` command.
|
||||
|
||||
Keep tool catalogs complete and self-describing; prefer capability- and session-scoped availability over schema truncation. Serialized bytes alone do not measure agent usability or model context cost.
|
||||
|
||||
+17
-2
@@ -7,8 +7,10 @@ semantic cases that these workflows cannot reliably reproduce. A test's path
|
||||
or suite name does not establish its behavioral depth.
|
||||
|
||||
Avoid tests that mirror getters, enum mappings, fixture helpers, or prescribed
|
||||
call sequences. Prefer one representative workflow over a Cartesian matrix
|
||||
when every row crosses the same boundary. Keep distinct command handlers,
|
||||
call sequences, or assert cache object identity without a caller-visible contract.
|
||||
Prefer representative workflows over a Cartesian matrix when rows cross the
|
||||
same boundary. Retain combinations that exercise a distinct interaction, such as
|
||||
filtering within a full-output envelope. Keep distinct command handlers,
|
||||
producer representations, failure reasons, and lifecycle states covered.
|
||||
Consolidate their shared setup without hiding the inputs or expected evidence.
|
||||
|
||||
@@ -21,6 +23,19 @@ tests consume a production module, trace CLI/MCP and verifier imports,
|
||||
including imports of compiled files, and remove abandoned scaffolding with its
|
||||
tests when it has no runtime consumer.
|
||||
|
||||
When a test expects a `Result` to succeed, throw its error before asserting the
|
||||
returned value. An `.ok` assertion immediately before the same throwing guard
|
||||
adds no coverage and hides the typed failure. A removed parser matrix can leave
|
||||
one golden malformed-input case and a real filesystem lifecycle that proves
|
||||
configure, update, readback, and removal; an add-only journey does not cover the
|
||||
whole lifecycle.
|
||||
|
||||
Keep in-memory MCP sessions in the MCP boundary lane and direct filesystem
|
||||
workflows in the filesystem boundary lane. Reserve acceptance paths for journeys
|
||||
through the compiled public CLI or MCP runtime. Calling a compiled private
|
||||
workflow or injecting a provider still establishes its narrower integration
|
||||
claim, even if a directory name calls it acceptance.
|
||||
|
||||
Measure slow files before removing capacity regressions. Optimize repeated
|
||||
process startup, fixture construction, and production algorithms when they
|
||||
dominate runtime. Keep at least one input beyond each formerly failing size or
|
||||
|
||||
Reference in New Issue
Block a user