* fix(hopper): normalize string-backed addresses and update verification * feat(native): add inspection primitives, dispatch traces and approved UI observation * test: prioritize real artifact E2E and captured golden regressions * fix(process): queue observed output before reactive rendering actions * test: prune redundant cases and strengthen boundary assertions * fix(cli): route native value tracing through shared workflows * test: replace synthetic success with real workflows and captured data * fix(catalog): separate portable metadata from native host availability
170 lines
15 KiB
Markdown
170 lines
15 KiB
Markdown
# Test suite audit
|
|
|
|
## Scope and priority
|
|
|
|
The expanded audit on 2026-10-04 reviewed every one of the 355 Vitest files
|
|
present at its start: 82 domain/contracts, 56 application/composition, 147
|
|
adapter/configuration/process/filesystem/CLI files, and 70 MCP/acceptance/
|
|
conformance/evaluation files. Three review passes checked the proposed removals
|
|
against existing stronger coverage and checked the final changes for lost
|
|
assertions. The remaining suite has 351 files.
|
|
|
|
Prefer full real-provider E2E, then integration across schemas and transports,
|
|
then real-data goldens. Remove lower-level duplication once the replacement
|
|
asserts the same behavior. Keep distinct failure regressions that a successful
|
|
E2E cannot exercise. Do not lower coverage thresholds to accommodate pruning.
|
|
|
|
## Changes made
|
|
|
|
| Previous test or assertion | Decision | Replacement or retained claim |
|
|
| ----------------------------------------------------------------------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
|
| Asset application tests using fabricated `.car` bytes and injected utility JSON | Remove two plumbing cases | `verify:asset-catalog`: real actool → assetutil → CLI and stdio MCP, including digest, full metadata, pagination and invalid input |
|
|
| Keyed archive positive binary/XML objects assembled with the same plist library as the reader | Remove two synthetic success cases | Captured Foundation XML plus reviewed complete object/reference graph golden; real Foundation binary/XML → CLI and stdio MCP |
|
|
| Generic contract schema/annotation/example loops and output-map inventory comparisons | Consolidate into one SDK wire test | Assert exported schemas, guidance, effect annotations and valid examples on the actual advertised MCP catalog |
|
|
| Ghidra function operation enum membership | Remove | Canonical catalog, provider handshake and actual operation roundtrips establish admission; malformed provider results remain covered |
|
|
| Asset page assertion pinning `10_001` | Remove | Invalid-page rejection through real MCP; malformed utility data still covered |
|
|
| Suggested prompt tool positions | Remove incidental ordering assertions | Keep authority, evidence, untrusted input and preparation constraints |
|
|
| Package “acceptance” assertions pinning CI matrix/shard values and fake launcher command arrays | Remove configuration-shaped cases | Actual package/install verifier remains; retain a compact release verification-before-publish and registry-canary static guard |
|
|
| Package failure classifier under acceptance | Move to package evaluation | Preserve all malformed/unrelated-failure rejection cases; do not call them E2E |
|
|
| Native value graph checks conditional on success | Strengthen | Require success before pagination/ambiguity assertions; unexpected errors can no longer pass |
|
|
| Native UI verifier treating permission denial as a positive pass | Separate outcomes | Default lane requires real capture/action success; opt-in permissions lane explicitly reports no positive E2E proof |
|
|
|
|
Ten previous test cases were removed or consolidated; three replacement
|
|
Vitest cases cover the captured goldens and release guard. The two real artifact
|
|
E2E scripts add product-level evidence outside the deterministic Vitest count.
|
|
|
|
## Exhaustive follow-up
|
|
|
|
The follow-up removes or consolidates another 31 test declarations, including
|
|
three redundant test files. Parameterized declarations and executed cases have
|
|
different counts; the figures above count source declarations. It also repairs
|
|
assertions that could pass without exercising their stated behavior.
|
|
|
|
| Area | Cleanup | Stronger retained evidence |
|
|
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
|
| Configuration | Merge duplicate target/loader tests and preserve custom launcher, explicit provider and actionable error diagnostics | Independent malformed inputs; each capability flag is rejected separately |
|
|
| Browser scenarios | Remove fake launch success and expression-selected artifact success | Real Chromium scenarios, actual artifacts, CLI capture and owned/external cleanup in `verify:browser` |
|
|
| CLI policy | Remove duplicate confirmation helper cases | Compiled CLI proves project files stay unchanged until `--yes`; distinct interactive choices remain |
|
|
| Provider capabilities | Remove numerical catalog freezes; consolidate Hopper byte-read discovery | Exact canonical capability identities, admission, mutation effects and detached descriptors |
|
|
| Evidence ledger | Merge reordered-key checks into serialized bundle import; remove one-record quota claims | Atomic import, conflicts, residual revisions, actual large collections and byte regressions |
|
|
| MCP batching and concurrency | Replace maximum concurrency and empty indistinguishable responses | Every distinct ordered batch result; concurrently gated requests complete in reverse order with matching payloads |
|
|
| MCP inventories and graphs | Remove traversal/inventory call order and redundant catalog/prose checks | Exact complete cyclic graph, full inline inventories, canonical SDK catalog and all-handler roundtrips |
|
|
| Authority and lifecycle | Attempt the refused Ghidra mutation; require successful setup, call and switch; close twice | Typed refusal with zero mutation dispatch, restored target calls and exact owned-resource cleanup |
|
|
| Process and provider output | Remove incidental environment read and command option order/defaults | PID/ownership validation, exact disabled-policy failure and complete decoded response above 10 MiB |
|
|
| Conformance and CI guards | Merge identical replay runs; remove duplicate missing-row fault, trivial hash/self equality and incidental worker/step counts | Distinct faulty evidence, digest order, actual test discovery, file existence and release/runner authority |
|
|
| Pure schemas | Remove fabricated comparison-dimension injection and impossible long boundary success | Real artifact comparison and SDK coverage evaluation with returned boundary identity |
|
|
|
|
The removed long boundary selector test only parsed a 1,000-character request;
|
|
workspace boundary identities admit at most 200 characters, so that positive
|
|
could never represent a successful evaluation. The actual MCP evaluation stays
|
|
covered. One unused ledger serialization helper was also retired after removing
|
|
its only test consumer; no CLI or MCP contract changed.
|
|
|
|
No further immediately safe pruning was identified in the final review. Retained
|
|
replacement candidates below are coverage gaps, rather than permission to delete
|
|
unique tests. Test-only domain scaffolding also stays pending a separate decision
|
|
about the corresponding source behavior; deleting those tests alone would leave
|
|
unverified code and fail unused-export checks.
|
|
|
|
## Stronger replacements from the closing pass
|
|
|
|
Three more synthetic success cases were removed. Native value composition now
|
|
runs through the production CLI and a separate stdio MCP subprocess with real
|
|
Ghidra, using a source-built native fixture. It checks argument/parameter/return
|
|
relationships, the fixture's global operand, authenticated Evidence, upstream
|
|
and workflow analysis profiles, identical CLI/MCP graphs, discovery availability
|
|
and successful MCP session closure. Ambiguity, cancellation, target changes,
|
|
missing flow and bounded-work regressions remain in the focused suite.
|
|
|
|
That replacement found missing CLI workflow routing for `trace_native_values`
|
|
and missing workflow profile classification. The routing and discovery now use
|
|
the composed operation's actual provider prerequisites; workflow Evidence commits
|
|
to the upstream Ghidra profile.
|
|
|
|
The lone hand-built client-document equality test is replaced by a real TOML
|
|
filesystem transaction: reversed environment keys leave the original bytes
|
|
unchanged and create no backup. A real ZIP entry with an application root longer
|
|
than 5,000 characters replaces injected schema-only positive data. Inventory
|
|
completeness tests partition actual scans into authenticated partial observations
|
|
and require exact assembled results and every source citation, including more
|
|
than 100 citations. These are consumer merge tests, not provider pagination
|
|
claims. Replay aggregation now covers all four statuses and retains actual runner
|
|
diagnostics; it does not claim to execute a real sandbox. Retained pixel-metric
|
|
fixtures now write correct PNG chunk checksums.
|
|
|
|
The all-zero screenshot buffer is replaced by a real Chromium PNG of 9,467,940
|
|
bytes, transported intact through CLI and stdio MCP and decoded by the production
|
|
pixel comparator. The verifier configures the SDK client's receive buffer because
|
|
its 10 MiB default is smaller than the inline JSON response. This does not claim
|
|
large comparison-request transport coverage. A real SDK initialize request,
|
|
captured during a successful production stdio handshake, replaces the JSON-RPC
|
|
decoder's field-count-only oracle with exact field/value expectations.
|
|
|
|
The closing pass also removed assertions tied to internal call order, defaults,
|
|
private cache sizes, duplicated polling and fabricated schema extensions. Unique
|
|
malformed-input, security-capacity and cleanup regressions remain.
|
|
|
|
## Real process regression found
|
|
|
|
The full suite exposed a renderer ordering race. Publishing a terminal output
|
|
journal entry could synchronously trigger a reactive resize before the preceding
|
|
output was queued for rendering. Queue that output first. The existing real
|
|
source-owned interactive PTY test now also requires the first resized frame to
|
|
contain the output that triggered the resize, preserving the original event
|
|
order rather than sorting or changing timestamps to conceal it.
|
|
|
|
## Coverage kept
|
|
|
|
Malformed archives, broken and cyclic references, unsupported metadata pointers,
|
|
relative Objective-C method layouts, ambiguous call candidates, digest mismatch,
|
|
cancellation, permission approval, bounded output and owned-resource cleanup
|
|
remain covered. These cases provide evidence beyond a successful real-tool run.
|
|
The hand-assembled relative method fixture stays until compiler-produced
|
|
relative-list fixtures establish equivalent assertions.
|
|
|
|
## Remaining priorities
|
|
|
|
1. Extend the real Ghidra CLI/stdio MCP journey from value tracing to dispatch
|
|
and UI handlers before pruning their hand-authored composition success cases.
|
|
Native value tracing now establishes full transport parity.
|
|
2. Real binary dispatch and nib → full product transports, with normalized
|
|
semantic goldens. The compiler-backed readers already run in the new Apple
|
|
CI lane; synthetic pointer/layout rejection tests remain useful.
|
|
3. An approved interactive macOS runner for native UI capture/actions, including
|
|
stdio MCP scenarios. Hosted runners without permissions cannot supply this
|
|
proof. No arbitrary application's windows should be opened for this lane.
|
|
4. Real macOS/Linux Ghidra CI runners with bring-your-own installations. This
|
|
change adds Apple artifact CI without installing Java or Ghidra.
|
|
5. Classify fake-runner replay/application acceptance accurately and replace
|
|
repeated happy paths only after real sandbox journeys assert the same claims.
|
|
The replay workflow's allow-unavailable mode must not imply executed replay.
|
|
6. Replace repeated-array schema ceiling tests with real producer/transport
|
|
regressions before deleting them. Existing integration tests do not yet
|
|
establish their large-input guarantees.
|
|
|
|
Some static guards protect irreversible release authority rather than product
|
|
behavior. Keep these few guards explicit and separate from E2E evidence.
|
|
|
|
## Verification after the expanded audit
|
|
|
|
- `npm run check:pr`: 351 files passed; 1,794 tests passed and two skipped.
|
|
Typecheck, lint, formatting, unused-code and generated-document gates passed.
|
|
- `npm run check:fast`: passed; 70 lint warnings and zero errors.
|
|
- `npm run verify:browser`: actual Chromium observation and scenario CLI routes,
|
|
retained artifacts, owned/external lifecycle checks and large CLI/stdio MCP PNG
|
|
capture passed.
|
|
- `npm run verify:ghidra`: macOS ARM64 with Ghidra 12.1.4/JDK 21 passed, including
|
|
real native value tracing through CLI and stdio MCP: 171 nodes, 298 edges and
|
|
six decompilations.
|
|
- `npm run verify:asset-catalog`: real actool/assetutil → CLI and stdio MCP,
|
|
complete records, pagination and malformed input checks passed.
|
|
- `npm run verify:keyed-archive`: real Foundation binary/XML → CLI and stdio MCP,
|
|
full graph golden, shared/cyclic identity and malformed scope checks passed.
|
|
|
|
- `npm run verify:fixtures`: seven source-built host fixtures, source/artifact
|
|
digests and exhaustive inventory checks passed.
|
|
|
|
No Hopper windows were opened for this audit. The closing pass runs real Ghidra
|
|
on macOS ARM64; earlier Hopper evidence remains separate. No new hosted CI success
|
|
is claimed.
|