Compare commits

..
Author SHA1 Message Date
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>morluto
a56f7bb668 chore(main): release rea-agents 6.4.0 (#1399)
* chore(main): release rea-agents 6.4.0

* docs(release): write contributor-aware 6.4.0 changelog

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 11:57:29 +08:00
morluto ee46323b49 fix(windows): preserve drive-root private runtime paths
Join the admitted parent and generated child name without duplicating a drive-root separator. Reuse that name for handle-relative creation and path provenance.

Add native snapshot, readback, and cleanup coverage through the Ghidra dot-prefixed temp fallback, plus a session-root contract regression.

Fixes #1690
2026-10-11 11:51:39 +08:00
morluto a99857df23 test(release): keep credits attached to their contribution 2026-10-11 11:51:17 +08:00
morluto 6b043dc3de fix(release): avoid duplicated contributor thanks 2026-10-11 11:49:49 +08:00
morluto 4003151ec1 fix(ci): allow browser runtime action transport latency 2026-10-11 11:45:46 +08:00
92dac97102 docs(website): add beginner help to the native analysis guide (#1689)
Add a short glossary, an analysis-tool chooser linked to native setup,
a note that REA does not include DX-Ball, and help for copying a file's
full path.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:43:06 -07:00
YsopwnableandGitHub 89ea2cad19 perf(domain): reuse canonical JSON key encodings
Reuse exact JSON key encodings within the existing 8,192-code-unit digest-buffer quantum and avoid ordinary member value frames. Preserve canonical bytes and cooperative yielding; resume objects when a pending property is omitted.

Flameox paired experiments establish a local digest-workload improvement, with complete-result/EvidenceID oracles unchanged. Full CLI acceleration and billed token reductions are not claimed.

Refs #1065.
2026-10-11 11:40:47 +08:00
ea06264a56 docs(website): list the setup guide first on the Guides page (#1688)
Move the "Set up REA" entry ahead of the native, JavaScript and browser
guides so new readers see setup before the worked examples.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:33:30 -07:00
morluto 59b4a3c319 test(windows): verify native addon in a renamed Node host 2026-10-11 11:32:07 +08:00
xeropresenceandGitHub ea1c164e5e fix(windows): load native controls in Node-API hosts not named node.exe (#1531)
fix(windows): resolve Node-API imports from the host executable

Delay-load the Node-API import library and bind "node.exe" to the running
process, matching node-gyp's win_delay_load_hook. The addon now loads under
any Node-API host (Node.js, Bun) regardless of its executable file name.
2026-10-11 11:30:41 +08:00
dc676b4d85 docs(website): add beginner help to the Get started page (#1684)
Add collapsible help sections for installing Node.js, choosing a coding
agent, opening a terminal, restarting a client, copying a folder's full
path and finding an Electron app's .asar file.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:27:12 +08:00
ad3e41d46e fix(browser): resolve XLink and SVG prefixes in XHTML dom_urls (#1683)
* fix(browser): resolve XLink and SVG prefixes in XHTML dom_urls

DOMSnapshot has no namespace URI. HTML documents still use the parser's
local names. In XHTML and other XML documents, resolve in-scope xmlns
declarations so an element is SVG only in the SVG namespace, and report
any XLink href as href when plain href is absent.

Refs #1586

* fix(browser): bound XML namespace resolution and reject cyclic ancestry

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 11:26:26 +08:00
YsopwnableandGitHub 17f704050f feat(evidence): discover retained records without full payloads (#1682)
* feat(evidence): discover retained records without full payloads

* fix(evidence): guide oversized results to retained discovery

* style(tests): format retained discovery recovery assertion
2026-10-11 11:18:26 +08:00
c8820e03db fix(apple): decode binary app Info.plist files without plutil (#1476)
* fix(apple): decode binary app Info.plist files without plutil

* test(apple): use Windows-legal names in the binary plist bundle test

* fix(apple): ignore __proto__ entries in binary app Info.plist files

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 11:14:15 +08:00
morluto ab80ec640a fix(ci): report package verifier stage progress 2026-10-11 11:11:38 +08:00
morluto 9ce6ef895e test: reuse stdio MCP server in acceptance file 2026-10-11 11:11:38 +08:00
morluto b3e977e6d0 fix(browser): bound screenshot comparison memory 2026-10-11 11:11:38 +08:00
morlutoandYou Wish a99b2fd3b5 test: retain focused CommonJS export regressions
Keep uncovered assignment and binding regressions in the shared CLI and MCP runtime oracle, with a POSIX escaped-path fixture.

Co-authored-by: You Wish <29498139+Y0UW1SH@users.noreply.github.com>
2026-10-11 11:07:15 +08:00
Rudy Celekliandmorluto 4e4b15b825 fix(browser): validate PNG chunks and preserve unsupported encodings
Validate chunk checksums, type bytes, ordering and end boundaries before pixel comparison. Distinguish malformed PNGs and corrupt DEFLATE streams from recognized unsupported encodings through CLI and MCP error projection.

Adapted from #1375 with additional decoder and public boundary regressions. Eighteen decoder tests and six compiled CLI/MCP cases passed.

Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 11:07:15 +08:00
morluto 21dbcf7a14 perf(cli): defer binary provider composition 2026-10-11 11:07:15 +08:00
Kaoru0822andmorluto 76296c5a38 fix(native): bind NativeAOT metadata to declared PE section extents
Section headers distinguish the declared virtual extent from the on-disk raw extent, which may include alignment padding. Resource RVA ownership and overlap use VirtualSize, falling back to SizeOfRawData when VirtualSize is zero, and payload reads still require file-backed bytes.

The NativeAOT snapshot reader applies the same range rule to RVA reads, metadata scans, executable targets, and derived overlay bounds. Page-rounded loader mappings may expose tail bytes, but those bytes lie outside the declared section range used by these parsers.
2026-10-11 11:07:15 +08:00
morlutoandYou Wish e38550462e test(native): stabilize resource-limited binary layout controls
Keep the configured launcher alive while its limited Python child runs so ownership checks retain the admitted executable identity. Trigger memory exhaustion with one oversized ELF zero-fill tail rather than racing section traversal against a timeout.

Adapted from #1357. JavaScript syntax, strict C syntax compilation, formatting, and static checks passed. The Linux pwntools verification lane was not run on this macOS host.

Co-authored-by: You Wish <29498139+Y0UW1SH@users.noreply.github.com>
2026-10-11 11:06:10 +08:00
morlutoandYou Wish 80896a1d64 fix(process): settle launcher exit without dropping descendant lineage
Refresh a mismatched launcher snapshot only when a new live process table proves its PID absent. Preserve uncertainty for detached descendants and continue checking ownership of surviving group members before signaling. Index retained lineage membership instead of rescanning it for each refreshed process.

Adapted from #1357. Validation: 71 source tests passed, one Linux-only identity case skipped; 32 real process-boundary tests passed on macOS.

Co-authored-by: You Wish <29498139+Y0UW1SH@users.noreply.github.com>
2026-10-11 11:06:10 +08:00
morluto e5bc4e85ec fix(ci): close speculative browser fixture connections 2026-10-11 11:05:08 +08:00
38221cb12e fix(native): bound PE resource RVAs by section virtual size (#1678)
The PE resource reader treated each section as covering
`max(VirtualSize, SizeOfRawData)` bytes, the mapping #1596 removed from
managed RVA reads. A resource payload in the raw alignment padding past
VirtualSize was reported with a hash of bytes the image loader never maps.
A later section whose virtual range starts inside that padding was refused
as an overlapping mapping.

Containment and overlap now use the loader's virtual extent: VirtualSize,
or SizeOfRawData when VirtualSize is zero. Ranges must still be file-backed.

Refs #1596

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:01:45 -07:00
3749ccec47 feat(ghidra): recover signed AArch64 relative jump tables (#1592)
* feat(ghidra): recover signed AArch64 relative jump tables

Generalise the AArch64 relative-table matcher to signed byte/halfword entries, UXTW indexes, byte or LSL #2 offsets, B.CS/B.HS bounds and a CBZ route for zero entries. Cross-check it against Ghidra's own load table when one exists, and add a fixture modelled on the RenderScript blend kernel.

Refs #487

* fix(ghidra): preserve typed labels for normalized switch selectors

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 10:45:35 +08:00
a67dbf6a48 fix: reject directory paths for the Wakaru limiter and mitmdump
`REA_JAVASCRIPT_PRLIMIT_COMMAND` and `REA_MITMDUMP_COMMAND` were admitted
after an access check alone, which a searchable directory passes. Admission
then succeeded, and the failure surfaced later as a launch-time adapter
error instead of the capability refusal that names the configured path.

Require a regular file, as #751 did for the firmware engine and limiter.
Linux-only tests mirror that change; both checks only run on Linux hosts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
morlutoandClaude Opus 5.5 bd3a9bbbaf fix(native): classify PE resource input failures
Report missing and non-file selections as input issues, malformed PE images as format errors, and unstable reads as artifact changes. Keep existing limit, permission, cancellation, and other I/O classifications.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
morlutoandClaude Opus 5.5 74050978be fix(javascript): keep dependency manifests out of application roots
Select the shallowest inventoried nondependency manifest and retain dependency package evidence without assigning Electron entry roles. Keep the artifact root when no application manifest is observed.

Adapted from PR #1392, with compiled CLI and stdio MCP regressions for root selection and dependency-only inputs.

Co-authored-by: Marnix Pluim <marnixpluim@users.noreply.github.com>

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
b212e92f9b ci: select semantic tests and fixture lanes for their guarded inputs (#1675)
Edits to `release.yml` or `real-ghidra-windows.yml` selected only the
workflow lane, so the shards holding releaseWorkflow, releaseCheckpoint
and realGhidraWindowsWorkflow tests were skipped. Actionlint checks
syntax only, so a PR could drop the package-before-publish gate, the
registry canary or the default-branch-only self-hosted runner restriction
without CI noticing.

Conformance fixture sources under tests/ set only the code lane, so the
conformance-fixtures and reconstruction-readiness jobs, the only callers
of `verify:fixtures` and `verify:readiness`, never ran for them. Route
the directories that build-conformance-fixtures.mjs compiles to both
lanes, and readiness applications to the readiness lane.

Both were raised by the Codex review of #1660 after it merged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:41:40 -07:00
morluto 990e1ad8c1 fix(browser): clear losing storage verifier timers 2026-10-11 10:35:42 +08:00
morluto 76d8f68cb9 feat(ghidra): inspect NativeAOT PE metadata read-only 2026-10-11 10:31:49 +08:00
fe574c0c9f fix(javascript): recover Electron APIs through default imports and optional calls (#1673)
`import electron from "electron"` bound the module with a leading `default`
export segment, so `electron.ipcMain.handle(...)`, `new
electron.BrowserWindow(...)` and destructuring from that binding matched no
Electron API. Electron's CommonJS exports define no `default` member, so the
segment can only denote the module object; drop it when classifying
Electron bindings.

The binding classifier already recorded optional-call roots, but the IPC,
context-bridge, utility-process and sender-validation visitors accepted only
plain calls. `ipcMain?.handle(...)`, `contextBridge?.exposeInMainWorld(...)`
and the common `event.senderFrame?.url.startsWith(...)` check were dropped
without a limitation.

Refs #1656

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:31:08 +08:00
3a7a3074e3 feat(javascript): recognize tRPC procedures and GraphQL operations (#1670)
* feat(javascript): recognize tRPC procedures and GraphQL operations

Record RPC client syntax as network endpoint candidates in both
analyze_javascript_application and analyze_web_bundle, distinguished by
mechanism:

- trpc:procedure:<method> keeps the static member path of
  root.<path>.<method>(...), which tRPC proxies preserve through
  identifier minification. React hooks and TanStack option factories are
  recognized directly. Vanilla query/mutate need a root bound to
  createTRPCClient/createTRPCProxyClient, or a tRPC source marker with a
  router path of at least two segments; subscribe needs the marker and a
  tRPC observer argument. `this` roots, browser globals and RTK Query
  `endpoints` chains are never procedures; cache-key helpers are excluded.
- trpc🔗<link> records the literal url of documented HTTP links.
- graphql:<type> records named operations from string and cooked template
  literals with a valid operation header, and from precompiled
  OperationDefinition objects; a graphql-tag loc.source.body copy is not
  reported twice.

Method, callback and factory names follow @trpc/client and React/TanStack
integration sources and documentation for 11.19. Carrier segments before
the router path, useUtils helpers and minifier-renamed links remain
documented limitations.

Validation: typecheck; domain tests for recognition and false positives
(Apollo, RxJS, stores, RTK Query, browser globals, prose); web bundle
analyzer test; end-to-end analysis of a minified fixture through the
built MCP server on Windows 11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): preserve RPC binding identity and bundled calls

* style(reference-source): fix CI formatting

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 10:28:52 +08:00
121 changed files with 18873 additions and 1888 deletions
+18 -11
View File
@@ -25,6 +25,7 @@ node scripts/release-notes.mjs inventory \
--manifest /tmp/rea-VERSION.contributions.json
node scripts/release-notes.mjs record \
--manifest /tmp/rea-VERSION.contributions.json \
--summary \
--output /tmp/rea-VERSION.record.md
```
@@ -56,25 +57,31 @@ version/date/link heading. Use single-line Markdown list items under:
PR/commit references required by the existing checkpoint validator.
Group related work by analyst workflow or user impact. Explain the resulting
behavior; internal cleanup, tests, CI, and routine documentation generally
belong only in the contribution record. Read ambiguous PRs/diffs before making
behavior in short, concrete entries. Highlights summarize the release; avoid
repeating their full descriptions in Changes. Internal cleanup, tests, CI, and
routine documentation generally belong only in the provenance file. Read ambiguous PRs/diffs before making
claims. Distinguish simulated-provider checks from real-provider verification.
Each bullet covering a PR carries its link and `Thanks @...` for its verified
human author, known co-authors, and confirmed closing-issue reporters. Grouping
must preserve **all** those credits in the same bullet. Include the issue
links. Direct-commit bullets carry the commit link and known human credit.
Credit each represented contribution once in the public notes with its link
and `Thanks @...` for its verified human author, known co-authors, and confirmed
closing-issue reporters. Grouping preserves all those credits in one line.
When Highlights, detailed entries or migrations reference the same work,
put its thanks in one of those entries, not all of them. Include issue links.
Direct-commit bullets carry the commit link and known human credit.
Exclude GitHub bots and known agent accounts. REA's observed Anthropic and
Cursor co-author trailers resolve to `@claude` and `@cursoragent` with GitHub
type `User`; the credit policy excludes them while retaining their metadata.
Do not copy OpenClaw's maintainer-specific exclusions.
At the REA maintainer's request, omit `@morluto` self-thanks while preserving
authorship in the inventory. Other human contributors remain eligible.
Use explicit GitHub links for cross-repository references to avoid ambiguous
`#NNN` numbers. Unknown handles stay unknown; they are not fabricated credit.
Append the generated `### Complete contribution record` verbatim as the last
section of this release. It accounts for every discovered in-range PR and
direct contribution, including internal work omitted from the prose. Do not
hand-edit generated credits to make validation pass. Refresh the inventory
Append the generated `record --summary` provenance link verbatim at the end
of this release. The linked inventory accounts for every discovered in-range
PR and direct contribution, including internal work omitted from the prose.
Do not paste the full `record` output into public notes: it duplicates thanks
and obscures the user-facing changes. Full records remain available for review.
Do not hand-edit generated credits to make validation pass. Refresh the inventory
when product history changes; a prose-only correction can reuse the snapshot.
## Finalize with Release Please
@@ -135,6 +135,16 @@ session to a selected path. The analysis remains complete in the ledger; avoid
repeating it merely to request the same oversized response. Complete CLI JSON
output also streams without a single MCP frame.
When the retained parent ID is unknown, use `get_evidence_bundle` with
`detail: "summary"` and exact filters for operation, target SHA-256,
analysis-profile digest or canonical procedure address. Select its returned
Evidence ID for the next view instead of requesting the complete bundle.
`retention: "complete-record"` means the whole record is stored; native
value-flow truncation and unknown metadata remain separate. For an exported
portable bundle, use `rea inspect-evidence-bundle` with JSON containing `path`,
`detail: "summary"` and the same `filters`. Reading that file still requires
validating its full contents within the local memory budget.
A client can also truncate a successful tool result without a REA transport
error. Treat that preview as incomplete; use the retained parent ID for smaller
views or export when the full record is needed. Do not parse the preview as
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "6.3.0"
".": "6.4.0"
}
+64
View File
@@ -1,5 +1,69 @@
# Changelog
## [6.4.0](https://github.com/morluto/rea/compare/rea-agents-6.3.0...rea-agents-6.4.0) (2026-10-11)
### Highlights
- Android: added device inspection, split-APK acquisition, logs, screenshots, file transfers and explicit app lifecycle actions through ADB, plus manifest and localized-string decoding with Apktool. ([#1552](https://github.com/morluto/rea/pull/1552), [#1556](https://github.com/morluto/rea/pull/1556)) Thanks @3QL.
- JEB: browse project units and decompile selected items through a caller-started local JEB MCP server. ([#1536](https://github.com/morluto/rea/pull/1536)) Thanks @3QL.
- Large analyses: discover retained Evidence without loading every payload, and inspect JavaScript graphs, binary layouts and function dossiers through summaries and selected views. ([#1682](https://github.com/morluto/rea/pull/1682), [#1568](https://github.com/morluto/rea/pull/1568), [#1570](https://github.com/morluto/rea/pull/1570), [#1345](https://github.com/morluto/rea/pull/1345), [#1507](https://github.com/morluto/rea/pull/1507), [#1247](https://github.com/morluto/rea/issues/1247)) Thanks @jhuang-tw, @kaoru0822-kitauji, @N0zoM1z0.
- JavaScript: isolate source analysis in an owned process with configurable heap and timeout limits; retain completed facts and explicit unknowns when analysis runs out of resources. ([#1651](https://github.com/morluto/rea/pull/1651), [#1563](https://github.com/morluto/rea/issues/1563), [#1565](https://github.com/morluto/rea/issues/1565)) Thanks @N0zoM1z0, @peter-research.
- Ghidra: configure import language/compiler specifications and seed functions, code and labels before auto-analysis. ([#1667](https://github.com/morluto/rea/pull/1667)) Thanks @sqmagellan.
- RPC discovery: recognize supported tRPC calls, HTTP-link URLs and named GraphQL operations in JavaScript applications and web bundles. ([#1670](https://github.com/morluto/rea/pull/1670)) Thanks @pompiliodonatoapp.
### Changes
- Added Ghidra support for Windows PE DLLs and bounded MIPS ELF analysis, including carved MIPS inputs with incomplete section tables. Windows P0 remains read-only. ([#1303](https://github.com/morluto/rea/pull/1303), [#1362](https://github.com/morluto/rea/pull/1362), [#1629](https://github.com/morluto/rea/pull/1629), [#1290](https://github.com/morluto/rea/issues/1290), [#718](https://github.com/morluto/rea/issues/718), [#1619](https://github.com/morluto/rea/issues/1619)) Thanks @bqw371, @jaelliot, @kvalls, @N0zoM1z0.
- Enabled `set_address_name` and `set_addresses_names` for Ghidra sessions with mutation authority, including namespace-aware data labels and per-address batch results. ([#1635](https://github.com/morluto/rea/pull/1635), [#1634](https://github.com/morluto/rea/issues/1634)) Thanks @giri256.
- Recovered additional signed AArch64 relative jump-table patterns. ([#1592](https://github.com/morluto/rea/pull/1592)) Thanks @RishTheFish21.
- Added PE resource inspection and read-only NativeAOT metadata parsing for supported x64 PE32+/.NET 8 RTR 9.1 inputs. The NativeAOT parser’s Windows-host Ghidra workflow remains unverified on a real Windows host. ([#1482](https://github.com/morluto/rea/pull/1482), [76d8f68](https://github.com/morluto/rea/commit/76d8f68cb918dfb43869f2b649a0b6c0c1cf266d), [#1254](https://github.com/morluto/rea/issues/1254)) Thanks @N0zoM1z0, @thangldw.
- Added Pi, Hermes and Qwen Code setup, including Hermes active-profile support. ([#1376](https://github.com/morluto/rea/pull/1376), [#1403](https://github.com/morluto/rea/pull/1403), [#1415](https://github.com/morluto/rea/pull/1415), [#1409](https://github.com/morluto/rea/issues/1409)) Thanks @rasatpetabit, @yiliang114, @zaydiscold.
- Added `REA_MCP_INPUT_SCHEMA_PROFILE=compact` for size-capped MCP clients such as Kimi/Moonshot, while retaining canonical runtime validation. ([#1514](https://github.com/morluto/rea/pull/1514), [#1580](https://github.com/morluto/rea/pull/1580), [#1484](https://github.com/morluto/rea/issues/1484), [#1581](https://github.com/morluto/rea/issues/1581)) Thanks @haythemsellami, @N0zoM1z0.
- Added live process-capture progress and optional `finalization_ms` to capture shutdown output before timeout escalation; cancellation remains immediate. ([#1483](https://github.com/morluto/rea/pull/1483), [#1583](https://github.com/morluto/rea/pull/1583), [#1252](https://github.com/morluto/rea/issues/1252), [#1253](https://github.com/morluto/rea/issues/1253), [modelcontextprotocol/typescript-sdk#2997](https://github.com/modelcontextprotocol/typescript-sdk/issues/2997)) Thanks @N0zoM1z0, @nikolay-dementiev, @sathariels.
- Improved Codex onboarding and recovery from observed client, PATH and provider blockers. ([#1613](https://github.com/morluto/rea/pull/1613)) Thanks @N0zoM1z0.
- Reduced redirected JSON and terminal output size, and deferred heavy imports until their commands need them. ([#1413](https://github.com/morluto/rea/pull/1413), [#1450](https://github.com/morluto/rea/pull/1450), [#1439](https://github.com/morluto/rea/pull/1439), [21dbcf7](https://github.com/morluto/rea/commit/21dbcf7a1458219989ad291e44ce7d017378b727)) Thanks @kaoru0822-kitauji.
### Fixes
- Electron analysis: follow renamed and default-imported APIs and optional calls, ignore overwritten identities and media preload attributes, and resolve packaged `app.asar` pages and unpacked companions. ([#1391](https://github.com/morluto/rea/pull/1391), [#1393](https://github.com/morluto/rea/pull/1393), [#1400](https://github.com/morluto/rea/pull/1400), [#1664](https://github.com/morluto/rea/pull/1664), [#1673](https://github.com/morluto/rea/pull/1673)) Thanks @kaoru0822-kitauji, @marnixpluim, @N0zoM1z0, @suninrain086.
- JavaScript data flow: preserve returned/yielded reference uncertainty, invalidate property-stored aliases correctly, and retain bundled-module runtime scope. ([#1437](https://github.com/morluto/rea/pull/1437), [#1445](https://github.com/morluto/rea/pull/1445), [#1471](https://github.com/morluto/rea/pull/1471), [#1577](https://github.com/morluto/rea/pull/1577), [#1428](https://github.com/morluto/rea/issues/1428), [#1442](https://github.com/morluto/rea/issues/1442), [#1449](https://github.com/morluto/rea/issues/1449), [#1576](https://github.com/morluto/rea/issues/1576)) Thanks @CassieFox23, @dakjdakd, @N0zoM1z0.
- JavaScript packages: resolve exact exports subpaths, preserve trailing-slash directory requests and exact CommonJS keys, and respect shadowed `require` bindings. ([#1424](https://github.com/morluto/rea/pull/1424), [#1615](https://github.com/morluto/rea/pull/1615), [5485483](https://github.com/morluto/rea/commit/5485483f7da2f6cb66441bb784aaf2a1a5f8731a), [8b2f851](https://github.com/morluto/rea/commit/8b2f8514c50e469c3add8ce9ba7ebebef888b14f), [f998fcb](https://github.com/morluto/rea/commit/f998fcb1cca9994c3589fb70d6e0127fc215f76f), [1e51c73](https://github.com/morluto/rea/commit/1e51c732bbf08191688d6526bf58ab2650b601e3), [#1365](https://github.com/morluto/rea/issues/1365)) Thanks @akshitkhokhani, @AWUQAQ, @immanuwell, @rudycelekli.
- Large JavaScript inputs: handle oversized inline source maps, deep returns and reassignment loops; avoid argument expansion limits and yield during large analysis passes. ([#1386](https://github.com/morluto/rea/pull/1386), [#1368](https://github.com/morluto/rea/pull/1368), [#1390](https://github.com/morluto/rea/pull/1390), [#1539](https://github.com/morluto/rea/pull/1539), [#1525](https://github.com/morluto/rea/pull/1525), [#1542](https://github.com/morluto/rea/pull/1542), [#1545](https://github.com/morluto/rea/pull/1545), [#1557](https://github.com/morluto/rea/pull/1557), [#1380](https://github.com/morluto/rea/issues/1380), [#1401](https://github.com/morluto/rea/issues/1401), [#1538](https://github.com/morluto/rea/issues/1538), [#1462](https://github.com/morluto/rea/issues/1462)) Thanks @cold-atom, @KadirJeager, @kaoru0822-kitauji, @marnixpluim, @N0zoM1z0, @rudycelekli.
- JavaScript receivers: expand escaped methods once, index returned-object targets, and avoid replaying earlier writes or duplicating retained graph allocations. ([#1517](https://github.com/morluto/rea/pull/1517), [#1537](https://github.com/morluto/rea/pull/1537), [#1444](https://github.com/morluto/rea/pull/1444), [#1601](https://github.com/morluto/rea/pull/1601), [#1495](https://github.com/morluto/rea/issues/1495)) Thanks @CassieFox23, @geekwolf, @kaoru0822-kitauji, @N0zoM1z0, @zeynelacikgoez.
- Browser captures: retain WebSocket URLs, empty connections and frame order; compare only common known payload coverage and preserve incomplete WebMCP replay coverage. ([db4e360](https://github.com/morluto/rea/commit/db4e3608465486466591b594cf77037ac150c988), [aa257ea](https://github.com/morluto/rea/commit/aa257ea4b16f2d2a35c12df63359d650b48b1154), [e52c0a7](https://github.com/morluto/rea/commit/e52c0a72c59c740315e89149881a27ce4d89a11c))
- Browser scenarios: seed storage in the correct partition, normalize loopback attach identities, retain declared-secret redaction and require verified lineage before Windows Electron signals. ([#1604](https://github.com/morluto/rea/pull/1604), [#1590](https://github.com/morluto/rea/pull/1590), [#1588](https://github.com/morluto/rea/pull/1588), [8b4c3de](https://github.com/morluto/rea/commit/8b4c3de3b3a3b580adec266f94f29ccedf1da228), [#1497](https://github.com/morluto/rea/issues/1497), [#1591](https://github.com/morluto/rea/issues/1591), [#1589](https://github.com/morluto/rea/issues/1589)) Thanks @Adolanium, @akushonkamen, @dakjdakd.
- Added Termux Android browser-scenario support. ([#1655](https://github.com/morluto/rea/pull/1655)) Thanks @tyloryang.
- Web inputs: accept BOM-prefixed HAR, preserve the first duplicate DOM attribute and resolve SVG/XLink URLs in HTML and XHTML. ([#1509](https://github.com/morluto/rea/pull/1509), [#1516](https://github.com/morluto/rea/pull/1516), [#1602](https://github.com/morluto/rea/pull/1602), [#1683](https://github.com/morluto/rea/pull/1683), [#1496](https://github.com/morluto/rea/issues/1496), [#1586](https://github.com/morluto/rea/issues/1586)) Thanks @CassieFox23, @kaoru0822-kitauji, @sathariels, @taljeri.
- Screenshot comparisons: validate PNG chunks, distinguish corruption from unsupported encodings and bound comparison memory. ([4e4b15b](https://github.com/morluto/rea/commit/4e4b15b8250b29e176d4191c9610a3931b693474), [b3e977e](https://github.com/morluto/rea/commit/b3e977e6d073ab1783cece8581279f87fc360411)) Thanks @rudycelekli.
- Archives: preserve case-distinct names, report collisions at their diverging segment, detect nested formats from bytes and reject Windows alternate-stream extraction paths. ([#1480](https://github.com/morluto/rea/pull/1480), [#1505](https://github.com/morluto/rea/pull/1505), [#1553](https://github.com/morluto/rea/pull/1553), [573598c](https://github.com/morluto/rea/commit/573598c6324698cc48ceafd24ed3864670e08d00), [#1470](https://github.com/morluto/rea/issues/1470), [#1491](https://github.com/morluto/rea/issues/1491)) Thanks @CassieFox23, @kaoru0822-kitauji.
- Cleanup: retain failed readers, ASAR handles, output trees and provider workers for retry without discarding completed output or partial observations. ([ffe0574](https://github.com/morluto/rea/commit/ffe0574e17b9d973860371083cb1a351be912131), [de0d341](https://github.com/morluto/rea/commit/de0d341524dc68250a45b41eb7e49fcecc1e32a9), [3f8c51e](https://github.com/morluto/rea/commit/3f8c51e9b506d88fc41aded7b1a22b9a09037c26), [d1235fa](https://github.com/morluto/rea/commit/d1235fadc901e1fb9ab3149d53063306c8545414), [026126e](https://github.com/morluto/rea/commit/026126eafe741e5a8b857433587fd615cdd44c19))
- Providers: canonicalize IDA workspace roots before identity checks, respect IDA cancellation, serialize JEB acquisition and correct Ghidra no-return/imported-symbol recovery. ([#1481](https://github.com/morluto/rea/pull/1481), [#1419](https://github.com/morluto/rea/pull/1419), [5f30ea6](https://github.com/morluto/rea/commit/5f30ea6ff2dd89aab7af6da6e8dfa5e508ed459b), [f88c990](https://github.com/morluto/rea/commit/f88c99021b88ce67e513025901f57478d8107d0c), [49e6f00](https://github.com/morluto/rea/commit/49e6f00ed08b225abf9ddb7c762553607fdab115)) Thanks @AS9929, @JonahSimon, @Y0UW1SH.
- Binary metadata: respect PE/.NET section extents, bound binary-plist expansion, scope keyed-archive traversal and decode app `Info.plist` files without `plutil`. ([#1678](https://github.com/morluto/rea/pull/1678), [#1515](https://github.com/morluto/rea/pull/1515), [#1378](https://github.com/morluto/rea/pull/1378), [#1476](https://github.com/morluto/rea/pull/1476), [975dff2](https://github.com/morluto/rea/commit/975dff2e25b14b634e14dd5e0db1537d0ca9815b), [76296c5](https://github.com/morluto/rea/commit/76296c5a3868ba3b54be6ed3896719283d6de5f8)) Thanks @jayhemnani9910, @kaoru0822-kitauji, @rudycelekli.
- Windows runtimes: preserve drive-root paths when creating private provider directories. ([ee46323](https://github.com/morluto/rea/commit/ee46323b493994902b04824a7dc8aae2a8117903))
- Compatibility: load Windows native controls under Node-API hosts with renamed executables, preserve dylib directory requirements, support Wakaru 1.x and reject directory-valued Hopper launchers. ([#1531](https://github.com/morluto/rea/pull/1531), [#1623](https://github.com/morluto/rea/pull/1623), [#1540](https://github.com/morluto/rea/pull/1540), [#1426](https://github.com/morluto/rea/pull/1426)) Thanks @evenafterallisaid, @Fazmin, @pionxzh, @xeropresence.
- Reference imports: normalize internal symlink lookup consistently, preserve exact paths and exclusion provenance, report root access failures and reject FIFO replacements without blocking. ([#1535](https://github.com/morluto/rea/pull/1535), [2ca3ad2](https://github.com/morluto/rea/commit/2ca3ad238608adfca80a7ff1aaadea6e59a58225), [6636045](https://github.com/morluto/rea/commit/6636045c9c5b472bba51afee9d434394861e93a4), [3967cfa](https://github.com/morluto/rea/commit/3967cfa43e786c65a57e89c265343983d19ef9d0), [f1c58cc](https://github.com/morluto/rea/commit/f1c58cc53fbd83b968e11b3e2eccebca51299c54)) Thanks @anujsingh-cse.
- Setup: honor active configuration and skill roots, preserve configured MCP environments, validate Hermes/OMP profiles before writing, recognize Omarchy and retain recovery backups. ([#1504](https://github.com/morluto/rea/pull/1504), [#1607](https://github.com/morluto/rea/pull/1607), [#1528](https://github.com/morluto/rea/pull/1528), [#1562](https://github.com/morluto/rea/pull/1562), [#1662](https://github.com/morluto/rea/pull/1662), [a011485](https://github.com/morluto/rea/commit/a01148553e2a37c72ae78405418ab648876a96a6), [#1606](https://github.com/morluto/rea/issues/1606), [#1521](https://github.com/morluto/rea/issues/1521), [#1561](https://github.com/morluto/rea/issues/1561)) Thanks @Adkid-Zephyr, @CassieFox23, @LuisTaffelli, @N0zoM1z0, @peter-research.
- Process capture: retain descendant lineage after launcher exit, preserve identity uncertainty before signaling and release terminals after observation failures. ([80896a1](https://github.com/morluto/rea/commit/80896a1d64b216ef893907249cd4252f86215d49), [c916850](https://github.com/morluto/rea/commit/c9168508527302ac57bc86385ccf0d824697b5dc)) Thanks @Y0UW1SH.
- CLI/MCP errors: accept JSON files for web comparisons, preserve native option help and union correction details, remove invalid aliases and validate doctor against its discovered output contract. ([#1397](https://github.com/morluto/rea/pull/1397), [#1395](https://github.com/morluto/rea/pull/1395), [#1425](https://github.com/morluto/rea/pull/1425), [#1461](https://github.com/morluto/rea/pull/1461), [8b4c96e](https://github.com/morluto/rea/commit/8b4c96e6c6b3ed6806e4cd72b556b5db6114304a), [#1421](https://github.com/morluto/rea/issues/1421), [#1454](https://github.com/morluto/rea/issues/1454)) Thanks @Fazmin, @kaoru0822-kitauji, @N0zoM1z0.
### ⚠ BREAKING CHANGES
- WebSocket results: use `network.websocket_connections` instead of `network.websocket_events`. Connections contain `request_id`, `url` and `events`; allow `payload_bytes: null` when size is unknown. No legacy alias remains. ([db4e360](https://github.com/morluto/rea/commit/db4e3608465486466591b594cf77037ac150c988))
- Direct resource APIs: pass a caller-owned `ArtifactResourceScope` to inventory/extraction, JavaScript analysis, reference imports and web exports; retain it through cleanup/shutdown. For `readJavaScriptArtifactFiles`, scope is argument three and abort signal is argument four. Keep each root inventory source’s single cleanup owner. Use `OwnedFileHandle` owners, injected ASAR `fd`/`read` handles instead of `createReadStream`, VCS `Result` values, and `WebScriptExportError`’s options object. CLI/MCP inputs for the scoped operations are unchanged. ([ffe0574](https://github.com/morluto/rea/commit/ffe0574e17b9d973860371083cb1a351be912131), [53e7fad](https://github.com/morluto/rea/commit/53e7fadbd078422ba3d55b5499a70d237c5f4908), [684199b](https://github.com/morluto/rea/commit/684199ba0be7822266d57ddb8a40e95f087fdc42), [be75ed5](https://github.com/morluto/rea/commit/be75ed5e4b4bfb146af497f8fa5d172a1ea19462), [be6272f](https://github.com/morluto/rea/commit/be6272f26115bd8e94f2ab0df4cf092004587a18), [e0e49e9](https://github.com/morluto/rea/commit/e0e49e9dd53d9d1f1d2330b23f7f880dc1c81774), [95a77b6](https://github.com/morluto/rea/commit/95a77b6e25420b3879bd0e32bdbb14d452b38776))
- Saved reference graphs: exclusions require their matching `pattern` and distinguish `project-ignored` from `default-ignored`. Re-import graphs missing that provenance. ([6585293](https://github.com/morluto/rea/commit/6585293c562853b95c3a70df02c12ad8cc17afc4))
- Terminal captures: keep initial and resized buffers within 1,000,000 cells using `columns × (rows + scrollback)`; oversized scenarios are rejected before launch. ([f9ea9a9](https://github.com/morluto/rea/commit/f9ea9a9268c854d31048da9f8a50cfa37703fe5a))
- Optional agent-verification reports: accept schema version 4, nullable `expectedFirstTool` and `targetClarificationPassed`. Production CLI/MCP contracts are unchanged. ([#1613](https://github.com/morluto/rea/pull/1613))
- Capture comparisons: recapture both sides with matching `literal-replacements-v1` semantics when custom normalization patterns are used. Legacy unversioned pattern captures remain readable but cannot be compared with new pattern captures; captures without patterns are unaffected. ([#1456](https://github.com/morluto/rea/pull/1456), [2b6f14f](https://github.com/morluto/rea/commit/2b6f14f528297cf12c776c960af0859f5cfbefbf)) Thanks @Madhavan006, @N0zoM1z0.
- Artifact graph edges: read child artifact IDs and paths from the occurrence selected by `occurrence_id`, not edge fields `child_artifact_id`/`logical_path`. Earlier inventory Evidence no longer validates against the new commitment. ([#1493](https://github.com/morluto/rea/pull/1493)) Thanks @kaoru0822-kitauji.
- Semantic graph schema: nodes and relations no longer include `identifier_strategy`, and relation IDs no longer hash it. Earlier Evidence no longer validates against this schema. ([#1474](https://github.com/morluto/rea/pull/1474)) Thanks @kaoru0822-kitauji.
- Semantic identities: select IDs from the parent graph; read exact event names from `event_name`, endpoints from `endpoint` and literal values from `properties.value`. New analyses use digest-based IDs for long/reserved event and endpoint names and `value-sha256` literal role keys, with shorter labels; string literal labels now read `string literal`. Previously stored graphs retain their original IDs and remain valid workflow inputs. ([#1427](https://github.com/morluto/rea/pull/1427), [#1423](https://github.com/morluto/rea/pull/1423), [#1407](https://github.com/morluto/rea/pull/1407)) Thanks @N0zoM1z0.
### Contributors
- Thanks also to @BaaRai, @commit-regrets, @coygeek, @Franky100-pig, @inii-sayaa, @jopemachine, @JosephDemarest, @keemsisi, @Totopo27 for contributions included in this release.
Full contribution record: [336 commits and 163 merged PRs](https://github.com/morluto/rea/blob/rea-agents-6.4.0/docs/releases/6.4.0.contributions.json).
## [6.3.0](https://github.com/morluto/rea/compare/rea-agents-6.2.0...rea-agents-6.3.0) (2026-10-09)
+389 -184
View File
@@ -2004,9 +2004,20 @@ public final class ReaGhidraBridge extends HeadlessScript {
);
JumpTable.LoadTable[] loadTables = jumpTable.getLoadTables();
RecoveredAArch64RelativeTable recoveredAArch64 =
loadTables.length == 0
? recoverAArch64RelativeTable(jumpTable)
: null;
recoverAArch64RelativeTable(jumpTable);
if (recoveredAArch64 != null && loadTables.length > 0) {
// A decoded table only adds evidence when Ghidra's own load table
// describes the same bytes; any disagreement keeps Ghidra's view.
boolean agrees = false;
for (JumpTable.LoadTable loadTable : loadTables) {
if (loadTable.getAddress().equals(recoveredAArch64.tableAddress) &&
loadTable.getSize() == recoveredAArch64.entrySize &&
loadTable.getNum() == recoveredAArch64.mappings.size()) {
agrees = true;
}
}
if (!agrees) recoveredAArch64 = null;
}
List<Address> referencedDataAddresses = loadTables.length == 0
? dispatchPathDataReferences(jumpTable)
: List.of();
@@ -2040,7 +2051,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
source.add("evidence", evidence);
dataSources.add(source);
}
if (recoveredAArch64 != null) {
if (recoveredAArch64 != null && loadTables.length == 0) {
JsonObject source = new JsonObject();
source.addProperty(
"address",
@@ -2055,9 +2066,16 @@ public final class ReaGhidraBridge extends HeadlessScript {
inferenceEvidence(
"jump-table",
"ghidra-instruction-and-memory",
"Decoded " + recoveredAArch64.mappings.size() +
"Decoded " + recoveredAArch64.mappings.size() + " " +
(recoveredAArch64.signedEntries ? "signed" : "unsigned") +
" entries of size " + recoveredAArch64.entrySize + " bytes from " + canonicalAddress(recoveredAArch64.tableAddress) +
"; the AArch64 LDRB-or-LDRH/ADR/ADD/BR chain uses a selector guarded by an unsigned upper-bound branch."
"; the AArch64 ADRP/ADD table, indexed load, ADR base, ADD" +
(recoveredAArch64.offsetShift == 0 ? "" : " LSL #" + recoveredAArch64.offsetShift) +
"/BR chain uses a selector bounded by CMP #" + recoveredAArch64.compareImmediate +
" and " + recoveredAArch64.guardMnemonic + " on one straight-line path" +
(recoveredAArch64.nullTarget == null
? "."
: "; zero entries are routed by CBZ to " + canonicalAddress(recoveredAArch64.nullTarget) + ".")
)
);
source.add("evidence", evidence);
@@ -2096,6 +2114,33 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.add("data_sources", dataSources);
Address[] targets = jumpTable.getCases();
TypedJumpTableCases typedCases = typedJumpTableCases(jumpTable, caseTokens);
// Table indices identify the register bounded by CMP. Typed labels may
// instead identify a selector before normalization (for example, SUB
// index, selector, #10). Do not merge those distinct value identities.
if (recoveredAArch64 != null && recoveredAArch64.priorSelectorDefinition != null) {
typedCases.limitations.add(
"REA decoded the relative table, but the compared selector has an earlier register definition at " +
canonicalAddress(recoveredAArch64.priorSelectorDefinition) + "; its table indices are not proven to use the typed switch selector's case values. Ghidra's typed labels are retained and additional case labels remain unknown."
);
recoveredAArch64 = null;
}
if (recoveredAArch64 != null) {
for (Map.Entry<Long, Address> typed : typedCases.cases.entrySet()) {
long value = typed.getKey();
if (value < 0 || value >= recoveredAArch64.mappings.size()) {
typedCases.limitations.add("Typed case labels and decoded table indices do not use a verified common selector; Ghidra's typed labels are retained.");
recoveredAArch64 = null;
break;
}
RecoveredAArch64Mapping decoded = recoveredAArch64.mappings.get((int) value);
boolean divertedZero = decoded.nullEntry && typed.getValue().equals(recoveredAArch64.unreachableZeroTarget);
if (!typed.getValue().equals(decoded.targetAddress) && !divertedZero) {
typedCases.limitations.add("Typed case evidence and the decoded relative table disagree on a destination; Ghidra's typed labels are retained.");
recoveredAArch64 = null;
break;
}
}
}
JsonArray mappings = new JsonArray();
JsonArray defaultTargets = new JsonArray();
for (Address target : typedCases.defaults) {
@@ -2103,27 +2148,30 @@ public final class ReaGhidraBridge extends HeadlessScript {
}
Set<Address> coveredTargets = new HashSet<>(typedCases.cases.values());
coveredTargets.addAll(typedCases.defaults);
if (recoveredAArch64 != null && recoveredAArch64.unreachableZeroTarget != null) {
Address unreachable = recoveredAArch64.unreachableZeroTarget;
coveredTargets.add(unreachable);
for (RecoveredAArch64Mapping recovered : recoveredAArch64.mappings) {
if (recovered.nullEntry && unreachable.equals(typedCases.cases.get((long) recovered.tableIndex))) {
typedCases.cases.remove((long) recovered.tableIndex);
}
}
typedCases.limitations.add(
"Ghidra recovered destination " + canonicalAddress(unreachable) +
" as the zero-offset decode of the relative table, but the CBZ before the dispatch diverts zero entries to " +
canonicalAddress(recoveredAArch64.nullTarget) + ", so the indirect branch never reaches it for an in-range case."
);
}
if (recoveredAArch64 != null) {
for (int index = 0; index < recoveredAArch64.mappings.size(); index += 1) {
monitor.checkCancelled();
RecoveredAArch64Mapping recovered = recoveredAArch64.mappings.get(index);
if (
typedCases.unknownTargets.contains(recovered.targetAddress) ||
typedCases.ambiguousValues.contains((long) recovered.caseValue)
typedCases.ambiguousValues.contains((long) recovered.tableIndex)
) continue;
Address typedTarget = typedCases.cases.get((long) recovered.caseValue);
if (typedTarget != null && !typedTarget.equals(recovered.targetAddress)) {
typedCases.cases.remove((long) recovered.caseValue);
typedCases.ambiguousValues.add((long) recovered.caseValue);
typedCases.unknownTargets.add(typedTarget);
typedCases.unknownTargets.add(recovered.targetAddress);
typedCases.limitations.add(
"Typed case evidence and the verified relative table disagree on a destination; both destinations remain unknown for that case."
);
continue;
}
JsonObject mapping = new JsonObject();
mapping.addProperty("case_value", recovered.caseValue);
mapping.addProperty("case_value", recovered.tableIndex);
String targetAddress = canonicalAddress(recovered.targetAddress);
mapping.addProperty("target_address", targetAddress);
mapping.addProperty("confidence", "high");
@@ -2132,14 +2180,18 @@ public final class ReaGhidraBridge extends HeadlessScript {
inferenceEvidence(
"jump-table",
"ghidra-instruction-and-memory",
"Case value " + recovered.caseValue + (recoveredAArch64.entrySize == 1 ? " indexes byte " : " indexes halfword at byte offset ") +
(recovered.caseValue * recoveredAArch64.entrySize) + " at " + canonicalAddress(recoveredAArch64.tableAddress) +
"; the loaded unsigned entry selects a four-byte instruction offset from " +
canonicalAddress(recoveredAArch64.branchBase) + ", reaching " + targetAddress + "."
"Case value " + recovered.tableIndex + (recoveredAArch64.entrySize == 1 ? " indexes byte " : " indexes halfword at byte offset ") +
(recovered.tableIndex * recoveredAArch64.entrySize) + " at " + canonicalAddress(recoveredAArch64.tableAddress) +
(recovered.nullEntry
? "; the entry is zero, so the CBZ before the dispatch branches to " + targetAddress + "."
: "; the loaded " + (recoveredAArch64.signedEntries ? "signed" : "unsigned") +
" entry " + recovered.encodedEntry + " selects a " +
(recoveredAArch64.offsetShift == 2 ? "four-byte instruction" : "byte") +
" offset from " + canonicalAddress(recoveredAArch64.branchBase) + ", reaching " + targetAddress + ".")
)
);
mapping.add("evidence", evidence);
if (!typedCases.cases.containsKey((long) recovered.caseValue)) {
if (!typedCases.cases.containsKey((long) recovered.tableIndex)) {
mappings.add(mapping);
}
coveredTargets.add(recovered.targetAddress);
@@ -2213,7 +2265,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
);
}
if (recoveredAArch64 == null && loadTables.length == 0 && currentProgram.getLanguageID().getIdAsString().startsWith("AARCH64:")) {
limitations.add("REA did not verify a supported byte/halfword relative dispatch chain with exact unsigned bounds and register definitions; unsupported encodings and ambiguous bounds remain unknown.");
limitations.add("REA did not verify a supported signed or unsigned byte/halfword relative dispatch chain with exact unsigned bounds and register definitions; unsupported encodings and ambiguous bounds remain unknown.");
}
if (targets.length == 0) {
limitations.add(
@@ -2343,12 +2395,21 @@ public final class ReaGhidraBridge extends HeadlessScript {
}
private static final class RecoveredAArch64Mapping {
private final int caseValue;
private final int tableIndex;
private final Address targetAddress;
private final long encodedEntry;
private final boolean nullEntry;
private RecoveredAArch64Mapping(int caseValue, Address targetAddress) {
this.caseValue = caseValue;
private RecoveredAArch64Mapping(
int tableIndex,
Address targetAddress,
long encodedEntry,
boolean nullEntry
) {
this.tableIndex = tableIndex;
this.targetAddress = targetAddress;
this.encodedEntry = encodedEntry;
this.nullEntry = nullEntry;
}
}
@@ -2356,24 +2417,70 @@ public final class ReaGhidraBridge extends HeadlessScript {
private final Address tableAddress;
private final Address branchBase;
private final int entrySize;
private final Address priorSelectorDefinition;
private final boolean signedEntries;
private final int offsetShift;
private final String guardMnemonic;
private final long compareImmediate;
private final Address nullTarget;
private final Address unreachableZeroTarget;
private final List<RecoveredAArch64Mapping> mappings;
private RecoveredAArch64RelativeTable(
Address tableAddress,
Address branchBase,
int entrySize,
Address priorSelectorDefinition,
boolean signedEntries,
int offsetShift,
String guardMnemonic,
long compareImmediate,
Address nullTarget,
Address unreachableZeroTarget,
List<RecoveredAArch64Mapping> mappings
) {
this.tableAddress = tableAddress;
this.branchBase = branchBase;
this.entrySize = entrySize;
this.priorSelectorDefinition = priorSelectorDefinition;
this.signedEntries = signedEntries;
this.offsetShift = offsetShift;
this.guardMnemonic = guardMnemonic;
this.compareImmediate = compareImmediate;
this.nullTarget = nullTarget;
this.unreachableZeroTarget = unreachableZeroTarget;
this.mappings = mappings;
}
}
private static final Pattern AARCH64_DISPATCH_ADD = Pattern.compile(
"(?i)^([wx]\\d+),\\s*([wx]\\d+),\\s*([wx]\\d+)(?:,\\s*LSL\\s*#(?:0x)?([0-9a-f]+))?$"
);
private static final Pattern AARCH64_TABLE_LOAD = Pattern.compile(
"(?i)^([wx]\\d+),\\s*\\[([wx]\\d+),\\s*([wx]\\d+)(?:,\\s*(LSL|UXTW)\\s*(?:#(?:0x)?([0-9a-f]+))?)?\\s*\\]$"
);
private static final Pattern AARCH64_REGISTER_AND_ADDRESS = Pattern.compile(
"(?i)^([wx]\\d+),\\s*(0x[0-9a-f]+)$"
);
private static final Pattern AARCH64_ADD_IMMEDIATE = Pattern.compile(
"(?i)^([wx]\\d+),\\s*([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$"
);
private static final Pattern AARCH64_COMPARE_IMMEDIATE = Pattern.compile(
"(?i)^([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$"
);
private static final Pattern AARCH64_MOVE_REGISTER = Pattern.compile(
"(?i)^([wx]\\d+),\\s*([wx]\\d+)$"
);
private static final Set<String> AARCH64_FLAG_PRESERVING =
Set.of("MOV", "ADRP", "ADR", "ADD");
/**
* Recover only the directly verified AArch64 byte/halfword-indexed relative branch
* table form. Other instruction sequences keep their case labels unknown.
* Recover only directly verified AArch64 relative branch tables:
* ADRP/ADD table base, LDRB/LDRH/LDRSB/LDRSH entry load with an exact
* index scale, ADR branch base, optional CBZ on the loaded entry, and
* ADD (byte or LSL #2 scale)/BR, with a selector bounded by CMP #imm and an
* unsigned B.HI or B.CS/B.HS guard on one straight-line path. Any other
* instruction sequence keeps its case labels unknown.
*/
private RecoveredAArch64RelativeTable recoverAArch64RelativeTable(
JumpTable jumpTable
@@ -2395,167 +2502,265 @@ public final class ReaGhidraBridge extends HeadlessScript {
if (instruction.getAddress().equals(jumpTable.getSwitchAddress())) break;
}
int branchIndex = instructions.size() - 1;
if (branchIndex < 1 ||
!instructions.get(branchIndex).getMnemonicString().equalsIgnoreCase("BR")) {
if (branchIndex < 1 || !mnemonic(instructions.get(branchIndex)).equals("BR")) {
return null;
}
Set<String> recoveredTargetSet = new HashSet<>();
for (Address target : jumpTable.getCases()) recoveredTargetSet.add(canonicalAddress(target));
if (recoveredTargetSet.isEmpty()) return null;
return matchAArch64RelativeDispatch(
instructions,
branchIndex,
recoveredTargetSet,
dispatchPathDataReferences(jumpTable)
);
}
private RecoveredAArch64RelativeTable matchAArch64RelativeDispatch(
List<Instruction> instructions,
int branchIndex,
Set<String> recoveredTargetSet,
List<Address> dataAddresses
) throws Exception {
String branchRegister = normalizedRegister(operands(instructions.get(branchIndex)));
// ADD Xd, Xn, Xm{, LSL #s}; BR Xd. LSL applies to Xm only.
Instruction dispatchAdd = instructions.get(branchIndex - 1);
if (!mnemonic(dispatchAdd).equals("ADD")) return null;
java.util.regex.Matcher addMatcher = AARCH64_DISPATCH_ADD.matcher(operands(dispatchAdd));
if (!addMatcher.matches() || !addMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) return null;
if (!normalizedRegister(addMatcher.group(1)).equals(branchRegister)) return null;
int offsetShift = addMatcher.group(4) == null ? 0 : Integer.parseInt(addMatcher.group(4), 16);
if (offsetShift != 0 && offsetShift != 2) return null;
String firstSource = normalizedRegister(addMatcher.group(2));
String secondSource = normalizedRegister(addMatcher.group(3));
if (firstSource.equals(secondSource)) return null;
// The entry register is the source whose latest definition is the
// table load; the other source must be defined by ADR.
int loadIndex = -1;
String entryRegister = null;
String baseRegister = null;
for (String candidate : offsetShift == 0 ? List.of(secondSource, firstSource) : List.of(secondSource)) {
int writer = latestWriter(instructions, branchIndex - 1, candidate);
if (writer >= 0 && Set.of("LDRB", "LDRH", "LDRSB", "LDRSH").contains(mnemonic(instructions.get(writer)))) {
loadIndex = writer;
entryRegister = candidate;
baseRegister = candidate.equals(firstSource) ? secondSource : firstSource;
break;
}
}
if (loadIndex < 0) return null;
Instruction load = instructions.get(loadIndex);
String loadMnemonic = mnemonic(load);
int entrySize = loadMnemonic.endsWith("B") ? 1 : 2;
boolean signedEntries = loadMnemonic.startsWith("LDRS");
java.util.regex.Matcher loadMatcher = AARCH64_TABLE_LOAD.matcher(operands(load));
if (!loadMatcher.matches()) return null;
// A signed load into a W register sign-extends only to 32 bits; the
// 64-bit dispatch ADD would then read a zero-extended value.
if (signedEntries && !loadMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) return null;
String tableBaseRegister = normalizedRegister(loadMatcher.group(2));
String indexRegister = normalizedRegister(loadMatcher.group(3));
boolean wordIndex = loadMatcher.group(3).toLowerCase(Locale.ROOT).startsWith("w");
String extend = loadMatcher.group(4) == null ? null : loadMatcher.group(4).toUpperCase(Locale.ROOT);
int indexShift = loadMatcher.group(5) == null ? 0 : Integer.parseInt(loadMatcher.group(5), 16);
if (wordIndex ? !"UXTW".equals(extend) : "UXTW".equals(extend)) return null;
if (indexShift != (entrySize == 1 ? 0 : 1)) return null;
// Only ADR and one CBZ on the entry may sit between load and ADD.
Address nullTarget = null;
for (int index = loadIndex + 1; index < branchIndex - 1; index += 1) {
Instruction instruction = instructions.get(index);
String name = mnemonic(instruction);
if (name.equals("ADR")) continue;
if (name.equals("CBZ") && nullTarget == null) {
String[] parts = operands(instruction).split(",");
Address[] flows = instruction.getFlows();
if (parts.length != 2 || flows.length != 1 ||
!normalizedRegister(parts[0]).equals(entryRegister)) return null;
nullTarget = flows[0];
continue;
}
return null;
}
Address[] recoveredTargets = jumpTable.getCases();
Set<String> recoveredTargetSet = new HashSet<>();
for (Address target : recoveredTargets) recoveredTargetSet.add(canonicalAddress(target));
List<Address> dataAddresses = dispatchPathDataReferences(jumpTable);
int baseIndex = latestWriter(instructions, branchIndex - 1, baseRegister);
if (baseIndex < 0 || !mnemonic(instructions.get(baseIndex)).equals("ADR")) return null;
java.util.regex.Matcher baseMatcher = AARCH64_REGISTER_AND_ADDRESS.matcher(operands(instructions.get(baseIndex)));
if (!baseMatcher.matches()) return null;
Address branchBase = addressFromOperand(baseMatcher.group(2));
for (int loadIndex = 0; loadIndex < branchIndex; loadIndex += 1) {
Instruction load = instructions.get(loadIndex);
int entrySize = load.getMnemonicString().equalsIgnoreCase("LDRB") ? 1 : load.getMnemonicString().equalsIgnoreCase("LDRH") ? 2 : 0;
if (entrySize == 0 || branchIndex != loadIndex + 2 || loadIndex < 3 ||
!instructions.get(loadIndex - 1).getMnemonicString().equalsIgnoreCase("ADR") ||
!instructions.get(loadIndex - 2).getMnemonicString().equalsIgnoreCase("ADD") ||
!instructions.get(loadIndex - 3).getMnemonicString().equalsIgnoreCase("ADRP")) continue;
String loadOperands = operands(load);
java.util.regex.Matcher loadMatcher = java.util.regex.Pattern
.compile(entrySize == 1 ? "(?i)^[wx]\\d+,\\s*\\[([wx]\\d+),\\s*([wx]\\d+)\\]$" : "(?i)^[wx]\\d+,\\s*\\[([wx]\\d+),\\s*([wx]\\d+),\\s*LSL\\s*#(?:0x)?1\\]$")
.matcher(loadOperands);
if (!loadMatcher.matches()) continue;
String loadedRegister = normalizedRegister(loadOperands.substring(0, loadOperands.indexOf(',')));
String tableBaseRegister = normalizedRegister(loadMatcher.group(1));
String indexRegister = normalizedRegister(loadMatcher.group(2));
int tableAddIndex = latestWriter(instructions, loadIndex, tableBaseRegister);
if (tableAddIndex < 0 || !mnemonic(instructions.get(tableAddIndex)).equals("ADD")) return null;
java.util.regex.Matcher tableAddMatcher = AARCH64_ADD_IMMEDIATE.matcher(operands(instructions.get(tableAddIndex)));
if (!tableAddMatcher.matches() ||
!normalizedRegister(tableAddMatcher.group(1)).equals(tableBaseRegister) ||
!normalizedRegister(tableAddMatcher.group(2)).equals(tableBaseRegister)) return null;
int tablePageIndex = latestWriter(instructions, tableAddIndex, tableBaseRegister);
if (tablePageIndex < 0 || !mnemonic(instructions.get(tablePageIndex)).equals("ADRP")) return null;
java.util.regex.Matcher pageMatcher = AARCH64_REGISTER_AND_ADDRESS.matcher(operands(instructions.get(tablePageIndex)));
if (!pageMatcher.matches() || !normalizedRegister(pageMatcher.group(1)).equals(tableBaseRegister)) return null;
Address tableAddress = addressFromOperand(pageMatcher.group(2)).add(parseInteger(tableAddMatcher.group(3)));
if (!currentProgram.getMemory().contains(tableAddress) ||
(!dataAddresses.isEmpty() && !dataAddresses.contains(tableAddress))) return null;
Address resolvedTableAddress = null;
for (int index = loadIndex - 3; index < loadIndex - 1; index += 1) {
Instruction instruction = instructions.get(index);
String mnemonic = instruction.getMnemonicString().toUpperCase(Locale.ROOT);
String text = operands(instruction);
if (mnemonic.equals("ADRP") &&
text.matches("(?i)^[wx]\\d+,\\s*0x[0-9a-f]+$")) {
String[] parts = text.split(",");
if (normalizedRegister(parts[0]).equals(tableBaseRegister)) {
resolvedTableAddress = addressFromOperand(parts[1]);
}
}
else if (mnemonic.equals("ADD") && resolvedTableAddress != null &&
text.matches("(?i)^[wx]\\d+,\\s*[wx]\\d+,\\s*#?(?:0x[0-9a-f]+|[0-9]+)$")) {
String[] parts = text.split(",");
if (normalizedRegister(parts[0]).equals(tableBaseRegister) &&
normalizedRegister(parts[1]).equals(tableBaseRegister)) {
resolvedTableAddress = resolvedTableAddress.add(parseInteger(parts[2].trim().replaceFirst("^#", "")));
}
}
// Unsigned guard: CMP Rs, #imm, then B.HI (imm + 1 entries) or
// B.CS/B.HS (imm entries), with only flag-preserving moves and
// address arithmetic between the compare and the load.
int guardIndex = -1;
for (int index = loadIndex - 1; index >= 0; index -= 1) {
String name = mnemonic(instructions.get(index));
if (name.startsWith("B.")) {
guardIndex = index;
break;
}
if (resolvedTableAddress == null || !currentProgram.getMemory().contains(resolvedTableAddress) ||
(!dataAddresses.isEmpty() && !dataAddresses.contains(resolvedTableAddress))) continue;
Address tableAddress = resolvedTableAddress;
Address branchBase = null;
boolean addressChain = false;
for (int index = loadIndex - 1; index < branchIndex; index += 1) {
Instruction instruction = instructions.get(index);
String mnemonic = instruction.getMnemonicString().toUpperCase(Locale.ROOT);
String text = operands(instruction);
if (index < loadIndex && mnemonic.equals("ADR") && text.matches("(?i)^[wx]\\d+,\\s*0x[0-9a-f]+$")) {
int comma = text.indexOf(',');
String destination = normalizedRegister(text.substring(0, comma));
if (!destination.equals(normalizedRegister(operands(instructions.get(branchIndex))))) continue;
branchBase = addressFromOperand(text.substring(comma + 1));
}
if (index > loadIndex && mnemonic.equals("ADD") && branchBase != null &&
text.matches("(?i)^[wx]\\d+,\\s*[wx]\\d+,\\s*[wx]\\d+,\\s*LSL\\s*#(?:0x)?2$")) {
String[] parts = text.split(",");
String destination = normalizedRegister(parts[0]);
String base = normalizedRegister(parts[1]);
String offset = normalizedRegister(parts[2]);
if (destination.equals(base) &&
base.equals(normalizedRegister(operands(instructions.get(branchIndex)))) &&
offset.equals(loadedRegister)) {
addressChain = true;
}
}
}
if (!addressChain || branchBase == null) continue;
int entryCount = -1;
Address defaultTarget = null;
for (int compareIndex = 0; compareIndex < loadIndex; compareIndex += 1) {
Instruction compare = instructions.get(compareIndex);
if (!compare.getMnemonicString().equalsIgnoreCase("CMP")) continue;
java.util.regex.Matcher compareMatcher = java.util.regex.Pattern
.compile("(?i)^([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$")
.matcher(operands(compare));
if (!compareMatcher.matches()) continue;
String comparedRegister = normalizedRegister(compareMatcher.group(1));
long upperBound = parseInteger(compareMatcher.group(2));
if (upperBound < 0 || upperBound >= 4096) continue;
for (int guardIndex = compareIndex + 1; guardIndex < loadIndex; guardIndex += 1) {
Instruction guard = instructions.get(guardIndex);
String mnemonic = guard.getMnemonicString().toUpperCase(Locale.ROOT);
if (!mnemonic.equals("B.HI")) continue;
Address[] flows = guard.getFlows();
if (flows.length != 1 || !recoveredTargetSet.contains(canonicalAddress(flows[0]))) continue;
boolean supportedSequence = true;
for (int sequenceIndex = compareIndex + 1; sequenceIndex < loadIndex; sequenceIndex += 1) {
String sequenceMnemonic = instructions.get(sequenceIndex)
.getMnemonicString().toUpperCase(Locale.ROOT);
if (!Set.of("B.HI", "MOV", "ADRP", "ADD", "ADR").contains(sequenceMnemonic)) {
supportedSequence = false;
break;
}
}
Set<String> selectorAliases = new HashSet<>();
selectorAliases.add(comparedRegister);
Set<String> boundedIndexes = new HashSet<>();
if (compareMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) boundedIndexes.add(comparedRegister);
for (int sequenceIndex = compareIndex + 1; sequenceIndex < loadIndex; sequenceIndex++) {
Instruction instruction = instructions.get(sequenceIndex);
String[] parts = operands(instruction).split(",");
if (parts.length == 0) continue;
if (instruction.getMnemonicString().equalsIgnoreCase("MOV")) {
String destination = normalizedRegister(parts[0]);
String source = parts.length == 2 ? normalizedRegister(parts[1]) : "";
boolean aliasesSelector = selectorAliases.contains(source);
boolean fullWidthBounded = boundedIndexes.contains(source) ||
(parts[0].trim().toLowerCase(Locale.ROOT).startsWith("w") && aliasesSelector);
if (aliasesSelector) selectorAliases.add(destination);
else selectorAliases.remove(destination);
if (fullWidthBounded) boundedIndexes.add(destination);
else boundedIndexes.remove(destination);
} else if (Set.of("ADRP", "ADD", "ADR").contains(instruction.getMnemonicString().toUpperCase(Locale.ROOT))) {
selectorAliases.remove(normalizedRegister(parts[0]));
boundedIndexes.remove(normalizedRegister(parts[0]));
}
}
boolean selectorFeedsTable = boundedIndexes.contains(indexRegister);
if (compareIndex < loadIndex - 3 && selectorFeedsTable && supportedSequence) {
entryCount = (int) upperBound + 1;
defaultTarget = flows[0];
break;
}
}
if (entryCount > 0) break;
}
if (entryCount <= 0 || entryCount > 4096 || defaultTarget == null) continue;
List<RecoveredAArch64Mapping> mappings = new ArrayList<>();
Set<String> mappedTargets = new HashSet<>();
boolean valid = true;
for (int caseValue = 0; caseValue < entryCount; caseValue += 1) {
Address entry = tableAddress.add((long) caseValue * entrySize);
int encodedOffset = entrySize == 1 ? currentProgram.getMemory().getByte(entry) & 0xff : currentProgram.getMemory().getShort(entry) & 0xffff;
Address target = branchBase.add((long) encodedOffset * 4L);
String canonicalTarget = canonicalAddress(target);
if (!recoveredTargetSet.contains(canonicalTarget)) {
valid = false;
break;
}
mappedTargets.add(canonicalTarget);
mappings.add(new RecoveredAArch64Mapping(caseValue, target));
}
if (!valid || mappedTargets.contains(canonicalAddress(defaultTarget))) continue;
Set<String> accountedTargets = new HashSet<>(mappedTargets);
accountedTargets.add(canonicalAddress(defaultTarget));
if (!accountedTargets.equals(recoveredTargetSet)) continue;
return new RecoveredAArch64RelativeTable(tableAddress, branchBase, entrySize, List.copyOf(mappings));
if (!AARCH64_FLAG_PRESERVING.contains(name)) return null;
}
return null;
if (guardIndex < 1) return null;
Instruction guard = instructions.get(guardIndex);
String guardMnemonic = mnemonic(guard);
if (!Set.of("B.HI", "B.CS", "B.HS").contains(guardMnemonic)) return null;
Address[] guardFlows = guard.getFlows();
if (guardFlows.length != 1) return null;
Address defaultTarget = guardFlows[0];
int compareIndex = -1;
for (int index = guardIndex - 1; index >= 0; index -= 1) {
String name = mnemonic(instructions.get(index));
if (name.equals("CMP")) {
compareIndex = index;
break;
}
if (!AARCH64_FLAG_PRESERVING.contains(name)) return null;
}
if (compareIndex < 0) return null;
java.util.regex.Matcher compareMatcher = AARCH64_COMPARE_IMMEDIATE.matcher(operands(instructions.get(compareIndex)));
if (!compareMatcher.matches()) return null;
String comparedRegister = normalizedRegister(compareMatcher.group(1));
boolean comparedFullWidth = compareMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x");
long compareImmediate = parseInteger(compareMatcher.group(2));
long entryCountLong = guardMnemonic.equals("B.HI") ? compareImmediate + 1 : compareImmediate;
if (entryCountLong <= 0 || entryCountLong > 4096) return null;
int entryCount = (int) entryCountLong;
// The loaded index must carry the compared value: either the compared
// register itself, unchanged since CMP, or a MOV copy made after CMP.
// A W compare bounds only the low 32 bits, so an X index needs a
// zero-extending W copy; a W index is zero-extended by UXTW.
int indexWriter = latestWriter(instructions, loadIndex, indexRegister);
boolean bounded = false;
if (indexWriter < compareIndex) {
bounded = indexRegister.equals(comparedRegister) && (wordIndex || comparedFullWidth);
}
else if (mnemonic(instructions.get(indexWriter)).equals("MOV")) {
java.util.regex.Matcher moveMatcher = AARCH64_MOVE_REGISTER.matcher(operands(instructions.get(indexWriter)));
if (moveMatcher.matches() &&
normalizedRegister(moveMatcher.group(2)).equals(comparedRegister) &&
latestWriter(instructions, indexWriter, comparedRegister) < compareIndex) {
boolean wordCopy = moveMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("w");
bounded = wordCopy || comparedFullWidth;
}
}
if (!bounded) return null;
// Every fact above holds only on one straight-line fall-through path
// with no other entry into the window.
int windowStart = Math.min(Math.min(tablePageIndex, baseIndex), compareIndex);
for (int index = windowStart + 1; index <= branchIndex; index += 1) {
monitor.checkCancelled();
Instruction previous = instructions.get(index - 1);
Instruction current = instructions.get(index);
Address fallThrough = previous.getFallThrough();
if (fallThrough == null || !fallThrough.equals(current.getAddress())) return null;
ReferenceIterator incoming = currentProgram.getReferenceManager().getReferencesTo(current.getAddress());
while (incoming.hasNext()) {
if (incoming.next().getReferenceType().isFlow()) return null;
}
}
List<RecoveredAArch64Mapping> mappings = new ArrayList<>();
Set<String> mappedTargets = new HashSet<>();
for (int tableIndex = 0; tableIndex < entryCount; tableIndex += 1) {
monitor.checkCancelled();
Address entry = tableAddress.add((long) tableIndex * entrySize);
if (!currentProgram.getMemory().contains(entry)) return null;
long encoded;
if (entrySize == 1) {
byte value = currentProgram.getMemory().getByte(entry);
encoded = signedEntries ? value : value & 0xffL;
}
else {
short value = currentProgram.getMemory().getShort(entry);
encoded = signedEntries ? value : value & 0xffffL;
}
if (encoded == 0 && nullTarget != null) {
mappings.add(new RecoveredAArch64Mapping(tableIndex, nullTarget, encoded, true));
continue;
}
Address target;
try {
target = branchBase.addNoWrap(encoded << offsetShift);
}
catch (ghidra.program.model.address.AddressOverflowException overflow) {
return null;
}
String canonicalTarget = canonicalAddress(target);
if (!recoveredTargetSet.contains(canonicalTarget)) return null;
mappedTargets.add(canonicalTarget);
mappings.add(new RecoveredAArch64Mapping(tableIndex, target, encoded, false));
}
if (mappedTargets.contains(canonicalAddress(defaultTarget))) return null;
// Ghidra's recovered destinations must be fully accounted for by the
// decoded entries, the guard default, and the CBZ null path.
Set<String> accountedTargets = new HashSet<>(mappedTargets);
accountedTargets.add(canonicalAddress(defaultTarget));
if (nullTarget != null) accountedTargets.add(canonicalAddress(nullTarget));
// Ghidra does not model the CBZ, so it may decode a zero entry as a
// branch to the base itself. Only a zero entry encodes offset 0, and
// the CBZ diverts every zero entry, so the BR never reaches the base.
Address unreachableZeroTarget = null;
boolean divertedZeroEntry = mappings.stream().anyMatch(mapping -> mapping.nullEntry);
if (divertedZeroEntry &&
recoveredTargetSet.contains(canonicalAddress(branchBase)) &&
!mappedTargets.contains(canonicalAddress(branchBase)) &&
!branchBase.equals(defaultTarget) &&
!branchBase.equals(nullTarget)) {
unreachableZeroTarget = branchBase;
accountedTargets.add(canonicalAddress(branchBase));
}
if (!accountedTargets.containsAll(recoveredTargetSet)) return null;
int selectorWriter = latestWriter(instructions, compareIndex, comparedRegister);
return new RecoveredAArch64RelativeTable(
tableAddress,
branchBase,
entrySize,
selectorWriter < 0 ? null : instructions.get(selectorWriter).getAddress(),
signedEntries,
offsetShift,
guardMnemonic,
compareImmediate,
nullTarget,
unreachableZeroTarget,
List.copyOf(mappings)
);
}
private static String mnemonic(Instruction instruction) {
return instruction.getMnemonicString().toUpperCase(Locale.ROOT);
}
/** Index of the last instruction before {@code before} that writes {@code register}, or -1. */
private static int latestWriter(List<Instruction> instructions, int before, String register) {
for (int index = before - 1; index >= 0; index -= 1) {
for (Object result : instructions.get(index).getResultObjects()) {
if (result instanceof Register written &&
normalizedRegister(written.getBaseRegister().getName()).equals(register)) {
return index;
}
}
}
return -1;
}
private static String operands(Instruction instruction) {
+23
View File
@@ -24,6 +24,29 @@ REA can attach to a user-owned Chrome-family browser through the Chrome DevTools
Target discovery returns the complete in-scope target array in one result.
## Screenshot comparison memory budget
`compare_web_screenshots` admits both PNGs against a 256 MiB estimated
working-memory budget before inflating either image. This is a processing
buffer budget for this operation, not a PNG dimension limit, CDP transport
limit, or process RSS guarantee. Header-valid dimensions that exceed the
budget return `resource_constraint` with `details.resource: "memory"` and the
estimated and maximum working-set bytes. Invalid PNG structure remains
malformed input; encodings outside supported pixel-comparison coverage remain
unsupported-target errors.
The conservative estimate includes the two decoder-owned decoded input
buffers, the current image's concatenated IDAT buffer, Node zlib's default
16 KiB output chunk backing rounded up for retained chunks, the concatenated
inflate result while its output chunks are retained, one possible extra output
chunk before zlib checks `maxOutputLength`, unfiltered scanline storage,
RGB-to-RGBA expansion, and the first retained RGBA image while the second is
decoded. RGBA output aliases its unfiltered buffer and is counted once.
Admission uses checked integer arithmetic before image-sized allocation.
Caller-owned JSON/base64 strings, input parsing and validation, transport
buffers, garbage-collector behavior, and other runtime memory are outside this
estimate; the budget does not promise a bound on total process memory.
Electron `file://` pages use a separate provider and target boundary; see [electron-observation.md](electron-observation.md).
## Provider authorities
+64 -35
View File
@@ -1,16 +1,25 @@
# Optional NativeAOT metadata recovery
REA integrates [Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot)
through an optional headless adapter. The original source is an opt-in submodule,
pinned to `effeb734fc570c32650f88b159608979dc7b423e`; its MIT license and existing
source notices are retained. No upstream source or algorithms are rewritten.
REA has a built-in read-only snapshot parser for x64 PE targets with .NET 8 RTR
9.1 metadata. It parses the exact admitted Ghidra session snapshot and uses an
in-memory overlay for rehydrated metadata; it does not define Ghidra DataTypes,
labels, functions, or memory bytes. The hydration semantics are adapted from
[Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot) at
`effeb734fc570c32650f88b159608979dc7b423e` (MIT), cross-checked against
.NET's `MethodTable.h` at runtime commit
`a3b2d40328be0be3f8dd43f0e5cc925b8827b044` (MIT). The required notice ships in
[`licenses/nativeaot-parser-MIT.txt`](../licenses/nativeaot-parser-MIT.txt).
The initial tested layout is .NET runtime **8.0.22**, NativeAOT RTR **9.1**,
x86-64 ELF and native Windows PE **targets analyzed on Linux x64**, using
Ghidra **12.1.4** and JDK **21**. The build accepts any Ghidra 12.1.x release and
javac 21 or newer, and it compiles with `--release 21`. Real verification of
this layout remains the Ghidra 12.1.4 and JDK 21 pair. Other NativeAOT layouts,
architectures, and hosts are unsupported.
The optional headless Ghidra adapter remains available for database annotations
and broader recovery. It uses the same pinned upstream commit as an opt-in
submodule and is a separate, more capable workflow.
The parser currently handles x64 PE32+ and RTR 9.1, including bounded
DEHYDRATED_DATA processing, MethodTable relationships, virtual slots, and
frozen strings. Other RTR versions and PE architectures are reported as
unsupported or partial. The matching .NET 8.0.22 fixture is source-built and
the parser's Windows-host Ghidra workflow has not yet had a real Windows-host
run. This does not establish Windows Ghidra import or decompiler behavior.
Windows Ghidra P0 has no database mutation authority. A PE/CLI or ReadyToRun
assembly should use `inspect_managed_artifact`; it is not NativeAOT.
@@ -43,42 +52,62 @@ arbitrary caller-supplied bytes. The bounded loader accepts regular JARs up to
## Use existing tools
Open the executable with provider `ghidra`. `inspect_native_load_image` returns
optional `observations.metadata_recovery` inline, including the detected header,
layout, recovery status, type category paths/addresses, diagnostics, annotation
coverage and derived-memory SHA. The enclosing `unsupported` load-image status
for PE/ELF concerns independent DOS loader verification; metadata recovery has
its own status. It is not a failed import.
Open the executable with provider `ghidra`. For an x64 PE, `inspect_native_load_image`
returns `observations.metadata_recovery` inline from the captured source snapshot,
including the RTR header, MethodTable addresses, frozen strings, derived overlay
identity, and examined/recovered counts. Its enclosing `unsupported` load-image
status means independent PE load-image verification was not performed; it does
not mean the metadata parser failed or the executable failed to load.
Pass one returned type path or its typed address to `inspect_native_data_type`.
Its optional `metadata_recovery` includes related/base type, implemented
interfaces and virtual slots with targets and known procedure names. Follow
these with `analyze_function`, string searches and ordinary cross-references.
For CLI use:
Pass a returned MethodTable address to `inspect_native_data_type`. A direct PE
query also derives the snapshot metadata from the authenticated Ghidra session
handshake, so it does not require a preceding load-image call. In a persistent
MCP session, load-image followed by type inspection reuses that same derivation.
The database result remains `unavailable` when Ghidra has no DataType at that
address; its `metadata_recovery.source` is `read-only-derived-overlay` and
carries related type, interface, slot, and target observations. Generated
address identities do not claim original managed names. The parser does not
merge frozen literals into Ghidra's database string inventory. Follow slot
targets with `analyze_function` and ordinary cross-references.
The built-in CLI path uses a MethodTable address returned by the parser or
another observation:
```sh
rea inspect-native-load-image ./NativeAotFixture --provider ghidra --json
rea inspect-native-data-type ./NativeAotFixture --type /NativeAOT/MethodTables/Class_ADDRESS_MT --provider ghidra --json
rea inspect-native-data-type ./NativeAotFixture --address 0xMETHODTABLE_ADDRESS --provider ghidra --json
rea function ./NativeAotFixture 0xADDRESS --provider ghidra --json
```
No new MCP workflow or approval flag is required. Configuration opts into the
adapter; without it ordinary Ghidra profiles/results remain unchanged.
The generated `/NativeAOT/...` category path belongs to the optional headless
adapter. Use `--type /NativeAOT/MethodTables/Class_ADDRESS_MT` only when that
adapter is configured and has created the corresponding Ghidra DataType.
Built-in PE analysis includes its parser contract and resource policy in the
Ghidra analysis profile; non-PE profiles do not include that contract.
## Evidence boundaries
Discovery prefers an imported RTR symbol, otherwise the upstream initialized,
non-executable-memory signature heuristic. REA rejects ambiguous candidates,
unknown layout versions, malformed directory rows, unsupported flags and
unmapped ranges before recovery. It requires the verified dehydrated layout.
The built-in parser uses an initialized non-executable PE signature scan and
requires a unique RTR 9.1 header and exactly one DEHYDRATED_DATA row. Rehydrated
bytes live only in a parser-owned overlay, identified by virtual address, size,
and SHA-256; the caller's source Buffer remains unchanged. Limits derive from
captured section extents and format field bounds. Separate resource policies
admit up to 128 MiB of captured PE bytes, 64 MiB-equivalent parser work, 64 MiB
of exact parser-owned overlay/index Buffers, and 4 MiB of serialized NativeAOT
report JSON. These are local parser limits, independent of MCP delivery; the
buffer counter does not claim to cap V8 heap or total process memory. Rows that
do not fit the report budget are omitted with explicit partial status and
coverage. Coverage counts describe the pointer scan, recovered MethodTables,
and frozen-string candidates examined, not every possible runtime type or
object. Cancellation is honored during snapshot I/O and checked immediately
before and after parsing; the synchronous parser call itself is bounded but
cannot be interrupted mid-call. Parser output never claims that a Ghidra
database type was defined.
The adapter performs rehydration, method-table recovery and frozen-object
annotation in one ephemeral Program transaction. Fatal failure/cancellation
rolls it back. Recovery may change analysis-memory bytes, never the original
executable file. Derived ranges have a digest and **no original file offset**;
they are not captured runtime state. Candidate/committed instance annotation
counts distinguish partial frozen-object coverage. Coverage concerns the
recovered candidate set, not every possible object or every runtime type.
The optional adapter performs its annotations in an ephemeral Program
transaction. That workflow may change analysis-memory bytes, never the original
executable file. It is unavailable on Windows P0, which has no database
mutation authority.
REA preserves pre-recovery calling conventions and uses the loaded compiler
specification default for newly created methods. The upstream universal
+3 -3
View File
@@ -8,9 +8,9 @@ Executable bytes remain unchanged.
Use [installation](installation.md#ghidra) for supported hosts, prerequisites,
configuration, startup deadlines, resource controls, and cleanup recovery.
[Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary;
[NativeAOT recovery](ghidra-nativeaot.md) is an optional, separately verified
extension. Check the [release boundary](installation.md#released-package-and-main)
when using npm.
[NativeAOT observations](ghidra-nativeaot.md) include a built-in read-only PE
snapshot parser and a separate optional annotation extension. Check the
[release boundary](installation.md#released-package-and-main) when using npm.
See [import language and analysis seeds](installation.md#ghidra-import-language-and-analysis-seeds)
to override the header-detected language or seed analysis.
+1 -1
View File
@@ -880,7 +880,7 @@ For a client that requires manual configuration, use:
"mcpServers": {
"rea": {
"command": "npx",
"args": ["-y", "rea-agents@6.3.0", "mcp"]
"args": ["-y", "rea-agents@6.4.0", "mcp"]
}
}
}
@@ -60,6 +60,12 @@ manifest and graph commitments, JavaScript Application Graph, static
Electron summary, reconstruction statistics, and explicit limitations. It does
not require a live Hopper, Ghidra, browser, or Electron process.
The application graph selects the shallowest `package.json` outside
`node_modules` as its package root, breaking equal-depth ties by Unicode
code-point order. Dependency manifests remain package observations but do not
declare Electron main or renderer entries. If only dependency manifests are
present, the inventory artifact remains the graph root.
## ASAR integrity
ASAR inventory checks Electron integrity metadata for embedded archive entries
@@ -247,6 +253,46 @@ while locally shadowed receivers can still contribute network endpoint
candidates. A template recovered with a parser error and no cooked value stays
dynamic; its raw spelling is not treated as a valid JavaScript string.
RPC client syntax also contributes network endpoint candidates, in both
`analyze_javascript_application` and `analyze_web_bundle`, distinguished by
`mechanism`:
- `trpc:procedure:<method>` records the static member path between the root
expression and the method, such as `invoice.list` in
`root.invoice.list.useQuery(...)`. tRPC proxies keep these property names
through identifier minification. React hooks and TanStack option factories
(`useQuery`, `useMutation`, `useInfiniteQuery`, `useSubscription`, suspense
and prefetch variants, `queryOptions`, `infiniteQueryOptions`,
`mutationOptions`, `subscriptionOptions`) are recognized directly. The
vanilla `query` and `mutate` methods are recognized on a root bound to
`createTRPCClient` or `createTRPCProxyClient` in the same lexical binding, or on a
path of at least two segments in a source containing `TRPCClientError`,
`trpc-accept`, or `/trpc`. Factory bindings must have one unconditional
initializer and no binding assignments; shadowed locals do not inherit client
identity. `subscribe` additionally requires a tRPC observer
argument (`onData`, `onError`, `onStarted`, `onStopped`, `onComplete`, or
`onConnectionStateChange`). `this` roots, browser globals, and any path
containing an RTK Query `endpoints` segment are never procedures; cache-key
helpers such as `queryKey` issue no request and are excluded.
- `trpc:link:<link>` records the literal `url` option of `httpLink`,
`httpBatchLink`, `httpBatchStreamLink`, and `httpSubscriptionLink` when the
link's callee name survives bundling, including `(0, module.httpBatchLink)`
import-call wrappers.
- `graphql:query`, `graphql:mutation`, and `graphql:subscription` record a
named operation found at the start of a string or cooked template literal —
the name must be followed by variable definitions (`($`), a directive, or a
selection set starting with a field name — or in a precompiled
`OperationDefinition` document object. A graphql-tag `loc.source.body` copy is
not reported twice. Anonymous operations are not reported.
A procedure path is the server-side procedure name, not a resolved URL; the
transport URL depends on the link configuration and batching. When the proxy is
reached through a member expression, such as a webpack module binding
(`l.S.invoice.list`) or a property holding the proxy (`o.api.post.byId`), the
recorded path keeps those leading carrier segments; the procedure path is its
suffix. Utility calls such as `useUtils().invoice.list.fetch()`, `useQueries`,
and links whose callee was renamed by a minifier are not recognized.
Each recovered bundle module retains the exact factory-source digest. A complete
bounded AST also receives a `babel-ast-v1` structural fingerprint that ignores
ordinary identifier names while retaining syntax, literals, operators, object
+17
View File
@@ -308,6 +308,23 @@ that reference for a summary, one section/module, or a stable page, or call
bundle exports stream canonical JSON into an atomically published file. A broad
follow-up or `get_evidence_bundle` can also exceed the response budget; exporting
preserves the complete session without sending it through a single MCP frame.
To discover existing records without their large payloads, call
`get_evidence_bundle` with `detail: "summary"`. Its `result.records` contain
Evidence IDs, artifact/provider identities, analysis-profile digests and native
procedure metadata. Optional `filters` match exact `evidence_id`, `operation`,
`target_sha256`, `analysis_profile_digest` or `procedure_address`. Use a discovered
ID with `inspect_analysis_view`; neither discovery nor retained selection starts
a provider. `retention: "complete-record"` describes stored Evidence, while
`native_dossier.value_flow` independently reports truncation, unavailable or
unknown metadata. Missing omission counts remain null. Summary output carries
every matching record; actual transport admission still applies. Omitted
`detail` preserves complete canonical bundle delivery; filters require summary.
For a portable exported bundle, the CLI shares this projection:
`rea inspect-evidence-bundle '{"path":"/tmp/evidence.json","detail":"summary","filters":{"operation":"analyze_function"}}' --format json`.
The CLI validates and materializes that local bundle under its existing memory
budget; summary delivery does not make arbitrary large files cheap to import.
Cancelling an export stops further serialization and removes its staging file.
The destination changes only when a complete export is atomically published.
File diff suppressed because it is too large Load Diff
+5 -1
View File
@@ -21,7 +21,11 @@ Use [the rea-changelog-update skill](https://github.com/morluto/rea/blob/main/.a
to turn the bot draft into grouped Highlights, Changes, and Fixes with inline
`Thanks @...` credit. `npm run release:notes -- inventory` collects the full
selected Git range and paginated GitHub PR/author/closing-issue metadata;
`record` produces the complete contribution record, `check` validates credits
`record --summary` produces a compact provenance link for public notes, while
`record` produces the full record for review. Credit a contribution once even
when it appears in highlights, details and migrations; omit maintainer
`@morluto` self-thanks. Keep complete authorship metadata in the inventory
instead of duplicating a credit ledger in the release body. `check` validates credits
and optional PR-body equality, and `render` writes the verified release section.
Run a subcommand without its required options to see its usage. GitHub reads
require an authenticated `gh`; record/check/render work offline from the saved
+16 -9
View File
@@ -450,15 +450,15 @@ installation (`application.java.min` through `application.java.max`). Current
12.1 releases require JDK 21 or newer and set no maximum. The lanes below still
prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
| Ghidra lane | Supported runner/target | Additional local tools |
| ------------------------------------------ | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:seeds` | POSIX host; source-owned 32-bit x86 raw fixture; function order and BSS labels | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE; `-- --x86` selects native x86 PE | Ghidra 12.1.4, full JDK 21, and the matching native artifact |
| Ghidra lane | Supported runner/target | Additional local tools |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:seeds` | POSIX host; source-owned 32-bit x86 raw fixture; function order and BSS labels | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE; `-- --x86` selects native x86 PE | Ghidra 12.1.4, full JDK 21, and the matching native artifact |
The optional Swift lane compiles a source-owned fixture and checks modern and
provider-demangled procedure names through real stdio MCP, with and without the
@@ -672,6 +672,13 @@ boundaries and budgets.
## Optional NativeAOT Ghidra analysis
The source parser lane is independent of the optional annotation JAR. Its
synthetic RTR 9.1 PE golden checks captured-byte digest admission, rehydration
into an immutable parser overlay, MethodTable/slot relationships, and frozen
literal extraction; provider tests exercise `inspect_native_load_image` and
address-based `inspect_native_data_type` with the database result still
unavailable. These tests do not establish Windows Ghidra host behavior.
This lane is separate from the default native lane. `build:fixtures:nativeaot`
requires an existing .NET SDK 8.0.416, the platform NativeAOT compiler/linker and
runtime pack 8.0.22. It builds benign sources into ignored `_reference/` and
+6 -3
View File
@@ -21,9 +21,12 @@ additional permission flag or degraded mode is required.
- the 25 read-only Ghidra inventory, memory, and function-analysis operations.
Loaded memory reads and file offsets preserve Ghidra's observed source mappings.
PE load-image inspection returns measurements with an explicit unsupported
attestation status; independent format-specific verification currently supports
DOS MZ only.
PE load-image inspection returns an explicit unsupported attestation status;
independent load-image verification currently supports DOS MZ only. For x64 PE
NativeAOT RTR 9.1, the read-only snapshot parser can report metadata through
`inspect_native_load_image` and `inspect_native_data_type` without a JAR or
database mutation. A real Windows-host NativeAOT Ghidra run is still required
to establish the complete host workflow.
This boundary does not establish general Windows feature parity. Ghidra GUI
state, annotations, other target architectures and formats, and Hopper-only
+28
View File
@@ -0,0 +1,28 @@
The MIT License (MIT)
=====================
Copyright © 2023-2025 Washi
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
The read-only NativeAOT parser adapts format and hydration semantics from
Washi1337/ghidra-nativeaot commit
effeb734fc570c32650f88b159608979dc7b423e (MIT). MethodTable field layout was
cross-checked against .NET runtime src/coreclr/nativeaot/Runtime/inc/MethodTable.h
at dotnet/runtime commit a3b2d40328be0be3f8dd43f0e5cc925b8827b044 (MIT).
+12
View File
@@ -1,5 +1,7 @@
#include "authority.hpp"
#include <delayimp.h>
namespace rea {
static const napi_type_tag RESOURCE_TAG = {0x45fcbf0b49e84fe3ULL, 0x9e3e5074130de843ULL};
@@ -121,3 +123,13 @@ static napi_value initialize(napi_env env, napi_value exports) {
} // namespace rea
NAPI_MODULE(NODE_GYP_MODULE_NAME, rea::initialize)
// Node-API symbols are delay-imported from "node.exe". Resolve them against the
// host executable so Node.js, Bun, and other Node-API hosts load the addon
// regardless of their executable file name.
static FARPROC WINAPI reaHostHook(unsigned event, PDelayLoadInfo info) {
if (event != dliNotePreLoadLibrary || _stricmp(info->szDll, "node.exe") != 0) return nullptr;
return reinterpret_cast<FARPROC>(GetModuleHandleW(nullptr));
}
decltype(__pfnDliNotifyHook2) __pfnDliNotifyHook2 = reaHostHook;
+3 -2
View File
@@ -219,9 +219,10 @@ std::unique_ptr<Runtime> createRuntime(const std::wstring& parentPath, const std
require(BCryptGenRandom(nullptr, random.data(), random.size(), BCRYPT_USE_SYSTEM_PREFERRED_RNG) == 0,
"Generate runtime identity failed", parentPath, ERROR_GEN_FAILURE);
auto result = std::make_unique<Runtime>();
result->path = parent->path + L"\\" + prefix + hex(random.data(), random.size());
const auto name = prefix + hex(random.data(), random.size());
result->path = parent->path + (parent->path.back() == L'\\' ? L"" : L"\\") + name;
Security security;
result->directory = createRelative(parent->get(), prefix + hex(random.data(), random.size()),
result->directory = createRelative(parent->get(), name,
DELETE | FILE_LIST_DIRECTORY | FILE_READ_ATTRIBUTES | READ_CONTROL,
true, security, result->path);
try {
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "rea-agents",
"version": "6.3.0",
"version": "6.4.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "rea-agents",
"version": "6.3.0",
"version": "6.4.0",
"license": "MIT",
"dependencies": {
"@babel/parser": "8.0.4",
+3 -2
View File
@@ -1,7 +1,7 @@
{
"name": "rea-agents",
"mcpName": "io.github.morluto/rea",
"version": "6.3.0",
"version": "6.4.0",
"description": "Reverse engineer anything from your terminal or agent with one CLI and MCP server.",
"license": "MIT",
"repository": {
@@ -81,7 +81,8 @@
"bridge/pwndbg/launch_impl.py",
"bridge/pwndbg/core_context.py",
"third_party/pwndbg/README.md",
"third_party/pwndbg/LICENSE.md"
"third_party/pwndbg/LICENSE.md",
"licenses/nativeaot-parser-MIT.txt"
],
"scripts": {
"release:notes": "node scripts/release-notes.mjs",
+3 -2
View File
@@ -84,7 +84,8 @@ if (!(await exists(join(headers, "node_api.h")))) {
}
const artifact = "rea-windows-x64.node";
// MSVC short-import libraries do not produce a correct Node import table with
// GNU ld. Generate a GNU import library for the exact stable Node-API symbols.
// GNU ld. Generate a GNU delay-import library for the exact stable Node-API
// symbols; addon.cc binds it to the host executable, whatever its file name.
const sources = ["addon.cc", "filesystem.cc", "process.cc"];
const sourceText = (
await Promise.all(
@@ -106,7 +107,7 @@ await writeFile(
definition,
`LIBRARY node.exe\nEXPORTS\n${symbols.join("\n")}\n`,
);
run("x86_64-w64-mingw32-dlltool", ["-d", definition, "-l", library]);
run("x86_64-w64-mingw32-dlltool", ["-d", definition, "-y", library]);
run(compiler, [
"-std=c++17",
"-O2",
+24
View File
@@ -14,6 +14,24 @@ const flags = [
"workflows",
...Object.keys(SCOPES),
];
// Actionlint validates workflow syntax only. The semantic suite asserts these
// workflows' release, publication and self-hosted runner safeguards.
const GUARDED_WORKFLOWS = new Set([
".github/workflows/release.yml",
".github/workflows/real-ghidra-windows.yml",
]);
// Sources compiled by build-conformance-fixtures.mjs. The readiness journey
// analyzes the built C fixture as well as its own applications.
const FIXTURE_SOURCES = [
"tests/conformance/c/**",
"tests/conformance/napi/**",
"tests/conformance/objc/**",
"tests/conformance/swift/**",
"tests/conformance/versions/**",
];
const READINESS_SOURCES = ["tests/conformance/readiness/**"];
const matchesAny = (file, patterns) =>
patterns.some((pattern) => matchesGlob(file, pattern));
const selected = Object.fromEntries(flags.map((flag) => [flag, false]));
const reasons = [];
const full = (reason) => {
@@ -122,6 +140,7 @@ if (
file === ".github/actionlint.yaml"
) {
selected.workflows = true;
if (GUARDED_WORKFLOWS.has(file)) selected.code = true;
if (file !== ".github/workflows/ci.yml") continue;
}
if (owners.length === 1 && owners[0] === "website") continue;
@@ -131,6 +150,11 @@ if (
continue;
}
selected.code = true;
if (matchesAny(file, FIXTURE_SOURCES)) {
selected.fixtures = true;
selected.readiness = true;
}
if (matchesAny(file, READINESS_SOURCES)) selected.readiness = true;
if (file.startsWith("tests/")) continue;
if (
/^(src\/process\/|src\/filesystem\/|src\/contracts\/tool|src\/server\/createServer|src\/cli\.ts|scripts\/ci\/)/.test(
@@ -0,0 +1,61 @@
/* Keep the configured launcher identity alive while its resource-limited
* Python child runs. Both remain in REA's owned process group. */
#define _POSIX_C_SOURCE 200809L
#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
int main(int argc, char **argv) {
char *python = getenv("REA_VERIFY_LIMITED_PYTHON");
char *limit = getenv("REA_VERIFY_LIMIT_OPTION");
if (python == NULL || python[0] != '/' || limit == NULL) {
fputs("limited Python fixture requires an absolute Python and limit\n", stderr);
return 127;
}
struct rlimit core = {0, 0};
if (setrlimit(RLIMIT_CORE, &core) != 0) {
perror("setrlimit");
return 127;
}
char **arguments = calloc((size_t)argc + 4, sizeof(char *));
if (arguments == NULL) return 127;
arguments[0] = "/usr/bin/prlimit";
arguments[1] = limit;
arguments[2] = "--";
arguments[3] = python;
for (int index = 1; index < argc; index++) arguments[index + 3] = argv[index];
pid_t child = fork();
if (child == 0) {
execv(arguments[0], arguments);
perror("execv");
_exit(127);
}
free(arguments);
if (child < 0) {
perror("fork");
return 127;
}
int status;
while (waitpid(child, &status, 0) < 0) {
if (errno == EINTR) continue;
perror("waitpid");
return 127;
}
if (WIFEXITED(status)) return WEXITSTATUS(status);
if (WIFSIGNALED(status)) {
int signum = WTERMSIG(status);
signal(signum, SIG_DFL);
sigset_t signals;
sigemptyset(&signals);
sigaddset(&signals, signum);
sigprocmask(SIG_UNBLOCK, &signals, NULL);
raise(signum);
return 128 + signum;
}
return 127;
}
+15 -3
View File
@@ -28,6 +28,11 @@ import {
isolatedFixtureUrl,
} from "./browser-runtime-world-e2e.mjs";
// Arming, progress delivery, isolated-world proof and the page action cross
// separate transports. Leave time for their round trips on shared CI runners;
// coverage can include a late action even after request collection has ended.
const executionObservationMs = 2_000;
/** Exercise native browser runtime attribution through public CLI and stdio MCP, including arming. */
export async function verifyBrowserRuntime(
executable,
@@ -210,7 +215,7 @@ async function executionMcp(
const response = await client.callTool(
{
name: "observe_web_execution",
arguments: { ...input, observation_ms: 300 },
arguments: { ...input, observation_ms: executionObservationMs },
},
{
timeout: 40_000,
@@ -259,7 +264,7 @@ async function executionCli(entrypoint, input, env, action, isolatedContext) {
input.cdp_endpoint,
input.target_id,
"--observation-ms",
"300",
String(executionObservationMs),
"--json",
],
{ env, stdio: ["ignore", "pipe", "pipe"] },
@@ -382,7 +387,14 @@ function assertExecution(evidence, site, requireBlock = true) {
const request = result.requests.find((item) =>
item.url.endsWith("/evidence?marker=chosen"),
);
assert.ok(request, "The selected request initiator must be retained");
assert.ok(
request,
`The selected request initiator must be retained: ${JSON.stringify({
window: result.window,
requests: result.requests.map((item) => item.url),
excluded_requests: result.excluded_requests,
})}`,
);
assert.ok(
request.callsites.some(
(site) =>
+24 -17
View File
@@ -99,6 +99,14 @@ const scenarioFor = (browser, origin) =>
capture: { after_each_step: ["dom"], at_end: ["dom"] },
});
const raceWithTimeout = (operation, timeoutMs, onTimeout = () => undefined) => {
const controller = new AbortController();
return Promise.race([
operation,
delay(timeoutMs, undefined, { signal: controller.signal }).then(onTimeout),
]).finally(() => controller.abort());
};
const storagePage = (kind, port) => {
if (kind === "main")
return `<!doctype html><html><body><pre id="main-report"></pre><pre id="frame-report"></pre><iframe src="http://localhost:${port}/storage-frame"></iframe><script>
@@ -168,6 +176,9 @@ const startStorageSite = async () => {
origin: `http://127.0.0.1:${port}`,
close: async () => {
server.close();
// Chrome can leave speculative TCP connections without an HTTP request;
// server.close() alone waits indefinitely for those owned sockets.
server.closeAllConnections();
await once(server, "close");
},
};
@@ -303,11 +314,10 @@ const verifyConnectedStorage = async (endpoint, targetId, origin) => {
await verifyCallerDebuggerPause(observer, origin, scenario);
await verifyCancelledStorage(observer, endpoint, targetId, scenario);
} finally {
await Promise.race([
restoreTarget(observer, target.url),
delay(1_000),
]).catch(() => undefined);
await Promise.race([observer.close(), delay(1_000)]).catch(() => undefined);
await raceWithTimeout(restoreTarget(observer, target.url), 1_000).catch(
() => undefined,
);
await raceWithTimeout(observer.close(), 1_000).catch(() => undefined);
}
};
@@ -500,14 +510,11 @@ const verifyCallerDebuggerPause = async (caller, origin, scenario) => {
unsubscribe();
});
try {
const callFrameId = await Promise.race([
paused,
delay(10_000).then(() => {
throw new Error(
`Caller debugger pause was not observed at ${debuggerUrl}; settled=${String(settled)} events=${JSON.stringify(protocolTrace)}`,
);
}),
]);
const callFrameId = await raceWithTimeout(paused, 10_000, () => {
throw new Error(
`Caller debugger pause was not observed at ${debuggerUrl}; settled=${String(settled)} events=${JSON.stringify(protocolTrace)}`,
);
});
const pausedState = await caller.send("Debugger.evaluateOnCallFrame", {
callFrameId,
expression: "window.reaDebuggerAfter === true",
@@ -536,10 +543,10 @@ const verifyCallerDebuggerPause = async (caller, origin, scenario) => {
} catch (cause) {
if (!pauseObserved) controller.abort(cause);
else await caller.send("Debugger.resume").catch(() => undefined);
const completion = await Promise.race([
const completion = await raceWithTimeout(
pending.catch((pendingCause) => ({ ok: false, error: pendingCause })),
delay(5_000).then(() => undefined),
]);
5_000,
);
if (completion === undefined)
throw new Error("Capture remained pending after cancellation", { cause });
const completionError =
@@ -579,7 +586,7 @@ const verifyCallerDebuggerPause = async (caller, origin, scenario) => {
});
} finally {
unsubscribe();
await Promise.race([caller.send("Debugger.disable"), delay(1_000)]).catch(
await raceWithTimeout(caller.send("Debugger.disable"), 1_000).catch(
() => undefined,
);
}
+33 -6
View File
@@ -101,10 +101,13 @@ export async function readReleaseHistory(run, base, target) {
// REA history maps Anthropic and Cursor's agent trailers to ordinary User actors.
// Keep those source identities in the inventory while excluding automation credit.
const AGENT_ACCOUNTS = new Set(["claude", "cursoragent"]);
// REA's maintainer requested external-contributor thanks without self-credit.
const MAINTAINER_ACCOUNTS = new Set(["morluto"]);
const human = (value) =>
value?.__typename === "User" &&
!value.login.endsWith("[bot]") &&
!AGENT_ACCOUNTS.has(value.login.toLowerCase())
!AGENT_ACCOUNTS.has(value.login.toLowerCase()) &&
!MAINTAINER_ACCOUNTS.has(value.login.toLowerCase())
? value.login
: undefined;
const unique = (values) =>
@@ -163,6 +166,12 @@ export function contributionRecord(raw) {
return `${lines.join("\n")}\n`;
}
/** Keep the full authorship snapshot available without duplicating public thanks. */
export function contributionSummary(raw) {
const inventory = releaseInventorySchema.parse(raw);
return `Full contribution record: [${inventory.commits.length} commits and ${inventory.pullRequests.length} merged PR${inventory.pullRequests.length === 1 ? "" : "s"}](https://github.com/${inventory.repository}/blob/rea-agents-${inventory.version}/docs/releases/${inventory.version}.contributions.json).\n`;
}
export function selectedReleaseNotes(text, version) {
const headings = [...text.matchAll(/^## \[([^\]]+)\].*$/gmu)];
if (headings[0]?.[1] !== version)
@@ -236,7 +245,8 @@ function requiredCredits(inventory, bullet, references) {
if (matches[0].parents.length > 1) continue;
for (const identity of matches[0].authors) {
const handle = human(identity.user);
if (handle) required.push({ handle, context: `commit ${match[2]}` });
if (handle)
required.push({ handle, context: `commit ${matches[0].sha}` });
}
}
return required;
@@ -264,6 +274,8 @@ function verifyEditorialCredits(inventory, prose) {
]).map((handle) => handle.toLowerCase()),
);
const errors = [];
const requiredByContext = new Map();
const thankedByContext = new Map();
for (const bullet of proseBullets(prose)) {
const references = bulletPrs(bullet, inventory.repository);
const start = bullet.indexOf("Thanks ");
@@ -282,13 +294,21 @@ function verifyEditorialCredits(inventory, prose) {
bullet,
references,
)) {
if (!thanked.has(handle.toLowerCase()))
errors.push(`Missing Thanks @${handle} in bullet for ${context}`);
const required = requiredByContext.get(context) ?? new Set();
required.add(handle);
requiredByContext.set(context, required);
const credited = thankedByContext.get(context) ?? new Set();
for (const login of thanked) credited.add(login);
thankedByContext.set(context, credited);
}
for (const handle of thanked)
if (!eligible.has(handle))
errors.push(`Unverified human credit @${handle}`);
}
for (const [context, required] of requiredByContext)
for (const handle of required)
if (!thankedByContext.get(context)?.has(handle.toLowerCase()))
errors.push(`Missing Thanks @${handle} for ${context}`);
return errors;
}
@@ -305,8 +325,15 @@ export function verifyReleaseNotes(raw, text, body) {
"Release notes exceed GitHub's body budget; shorten prose without dropping credit",
);
const record = contributionRecord(inventory).trim();
const start = notes.indexOf("### Complete contribution record");
if (start < 0 || notes.slice(start).trim() !== record)
const summary = contributionSummary(inventory).trim();
const start = notes.endsWith(summary)
? notes.length - summary.length
: notes.indexOf("### Complete contribution record");
if (
start < 0 ||
(notes.slice(start).trim() !== record &&
notes.slice(start).trim() !== summary)
)
throw new Error(
"Contribution record differs from inventory; regenerate it with release-notes record",
);
+8 -3
View File
@@ -6,6 +6,7 @@ import { collectGitHubCredits } from "./lib/release-notes-github.mjs";
import { verifyNotesPublication } from "./lib/release-notes-publication.mjs";
import {
contributionRecord,
contributionSummary,
readReleaseHistory,
releaseInventorySchema,
selectedReleaseNotes,
@@ -73,6 +74,7 @@ async function main() {
"pr-body": { type: "string" },
source: { type: "string" },
output: { type: "string" },
summary: { type: "boolean", default: false },
},
});
const [command] = positionals;
@@ -92,7 +94,7 @@ async function main() {
!values.manifest
)
throw new Error(
"Usage: release-notes inventory|record|check|render --manifest FILE [--repository OWNER/REPO --base TAG --target SHA --version VERSION] [--notes FILE --pr-body FILE --source SHA --output FILE]",
"Usage: release-notes inventory|record|check|render --manifest FILE [--repository OWNER/REPO --base TAG --target SHA --version VERSION] [--notes FILE --pr-body FILE --source SHA --output FILE] [--summary for record]",
);
if (command === "inventory") {
await createInventory(values);
@@ -103,11 +105,14 @@ async function main() {
);
await verifyInventoryHistory(run, inventory, values.source);
if (command === "record") {
const record = values.summary
? contributionSummary(inventory)
: contributionRecord(inventory);
if (values.output)
await writeFile(values.output, contributionRecord(inventory), {
await writeFile(values.output, record, {
flag: "wx",
});
else process.stdout.write(contributionRecord(inventory));
else process.stdout.write(record);
return;
}
const notes = await readFile(values.notes, "utf8");
+84 -40
View File
@@ -15,6 +15,10 @@ import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { PrivateRuntimeRoot } from "../dist/process/PrivateRuntimeRoot.js";
import {
OwnedCommandFailure,
runOwnedCommand,
} from "../dist/process/OwnedCommand.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import { mcpTextValue } from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
@@ -56,6 +60,13 @@ const root = await PrivateRuntimeRoot.create({
const environment = Object.fromEntries(
Object.entries(process.env).filter(([, value]) => typeof value === "string"),
);
const limitedPython = join(root.path, "limited-python");
const limitedEnvironment = (option) => ({
...environment,
REA_PWNTOOLS_PYTHON: limitedPython,
REA_VERIFY_LIMITED_PYTHON: python,
REA_VERIFY_LIMIT_OPTION: option,
});
const client = new Client({ name: "binary-layout-verifier", version: "1" });
const transport = new StdioClientTransport({
command: process.execPath,
@@ -73,6 +84,46 @@ let cases = 0;
let bootstrapCases = 0;
const failures = [];
try {
await execute(
"gcc",
[
"-std=c11",
"-Wall",
"-Wextra",
"-Werror",
fileURLToPath(
new URL("./fixtures/binary-layout-limited-python.c", import.meta.url),
),
"-o",
limitedPython,
],
{ timeout: 30_000, maxBuffer: 1024 * 1024 },
);
// A live limited child must retain its configured launcher's identity so
// timeout cleanup can still verify and terminate the entire owned group.
await assert.rejects(
runOwnedCommand(
{
command: limitedPython,
arguments: [
"-I",
"-c",
"import time; print('limited child ready', flush=True); time.sleep(60)",
],
cwd: root.path,
runId: `${run.run_id}-limited-python-timeout`,
hostEnvironment: limitedEnvironment("--as=2147483648"),
},
{ timeoutMs: 2000, diagnosticBytes: 1024 },
),
(error) => {
assert.ok(error instanceof OwnedCommandFailure);
assert.equal(error.reason, "timeout");
assert.equal(error.cleanupFailure, null);
assert.ok(error.snapshot.stdout.text.includes("limited child ready"));
return true;
},
);
const bootstrapRoot = join(root.path, "bootstrap-boundary");
await mkdir(bootstrapRoot);
const bootstrapPath = join(bootstrapRoot, "layout.py");
@@ -796,19 +847,12 @@ try {
assert.deepEqual(await readFile(extendedPath), extendedObject);
cases++;
}
for (const [name, option, expected] of [
["address-space", "--as=2147483648", '"address_space_bytes": 2147483648'],
["cpu", "--cpu=20", '"cpu_seconds": 20'],
["file-size", "--fsize=33554432", '"file_size_bytes": 33554432'],
for (const [option, expected] of [
["--as=2147483648", '"address_space_bytes": 2147483648'],
["--cpu=20", '"cpu_seconds": 20'],
["--fsize=33554432", '"file_size_bytes": 33554432'],
]) {
const wrapper = join(root.path, `python-limit-${name}`);
const quotedPython = "'" + python.replaceAll("'", "'\"'\"'") + "'";
await writeFile(
wrapper,
`#!/bin/sh\nexec /usr/bin/prlimit ${option} -- ${quotedPython} "$@"\n`,
{ mode: 0o700 },
);
const limitedEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: wrapper };
const selectedEnvironment = limitedEnvironment(option);
const limitedClient = new Client({
name: "limited-layout-verifier",
version: "1",
@@ -816,7 +860,7 @@ try {
const limitedTransport = new StdioClientTransport({
command: process.execPath,
args: [entrypoint, "mcp"],
env: limitedEnvironment,
env: selectedEnvironment,
stderr: "pipe",
});
try {
@@ -826,7 +870,7 @@ try {
mode,
join(root.path, "protected"),
undefined,
limitedEnvironment,
selectedEnvironment,
limitedClient,
);
assert.ok(value.limitations.some((item) => item.includes(expected)));
@@ -862,17 +906,29 @@ try {
sectionHeavy.writeUInt16LE(sectionCount, 60);
const sectionHeavyPath = join(root.path, "section-heavy.o");
await writeFile(sectionHeavyPath, sectionHeavy);
const memoryWrapper = join(root.path, "python-memory-constraint");
const quotedPython = "'" + python.replaceAll("'", "'\"'\"'") + "'";
await writeFile(
memoryWrapper,
`#!/bin/sh\nexec /usr/bin/prlimit --as=100663296 -- ${quotedPython} "$@"\n`,
{ mode: 0o700 },
// pwntools materializes each PT_LOAD zero-filled tail during inspection.
// One allocation larger than the whole address-space budget fails promptly;
// a large section count instead races CPU work against the command deadline.
const memoryHeavy = Buffer.from(await readFile(join(root.path, "protected")));
const memoryPhOffset = Number(memoryHeavy.readBigUInt64LE(32));
const memoryPhSize = memoryHeavy.readUInt16LE(54);
const memoryPhCount = memoryHeavy.readUInt16LE(56);
const lastLoadIndex = Array.from(
{ length: memoryPhCount },
(_, index) => index,
).findLast(
(index) =>
memoryHeavy.readUInt32LE(memoryPhOffset + index * memoryPhSize) === 1,
);
const memoryEnvironment = {
...environment,
REA_PWNTOOLS_PYTHON: memoryWrapper,
};
assert.notEqual(lastLoadIndex, undefined);
const memorySizeOffset = memoryPhOffset + lastLoadIndex * memoryPhSize + 40;
memoryHeavy.writeBigUInt64LE(
memoryHeavy.readBigUInt64LE(memorySizeOffset) + 128n * 1024n * 1024n,
memorySizeOffset,
);
const memoryHeavyPath = join(root.path, "memory-zero-tail");
await writeFile(memoryHeavyPath, memoryHeavy);
const memoryEnvironment = limitedEnvironment("--as=100663296");
const memoryClient = new Client({
name: "memory-layout-verifier",
version: "1",
@@ -888,7 +944,7 @@ try {
for (const mode of ["cli", "mcp"]) {
const error = await inspect(
mode,
sectionHeavyPath,
memoryHeavyPath,
"resource_constraint",
memoryEnvironment,
memoryClient,
@@ -903,7 +959,7 @@ try {
assert.ok(error.remediation.action.includes("memory"));
cases++;
}
assert.deepEqual(await readFile(sectionHeavyPath), sectionHeavy);
assert.deepEqual(await readFile(memoryHeavyPath), memoryHeavy);
} finally {
try {
await memoryClient.close();
@@ -924,13 +980,7 @@ try {
cpuHeavy.writeBigUInt64LE(BigInt(cpuSectionCount), sectionOffset + 32);
const cpuHeavyPath = join(root.path, "cpu-section-heavy.o");
await writeFile(cpuHeavyPath, cpuHeavy);
const cpuWrapper = join(root.path, "python-cpu-constraint");
await writeFile(
cpuWrapper,
`#!/bin/sh\nexec /usr/bin/prlimit --cpu=1: --core=0: -- ${quotedPython} "$@"\n`,
{ mode: 0o700 },
);
const cpuEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: cpuWrapper };
const cpuEnvironment = limitedEnvironment("--cpu=1:");
const cpuClient = new Client({ name: "cpu-layout-verifier", version: "1" });
const cpuTransport = new StdioClientTransport({
command: process.execPath,
@@ -963,13 +1013,7 @@ try {
}
}
for (const limit of [1024, 1]) {
const wrapper = join(root.path, `python-file-size-${limit}`);
await writeFile(
wrapper,
`#!/bin/sh\nexec /usr/bin/prlimit --fsize=${limit}: --core=0: -- ${quotedPython} "$@"\n`,
{ mode: 0o700 },
);
const fileEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: wrapper };
const fileEnvironment = limitedEnvironment(`--fsize=${limit}:`);
const fileClient = new Client({
name: "file-size-layout-verifier",
version: "1",
+145 -97
View File
@@ -38,104 +38,147 @@ let update;
let repositorySkill;
let report;
const runStage = async (name, action) => {
const started = performance.now();
process.stderr.write(`[verify:package] ${name}: start\n`);
const result = await action();
process.stderr.write(
`[verify:package] ${name}: complete (${Math.round(performance.now() - started)} ms)\n`,
);
return result;
};
try {
repositorySkill = await verifyRepositorySkill({ root, workspace });
({ tarball } = await verifyPackagePack({ root, workspace }));
const environmentData = await verifyPackageEnvironment({
root,
workspace,
evidenceRoot,
referenceRoot,
});
const { cli, packageRunnerCli } = await verifyPackageInstall({
tarball,
prefix: environmentData.prefix,
workspace,
environment: environmentData.environment,
});
update = await verifyPackageUpdate({
prefix: environmentData.prefix,
packageRoot: join(
environmentData.prefix,
"lib",
"node_modules",
PRODUCT_IDENTITY.packageName,
),
tarball,
workspace,
environment: environmentData.environment,
});
mcpStartup = await measureMcpStartup({
command: cli,
args: ["mcp"],
environment: {
...environmentData.environment,
},
policy: MCP_STARTUP_POLICY,
});
mcpModuleLoading = await profileMcpModuleLoading({
command: cli,
args: ["mcp"],
environment: {
...environmentData.environment,
},
policy: MCP_STARTUP_POLICY,
packageName: PRODUCT_IDENTITY.packageName,
});
const { supportedSetupHost, hopperSetupSupported } =
await verifyPackageDiscovery({
repositorySkill = await runStage("repository skill", () =>
verifyRepositorySkill({ root, workspace }),
);
({ tarball } = await runStage("package archive", () =>
verifyPackagePack({ root, workspace }),
));
const environmentData = await runStage("package environment", () =>
verifyPackageEnvironment({
root,
workspace,
evidenceRoot,
referenceRoot,
}),
);
const { cli, packageRunnerCli } = await runStage("package install", () =>
verifyPackageInstall({
tarball,
prefix: environmentData.prefix,
workspace,
environment: environmentData.environment,
}),
);
update = await runStage("package update", () =>
verifyPackageUpdate({
prefix: environmentData.prefix,
packageRoot: join(
environmentData.prefix,
"lib",
"node_modules",
PRODUCT_IDENTITY.packageName,
),
tarball,
workspace,
environment: environmentData.environment,
}),
);
mcpStartup = await runStage("MCP startup", () =>
measureMcpStartup({
command: cli,
args: ["mcp"],
environment: {
...environmentData.environment,
},
policy: MCP_STARTUP_POLICY,
}),
);
mcpModuleLoading = await runStage("MCP module loading", () =>
profileMcpModuleLoading({
command: cli,
args: ["mcp"],
environment: {
...environmentData.environment,
},
policy: MCP_STARTUP_POLICY,
packageName: PRODUCT_IDENTITY.packageName,
}),
);
const { supportedSetupHost, hopperSetupSupported } = await runStage(
"package discovery",
() =>
verifyPackageDiscovery({
cli,
environment: environmentData.environment,
}),
);
const { artifactArchive } = await runStage("artifact and Electron CLI", () =>
verifyPackageArtifactAndElectron({
cli,
workspace,
environment: environmentData.environment,
}),
);
await runStage("managed analysis CLI", () =>
verifyManaged({
cli,
workspace,
environment: environmentData.environment,
}),
);
await runStage("unknown provider CLI", () =>
verifyUnknownProvider({
cli,
environment: environmentData.environment,
});
const { artifactArchive } = await verifyPackageArtifactAndElectron({
cli,
workspace,
environment: environmentData.environment,
});
await verifyManaged({
cli,
workspace,
environment: environmentData.environment,
});
await verifyUnknownProvider({
cli,
environment: environmentData.environment,
});
await verifyPackagePlatform({
cli,
environment: environmentData.environment,
});
await verifyPackageCapabilitiesAndSearch({
cli,
environment: environmentData.environment,
});
await verifyPackageEvidence({
cli,
evidenceRoot,
referenceRoot,
environment: environmentData.environment,
});
await verifyPackageSetup({
cli,
packageRunnerCli,
environment: environmentData.environment,
home: environmentData.home,
npxLog: environmentData.npxLog,
claudeConfig: environmentData.claudeConfig,
codexConfig: environmentData.codexConfig,
cursorConfig: environmentData.cursorConfig,
codexTarget: environmentData.codexTarget,
cursorTarget: environmentData.cursorTarget,
supportedSetupHost,
hopperSetupSupported,
root,
});
await verifyPackageMcp({
cli,
environment: environmentData.environment,
evidenceRoot,
artifactArchive,
});
}),
);
await runStage("platform CLI", () =>
verifyPackagePlatform({
cli,
environment: environmentData.environment,
}),
);
await runStage("capabilities and search CLI", () =>
verifyPackageCapabilitiesAndSearch({
cli,
environment: environmentData.environment,
}),
);
await runStage("evidence CLI", () =>
verifyPackageEvidence({
cli,
evidenceRoot,
referenceRoot,
environment: environmentData.environment,
}),
);
await runStage("setup and uninstall CLI", () =>
verifyPackageSetup({
cli,
packageRunnerCli,
environment: environmentData.environment,
home: environmentData.home,
npxLog: environmentData.npxLog,
claudeConfig: environmentData.claudeConfig,
codexConfig: environmentData.codexConfig,
cursorConfig: environmentData.cursorConfig,
codexTarget: environmentData.codexTarget,
cursorTarget: environmentData.cursorTarget,
supportedSetupHost,
hopperSetupSupported,
root,
}),
);
await runStage("MCP tools and evidence", () =>
verifyPackageMcp({
cli,
environment: environmentData.environment,
evidenceRoot,
artifactArchive,
}),
);
report = {
cli: true,
analysisCli: true,
@@ -173,8 +216,13 @@ try {
mcpModuleLoading,
};
} finally {
await rm(workspace, { recursive: true, force: true });
await runStage("workspace cleanup", () =>
rm(workspace, { recursive: true, force: true }),
);
}
const verifierLineage = await runStage("verifier lineage", () =>
completeVerifierRun(verifierRun),
);
process.stdout.write(
`${JSON.stringify({ verifier_run: await completeVerifierRun(verifierRun), ...report })}\n`,
`${JSON.stringify({ verifier_run: verifierLineage, ...report })}\n`,
);
+4
View File
@@ -277,11 +277,13 @@ try {
throw new Error("Scenario attachment terminated its external browser");
const profilesBefore = await scenarioProfiles();
process.stderr.write("Browser verifier: scenario environment\n");
const scenarioEnvironment = await verifyScenarioEnvironment(
endpoint,
target,
site.origin,
);
process.stderr.write("Browser verifier: scenario storage\n");
const scenarioStorage = await verifyScenarioStorage({
executable,
endpoint,
@@ -339,7 +341,9 @@ try {
`Browser scenario included unselected structured value: ${unselectedValue}`,
);
process.stderr.write("Browser verifier: large screenshot\n");
const largeScreenshot = await verifyLargeScreenshotE2e(endpoint, site.origin);
process.stderr.write("Browser verifier: popup coverage\n");
const popupEvents = await verifyPopupEventCoverage(executable);
const networkEvidence = await verifyBrowserNetworkEvidence(
executable,
+107
View File
@@ -674,6 +674,113 @@ export async function verifyRelativeSwitch(client, procedures, entrySize) {
};
}
/**
* Oracle for relative-switch-signed.S: signed halfword byte offsets from a base
* inside the case bodies, a CBZ route for the zero entry, and a B.CS bound.
*/
const SIGNED_RELATIVE_SWITCH_CASES = [
{ value: 0, immediate: 200 },
{ value: 1, immediate: 201 },
{ value: 2, immediate: 202 },
{ value: 3, immediate: 250 },
{ value: 4, immediate: 204 },
];
export async function verifyRelativeSignedSwitch(client, selectorBias = 0) {
const names = await inventoryCall(client, "list_names", {
document: null,
address: null,
});
const procedure = requireProcedure(names, "rea_relative_switch_signed");
const dossier = await functionCall(client, "analyze_function", {
procedure: procedure.address,
});
const table = dossier.native_api?.jump_tables.find((item) =>
item.data_sources.some(
(source) =>
source.entry_size_bytes === 2 &&
source.entry_count === SIGNED_RELATIVE_SWITCH_CASES.length,
),
);
if (table === undefined)
throw new Error(
`Exact signed halfword relative table unavailable: ${JSON.stringify(dossier.native_api)}`,
);
const expectedValues = SIGNED_RELATIVE_SWITCH_CASES.map(
({ value }) => value + selectorBias,
);
if (
table.mappings.some(
(mapping) =>
mapping.case_value !== null &&
!expectedValues.includes(mapping.case_value),
)
)
throw new Error(
`Signed relative switch emitted an unsupported case label: ${JSON.stringify(table)}`,
);
if (table.mappings.length !== SIGNED_RELATIVE_SWITCH_CASES.length)
throw new Error(
`Signed relative switch has unexpected mappings: ${JSON.stringify(table)}`,
);
if (selectorBias !== 0) {
if (
!table.limitations.some((limitation) =>
limitation.includes("table indices are not proven"),
) ||
!table.mappings.some((mapping) => mapping.case_value === null) ||
table.mappings.some((mapping) => mapping.case_value === selectorBias + 3)
)
throw new Error(
`Normalized selector must retain unknown case evidence: ${JSON.stringify(table)}`,
);
}
for (const { value: tableIndex, immediate } of SIGNED_RELATIVE_SWITCH_CASES) {
const value = tableIndex + selectorBias;
// Without a proven relationship between the machine index and the typed
// switch selector, the CBZ-only case must stay unknown rather than being
// assigned the normalized index or Ghidra's unreachable base destination.
if (selectorBias !== 0 && tableIndex === 3) continue;
const mapping = table.mappings.find((item) => item.case_value === value);
if (
mapping === undefined ||
mapping.confidence !== "high" ||
(selectorBias !== 0 &&
!mapping.evidence.some(
(evidence) => evidence.source === "ghidra-clang-case-token",
))
)
throw new Error(
`Signed relative switch case ${value} was not verified: ${JSON.stringify(table)}`,
);
const instruction = await functionCall(
client,
"inspect_native_instruction",
{ address: mapping.target_address },
);
if (
instruction.status !== "decoded" ||
!instruction.operands.some((operand) =>
operand.components.some(
(component) =>
component.kind === "immediate" &&
component.value === `0x${immediate.toString(16)}`,
),
)
)
throw new Error(
`Signed relative switch target does not implement source case ${value}: ${JSON.stringify(instruction)}`,
);
}
return {
entry_size: 2,
signed: true,
selector_bias: selectorBias,
dispatch_address: table.dispatch_address,
mappings: table.mappings.length,
};
}
export async function verifyNativeValueTrace(client, procedures, target) {
const entry = requireProcedure(procedures, "rea_ghidra_inventory_entry");
const names = await inventoryCall(client, "list_names", {
+53 -17
View File
@@ -38,6 +38,7 @@ import {
verifyJumpTableDenseSwitch,
verifyNativeTypeLayout,
verifyRelativeSwitch,
verifyRelativeSignedSwitch,
verifyNativeValueTrace,
} from "./verify-real-ghidra-function.mjs";
const exec = promisify(execFile);
@@ -173,6 +174,36 @@ try {
]);
}
}
const signedRelativeSource = fileURLToPath(
new URL(
"../tests/conformance/ghidra/relative-switch-signed.S",
import.meta.url,
),
);
for (const selectorBias of [0, 2, 10]) {
for (const format of [
"elf",
...(expectedNativeTarget.architecture === "arm64" ? ["mach-o"] : []),
]) {
const targetPath = join(
fixtureRoot,
`relative-signed-${selectorBias}-${format}.o`,
);
await exec(clang, [
...(format === "elf" ? ["--target=aarch64-linux-gnu"] : []),
`-DREA_SELECTOR_BIAS=${selectorBias}`,
"-c",
signedRelativeSource,
"-o",
targetPath,
]);
crossTargets.push([
targetPath,
`relative-signed-${selectorBias}`,
{ format, architecture: "arm64" },
]);
}
}
await exec(clang, [
"--target=aarch64-linux-gnu",
@@ -522,25 +553,30 @@ async function verifyTarget(targetPath, variant, expectedTarget = null) {
client,
})
: null;
const probes = variant.startsWith("relative-")
? await verifyRelativeSwitch(
const probes = variant.startsWith("relative-signed-")
? await verifyRelativeSignedSwitch(
client,
procedures,
Number(variant.slice(-1)),
Number(variant.split("-").at(-1)),
)
: variant === "type-layout"
? await verifyNativeTypeLayout(client, names)
: variant === "jump-table"
? await verifyJumpTableDenseSwitch(client, procedures)
: variant === "custom" || variant === "aarch64-jump-table"
? null
: await verifyInventoryOperations({
client,
variant,
procedures,
names,
strings,
});
: variant.startsWith("relative-")
? await verifyRelativeSwitch(
client,
procedures,
Number(variant.slice(-1)),
)
: variant === "type-layout"
? await verifyNativeTypeLayout(client, names)
: variant === "jump-table"
? await verifyJumpTableDenseSwitch(client, procedures)
: variant === "custom" || variant === "aarch64-jump-table"
? null
: await verifyInventoryOperations({
client,
variant,
procedures,
names,
strings,
});
const nativeValues =
variant === "debug"
? await verifyNativeValueTrace(client, procedures, parsedTarget.value)
+71 -1
View File
@@ -6,6 +6,7 @@ import { createHash } from "node:crypto";
import { once } from "node:events";
import {
access,
copyFile,
mkdir,
mkdtemp,
readFile,
@@ -16,7 +17,7 @@ import {
open,
} from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { dirname, join, parse, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import { verifyWindowsProtocolInput } from "./lib/windows-protocol-input.mjs";
@@ -32,6 +33,9 @@ const native = requireWindowsNativeAuthority();
const { WindowsPrivateRuntime, windowsPrivateRuntime } = await import(
pathToFileURL(join(packageRoot, "dist/windows/WindowsPrivateRuntime.js"))
);
const { ghidraSessionRoot } = await import(
pathToFileURL(join(packageRoot, "dist/ghidra/GhidraSessionRoot.js"))
);
const workspace = await mkdtemp(join(tmpdir(), "rea-native-conformance-"));
const environment = Object.entries(process.env)
.filter(([, value]) => value !== undefined)
@@ -77,6 +81,31 @@ const wait = async (child) => {
};
const report = { ok: false, artifact: "verified", controls: {}, processes: {} };
try {
// Exercise the loader in a process whose image is not named node.exe. The
// ordinary Windows lane otherwise cannot detect eager Node-API imports.
const renamedHost = join(workspace, "rea-renamed-node.exe");
const renamedHostFixture = join(workspace, "renamed-host.mjs");
const loaderUrl = pathToFileURL(
join(packageRoot, "dist/windows/WindowsNativeLoader.js"),
).href;
await copyFile(process.execPath, renamedHost);
await writeFile(
renamedHostFixture,
`import {basename} from 'node:path';
import {requireWindowsNativeAuthority} from ${JSON.stringify(loaderUrl)};
const {inspection}=requireWindowsNativeAuthority();
process.stdout.write(JSON.stringify({executable:basename(process.execPath),abiVersion:inspection.abiVersion}));\n`,
);
const renamedHostResult = execFileSync(renamedHost, [renamedHostFixture], {
encoding: "utf8",
timeout: 20_000,
windowsHide: true,
});
assert.deepEqual(JSON.parse(renamedHostResult), {
executable: "rea-renamed-node.exe",
abiVersion: 1,
});
report.controls.renamedHostNodeApi = true;
const sourceDirectory = join(workspace, "source");
await mkdir(sourceDirectory);
const source = join(sourceDirectory, "target.bin");
@@ -110,6 +139,47 @@ try {
])
assert.throws(() => native.call("open", [rejected]));
report.controls.ordinaryDriveSeparatorsAndRequestedIdentity = true;
const driveRoot = parse(workspace).root;
const runtimeParent = ghidraSessionRoot({
base: join(driveRoot, ".Tmp", "scratch"),
});
assert.equal(runtimeParent, driveRoot);
const driveRootRuntime = native.call("runtime_create", [
runtimeParent,
"rea-drive-root-",
]);
try {
assert.equal(driveRootRuntime.path.startsWith(`${driveRoot}\\`), false);
assert.equal(dirname(driveRootRuntime.path), driveRoot);
const snapshot = await native.call("runtime_snapshot", [
driveRootRuntime.handle,
source,
"probe.bin",
]);
const sourceBytes = await readFile(source);
assert.equal(
snapshot.sha256,
createHash("sha256").update(sourceBytes).digest("hex"),
);
assert.equal(snapshot.source.requestedPath, source);
const readback = native.call("runtime_open", [
driveRootRuntime.handle,
"probe.bin",
]);
try {
assert.equal(readback.size, sourceBytes.byteLength);
assert.deepEqual(
native.call("read", [readback.handle, 0, 65536]),
sourceBytes.subarray(0, 65536),
);
} finally {
native.call("close", [readback.handle]);
}
} finally {
native.call("runtime_close", [driveRootRuntime.handle]);
}
assert.equal(await exists(driveRootRuntime.path), false);
report.controls.driveRootRuntimeParent = true;
const snapshotOwner = WindowsPrivateRuntime.create(
workspace.replace(/\\/u, "/"),
"single-flight-",
+2 -2
View File
@@ -8,13 +8,13 @@
"url": "https://github.com/morluto/rea",
"source": "github"
},
"version": "6.3.0",
"version": "6.4.0",
"packages": [
{
"registryType": "npm",
"registryBaseUrl": "https://registry.npmjs.org",
"identifier": "rea-agents",
"version": "6.3.0",
"version": "6.4.0",
"runtimeHint": "npx",
"transport": {
"type": "stdio"
+21 -20
View File
@@ -1,17 +1,21 @@
import { execFile } from "node:child_process";
import { lstat, opendir, readFile, realpath, stat } from "node:fs/promises";
import { join } from "node:path";
import { promisify } from "node:util";
import { parseBinary } from "plist";
import { z } from "zod";
import { preflightPropertyListDecode } from "../artifacts/apple/InterfaceBuilderDecodeBudget.js";
import { BinaryTargetError } from "../domain/configurationErrors.js";
import { isPathWithinRoot } from "../domain/localPath.js";
import { parseXmlPropertyList } from "../domain/propertyListKeys.js";
import {
omitPrototypeKeys,
parseXmlPropertyList,
} from "../domain/propertyListKeys.js";
import { decodeXmlPlistText } from "../domain/propertyListXmlText.js";
import { err, ok, type Result } from "../domain/result.js";
const execFileAsync = promisify(execFile);
/** Decoded representation bound for one binary Info.plist. */
const BINARY_PLIST_DECODE_BYTES = 16 * 1024 * 1024;
/** Where one app bundle layout keeps its Info.plist and program file. */
interface BundleLayout {
@@ -193,9 +197,8 @@ const resolveLayoutExecutable = async (
try {
name =
plist.subarray(0, 6).toString("ascii") === "bplist"
? await (fileSystem.decodeBinaryPlist ?? readBinaryPlistExecutable)(
plistPath,
)
? await (fileSystem.decodeBinaryPlist?.(plistPath) ??
parseBinaryPlistExecutable(plist))
: parseXmlPlistExecutable(decodeXmlPlistText(plist));
} catch (cause: unknown) {
return err(
@@ -248,6 +251,10 @@ const resolveLayoutExecutable = async (
const parseXmlPlistExecutable = (plist: string): string => {
// An unrelated `__proto__` entry must not make the bundle unreadable.
const { value } = parseXmlPropertyList(plist);
return executableName(value);
};
const executableName = (value: unknown): string => {
const executable = executableEntrySchema.safeParse(value);
if (!executable.success) throw new Error("CFBundleExecutable is missing");
return executable.data.CFBundleExecutable;
@@ -255,19 +262,13 @@ const parseXmlPlistExecutable = (plist: string): string => {
const executableEntrySchema = z.looseObject({ CFBundleExecutable: z.string() });
const readBinaryPlistExecutable = async (plistPath: string): Promise<string> =>
// `-n` strips only the newline plutil appends; it requires macOS 12+.
(
await execFileAsync("/usr/bin/plutil", [
"-extract",
"CFBundleExecutable",
"raw",
"-n",
"-o",
"-",
plistPath,
])
).stdout;
/** Decode a binary plist in-process, so no host plist tool is required. */
const parseBinaryPlistExecutable = (plist: Buffer): string => {
preflightPropertyListDecode(plist, BINARY_PLIST_DECODE_BYTES);
// `plist` assigns a `__proto__` key as the prototype; copying drops it, as
// the XML path does.
return executableName(omitPrototypeKeys(parseBinary(plist)).value);
};
const isSafeExecutableName = (name: string): boolean =>
name.length > 0 &&
+2 -2
View File
@@ -138,7 +138,7 @@ export const runManagedProviderExecution = async (
signal?: AbortSignal,
): Promise<Result<AnalysisExecution, AnalysisError>> =>
withProcessCancellation(signal, async (operationSignal) => {
const session = dependencies.createManagedBinarySession();
const session = await dependencies.createManagedBinarySession();
return withSessionCleanup(
session,
async () => {
@@ -194,7 +194,7 @@ const runAnalysis = async (
const { logger, signal, snapshotPath } = options;
const config = dependencies.readConfiguration();
if (!config.ok) return cliError(config.error);
const session = dependencies.createBinarySession(config.value, logger);
const session = await dependencies.createBinarySession(config.value, logger);
return withSessionCleanup(
session,
async () => {
@@ -10,6 +10,8 @@ export interface DirectAnalysisDependencies {
readonly createBinarySession: (
config: AppConfig,
logger: Logger,
) => BinarySession;
readonly createManagedBinarySession: () => BinarySession;
) => BinarySession | Promise<BinarySession>;
readonly createManagedBinarySession: () =>
| BinarySession
| Promise<BinarySession>;
}
+1 -1
View File
@@ -15,7 +15,7 @@ export const runSessionStatus = async (
): Promise<JsonValue> => {
const config = dependencies.readConfiguration();
if (!config.ok) return { error: projectAnalysisError(config.error) };
const session = dependencies.createBinarySession(config.value, logger);
const session = await dependencies.createBinarySession(config.value, logger);
try {
return {
...jsonObjectSchema.parse(session.status()),
@@ -0,0 +1,35 @@
import type { EvidenceBundle } from "../../domain/evidenceBundle.js";
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
import { err, ok, type Result } from "../../domain/result.js";
import {
summarizeEvidenceBundle,
type EvidenceBundleSummary,
type EvidenceRecordFilters,
} from "../../domain/evidenceRecordSummary.js";
/** Select complete bundle delivery or discovery metadata without analysis payloads. */
export const inspectEvidenceBundle = (
bundle: EvidenceBundle,
options: {
readonly detail: "complete" | "summary";
readonly filters?: EvidenceRecordFilters | undefined;
},
): Result<EvidenceBundle | EvidenceBundleSummary, AnalysisInputError> => {
if (options.detail === "complete" && options.filters !== undefined)
return err(
new AnalysisInputError("get_evidence_bundle", undefined, [
{
path: ["filters"],
reason: "invalid_value",
message:
"Record filters require detail: summary; complete returns the canonical bundle.",
expected: "detail: summary",
},
]),
);
return ok(
options.detail === "summary"
? summarizeEvidenceBundle(bundle, options.filters)
: bundle,
);
};
@@ -26,6 +26,7 @@ import {
staticInferenceEvidence,
} from "./JavaScriptArtifactGraphEvidence.js";
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
import { compareUnicodeCodePoints } from "../../domain/unicodeCodePointOrder.js";
interface PackageRoleInput {
readonly packageNode: ApplicationNode;
@@ -281,6 +282,7 @@ export const addJavaScriptPackageNodes = (
context: JavaScriptArtifactGraphContext,
): ApplicationNode[] => {
const roots: ApplicationNode[] = [];
const rootPath = applicationManifestPath(context);
for (const packageValue of context.analysis.packages) {
const file = context.filesByPath.get(packageValue.path);
if (file === undefined) continue;
@@ -309,7 +311,7 @@ export const addJavaScriptPackageNodes = (
},
],
});
if (roots.length === 0) {
if (file.path === rootPath) {
roots.push(node);
context.accumulator.addEdge({
source_node_id: node.node_id,
@@ -330,6 +332,7 @@ export const addJavaScriptPackageNodes = (
file,
operation: "inventory-package",
});
if (dependencyManifest(file.path)) continue;
addPackageRole(context, {
packageNode: node,
packageFile: file,
@@ -346,6 +349,24 @@ export const addJavaScriptPackageNodes = (
return roots;
};
/** Dependency-only artifacts retain their inventory root. */
const applicationManifestPath = (
context: JavaScriptArtifactGraphContext,
): string | undefined =>
context.analysis.packages
.map(({ path }) => path)
.filter(
(path) => context.filesByPath.has(path) && !dependencyManifest(path),
)
.sort(
(left, right) =>
left.split("/").length - right.split("/").length ||
compareUnicodeCodePoints(left, right),
)[0];
const dependencyManifest = (path: string): boolean =>
path.split("/").slice(0, -1).includes("node_modules");
const createFileTarget = (
context: JavaScriptArtifactGraphContext,
file: JavaScriptArtifactFile,
+7
View File
@@ -43,6 +43,7 @@ import {
type WebScreenshotDiff,
} from "../domain/webScreenshot.js";
import { AnalysisError } from "../domain/analysisErrorBase.js";
import { AnalysisInputError } from "../domain/analysisErrorCore.js";
import { BrowserObservationError } from "../domain/browserObservationError.js";
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { type BrowserObservationOperation } from "../domain/browserObservationErrors.js";
@@ -314,6 +315,12 @@ export class CdpBrowserProvider implements BrowserObservationPort {
try {
return ok(webScreenshotDiffSchema.parse(comparePngScreenshots(input)));
} catch (cause: unknown) {
if (cause instanceof TypeError)
return err(
new AnalysisInputError("compare_web_screenshots", { cause }, [
{ path: [], reason: "invalid_format", message: cause.message },
]),
);
return err(providerError(cause, "compare_web_screenshots"));
}
}
@@ -606,3 +606,402 @@ describe("SVG xlink:href context", () => {
expect(dangling.urls).toEqual([]);
});
});
const SVG_NAMESPACE = "http://www.w3.org/2000/svg";
const XLINK_NAMESPACE = "http://www.w3.org/1999/xlink";
const XHTML_NAMESPACE = "http://www.w3.org/1999/xhtml";
const xhtmlRoot = (
attributes: readonly (readonly [string, string])[] = [
["xmlns", XHTML_NAMESPACE],
["xmlns:svg", SVG_NAMESPACE],
["xmlns:xlink", XLINK_NAMESPACE],
],
): SvgSnapshotNode => ({
name: "html",
parent: 0,
attributes,
});
const approvedHref = (
nodeIndex: number,
url: string,
): {
readonly node_index: number;
readonly attribute: "href";
readonly url: string;
readonly destination_scope: "approved";
} => ({
node_index: nodeIndex,
attribute: "href",
url,
destination_scope: "approved",
});
describe("XML SVG roots and namespace scope restoration", () => {
it.each(["svg", "document"])(
"resolves namespaces under an unprefixed XML %s root",
(name) => {
const result = captureSvg(
svgDocument([
{
name,
parent: 0,
attributes: [
["xmlns", SVG_NAMESPACE],
["xmlns:p", XLINK_NAMESPACE],
],
},
{ name: "a", parent: 1, attributes: [["p:href", "/linked"]] },
]),
);
expect(result.urls).toEqual([approvedHref(2, `${origin}/linked`)]);
},
);
it.each([2, 3, 99])(
"rejects self-parent, cyclic and dangling XML ancestry (%s)",
(parent) => {
const result = captureSvg(
svgDocument([
xhtmlRoot(),
{
name: "svg:a",
parent,
attributes: [
["xmlns:svg", SVG_NAMESPACE],
["xmlns:p", XLINK_NAMESPACE],
["p:href", "/invalid"],
],
},
{ name: "svg:g", parent: 2 },
]),
);
expect(result.urls).toEqual([]);
},
);
it("restores namespace declarations and undeclarations before visiting siblings", () => {
const result = captureSvg(
svgDocument([
xhtmlRoot(),
{
name: "svg:a",
parent: 1,
attributes: [
["xmlns:xlink", ""],
["xlink:href", "/undeclared"],
],
},
{
name: "svg:a",
parent: 1,
attributes: [["xlink:href", "/inherited"]],
},
{
name: "svg:a",
parent: 1,
attributes: [
["xmlns:svg", "other"],
["xlink:href", "/rebound"],
],
},
{ name: "svg:a", parent: 1, attributes: [["xlink:href", "/restored"]] },
]),
);
expect(result.urls).toEqual([
approvedHref(3, `${origin}/inherited`),
approvedHref(5, `${origin}/restored`),
]);
});
});
describe("XHTML and XML SVG namespace prefixes", () => {
it("reports prefixed svg:a, svg:image, and an svg:a with no svg parent", () => {
const result = captureSvg(
svgDocument([
xhtmlRoot(),
{ name: "svg:svg", parent: 1 },
{
name: "svg:a",
parent: 2,
attributes: [
["xlink:href", "/xl-pa1"],
["xlink:href", "/later"],
],
},
{
name: "svg:image",
parent: 2,
attributes: [["xlink:href", "guide"]],
},
{
name: "svg:a",
parent: 1,
attributes: [["xlink:href", "/xl-orphan"]],
},
]),
);
expect(result.nodes[3]?.node_name).toBe("svg:a");
expect(result.nodes[3]?.attribute_names).toEqual(["xlink:href"]);
expect(result.urls).toEqual([
approvedHref(3, `${origin}/xl-pa1`),
approvedHref(4, `${origin}/screens/assets/guide`),
approvedHref(5, `${origin}/xl-orphan`),
]);
});
it("reports svg:a under svg:foreignObject and skips XHTML anchors there", () => {
const result = captureSvg(
svgDocument([
xhtmlRoot([
["xmlns", XHTML_NAMESPACE],
["xmlns:svg", SVG_NAMESPACE],
["xmlns:xlink", XLINK_NAMESPACE],
["xmlns:h", XHTML_NAMESPACE],
]),
{ name: "svg:svg", parent: 1 },
{ name: "svg:foreignObject", parent: 2 },
{
name: "a",
parent: 3,
attributes: [["xlink:href", "/xl-html-fo"]],
},
{
name: "h:a",
parent: 3,
attributes: [["xlink:href", "/xl-h"]],
},
{
name: "svg:a",
parent: 3,
attributes: [["xlink:href", "/xl-fo-svg"]],
},
]),
);
expect(urlFor(result, 4)).toBeUndefined();
expect(urlFor(result, 5)).toBeUndefined();
expect(result.urls).toEqual([approvedHref(6, `${origin}/xl-fo-svg`)]);
});
it("reads XLink by namespace, including a prefix other than xlink", () => {
const result = captureSvg(
svgDocument([
xhtmlRoot([["xmlns", XHTML_NAMESPACE]]),
{
name: "svg",
parent: 1,
attributes: [
["xmlns", SVG_NAMESPACE],
["xmlns:xlink", "http://example.com/not-xlink"],
["xmlns:x", XLINK_NAMESPACE],
],
},
{
name: "a",
parent: 2,
attributes: [
["xlink:href", "/not-xlink"],
["x:href", "/xl-custom"],
["x:href", "/later"],
],
},
]),
);
expect(result.nodes[3]?.attribute_names).toEqual(["xlink:href", "x:href"]);
expect(result.urls).toEqual([approvedHref(3, `${origin}/xl-custom`)]);
});
it("keeps plain href ahead of a namespaced XLink href, including an empty href", () => {
const baseUrl = `${origin}/screens/assets/`;
const anchor = (
parent: number,
attributes: readonly (readonly [string, string])[],
): SvgSnapshotNode => ({
name: "svg:a",
parent,
attributes,
});
const result = captureSvg(
svgDocument(
[
xhtmlRoot(),
{ name: "svg:svg", parent: 1 },
anchor(2, [
["xlink:href", "/xlink-loses"],
["href", "/href-wins"],
]),
anchor(2, [
["href", "/href-wins"],
["xlink:href", "/xlink-loses"],
]),
anchor(2, [
["href", ""],
["xlink:href", "/xlink-should-lose"],
]),
],
baseUrl,
),
);
expect(result.urls).toEqual([
approvedHref(3, `${origin}/href-wins`),
approvedHref(4, `${origin}/href-wins`),
approvedHref(5, baseUrl),
]);
});
});
describe("XHTML SVG exclusions, HTML documents, and prefix scope", () => {
it("classifies a prefixed SVG xlink:href outside the allowed origins as outside_policy", () => {
const result = captureSvg(
svgDocument([
xhtmlRoot(),
{ name: "svg:svg", parent: 1 },
{
name: "svg:a",
parent: 2,
attributes: [["xlink:href", "https://cdn.example.test/out"]],
},
]),
);
expect(result.urls).toEqual([
{
node_index: 3,
attribute: "href",
url: null,
destination_scope: "outside_policy",
},
]);
});
it("does not treat bogus:svg in another namespace as SVG", () => {
const result = captureSvg(
svgDocument([
xhtmlRoot([
["xmlns", XHTML_NAMESPACE],
["xmlns:svg", SVG_NAMESPACE],
["xmlns:xlink", XLINK_NAMESPACE],
["xmlns:bogus", "http://example.com/not-svg"],
]),
{
name: "bogus:svg",
parent: 1,
attributes: [["xlink:href", "/xl-bogus"]],
},
{
name: "bogus:a",
parent: 2,
attributes: [["xlink:href", "/xl-bogus-a"]],
},
{
name: "svg:a",
parent: 1,
attributes: [["xlink:href", "/xl-kept"]],
},
]),
);
expect(result.nodes[2]?.node_name).toBe("bogus:svg");
expect(result.urls).toEqual([approvedHref(4, `${origin}/xl-kept`)]);
});
it("leaves HTML documents on literal svg and xlink:href names", () => {
const result = captureSvg(
svgDocument([
{ name: "HTML", parent: 0 },
{
name: "svg",
parent: 1,
attributes: [
["xmlns", SVG_NAMESPACE],
["xmlns:x", XLINK_NAMESPACE],
],
},
{
name: "a",
parent: 2,
attributes: [["xlink:href", "/xl-literal"]],
},
{
name: "a",
parent: 2,
attributes: [["x:href", "/xl-custom-html"]],
},
{
name: "svg:a",
parent: 1,
attributes: [
["xmlns:svg", SVG_NAMESPACE],
["xmlns:xlink", XLINK_NAMESPACE],
["xlink:href", "/xl-prefixed-html"],
],
},
{
name: "A",
parent: 1,
attributes: [["xlink:href", "/xl-html-anchor"]],
},
]),
);
expect(result.urls).toEqual([approvedHref(3, `${origin}/xl-literal`)]);
});
it("applies an element's own xmlns declarations to its name and href", () => {
const result = captureSvg(
svgDocument([
{ name: "html", parent: 0, attributes: [["xmlns", XHTML_NAMESPACE]] },
{
name: "svg:a",
parent: 1,
attributes: [
["xmlns:svg", SVG_NAMESPACE],
["xmlns:xlink", XLINK_NAMESPACE],
["xlink:href", "/xl-self"],
],
},
]),
);
expect(result.urls).toEqual([approvedHref(2, `${origin}/xl-self`)]);
});
it("resolves a prefix declared on a later ancestor", () => {
const result = captureSvg(
svgDocument([
{
name: "svg:a",
parent: 2,
attributes: [["xlink:href", "/forward"]],
},
{ name: "svg:svg", parent: 3 },
xhtmlRoot(),
]),
);
expect(result.urls).toEqual([approvedHref(1, `${origin}/forward`)]);
});
it("resolves a deep chain with distinct namespace declarations without quadratic expansion", () => {
const depth = 12_000;
const elements: SvgSnapshotNode[] = [
xhtmlRoot(),
{ name: "svg:svg", parent: 1 },
];
let parent = 2;
for (let index = 0; index < depth; index += 1) {
elements.push({
name: "svg:g",
parent,
attributes: [[`xmlns:p${index}`, `urn:prefix:${index}`]],
});
parent = elements.length;
}
elements.push({
name: "svg:a",
parent,
attributes: [["xlink:href", "/deep"]],
});
const snapshot = svgDocument(elements);
const result = captureSvg(snapshot);
expect(result.urls).toEqual([
approvedHref(elements.length, `${origin}/deep`),
]);
}, 30_000);
});
+40 -60
View File
@@ -11,6 +11,7 @@ import {
} from "./CdpCaptureValues.js";
import { exclusionReasonForUrl } from "./CdpCaptureEventHelpers.js";
import type { CdpCaptureCompleteness } from "./CdpCaptureCompleteness.js";
import { createDocumentSvgAncestry, domSvgLink } from "./CdpCaptureDomSvg.js";
export type CapturedResource = Omit<
WebPageInspection["resources"][number],
@@ -171,24 +172,20 @@ export const captureDom = (
? documentNodes.attributes
: [];
const baseIndex = nodes.length;
const svgContexts = new Map<number, boolean>();
const ancestry = createDocumentSvgAncestry({
nodeTypes,
nodeNames,
parents,
strings,
attributes,
});
total += nodeTypes.length;
for (let index = 0; index < nodeTypes.length; index += 1) {
const attributeIndexes = numberArray(attributes[index]);
const parent = Math.trunc(parents[index] ?? -1);
const nodeIndex = nodes.length;
const nodeName = indexedString(strings, nodeNames[index]);
const rawAttributeNames = attributeIndexes
.filter((_value, attributeIndex) => attributeIndex % 2 === 0)
.map((value) => indexedString(strings, value));
const seenAttributeNames = new Set<string>();
const attributeNames: string[] = [];
for (const name of rawAttributeNames) {
if (!seenAttributeNames.has(name)) {
seenAttributeNames.add(name);
attributeNames.push(name);
}
}
const attributeNames = uniqueAttributeNames(attributeIndexes, strings);
nodes.push({
index: nodeIndex,
parent_index: parent < 0 ? -1 : baseIndex + parent,
@@ -206,13 +203,7 @@ export const captureDom = (
nodeIndex,
nodeName,
allowedOrigins,
svgElement: elementInSvgContext(
index,
nodeNames,
parents,
strings,
svgContexts,
),
...domSvgLink(ancestry, index),
});
for (const url of metadata.urls) {
urls.push(url);
@@ -367,6 +358,21 @@ const indexedString = (strings: readonly string[], index: unknown): string => {
return integer === undefined ? "" : (strings[Math.trunc(integer)] ?? "");
};
const uniqueAttributeNames = (
attributeIndexes: readonly number[],
strings: readonly string[],
): string[] => {
const names: string[] = [];
const seen = new Set<string>();
for (let index = 0; index < attributeIndexes.length; index += 2) {
const name = indexedString(strings, attributeIndexes[index]);
if (seen.has(name)) continue;
seen.add(name);
names.push(name);
}
return names;
};
const numberArray = (value: unknown): readonly number[] =>
Array.isArray(value)
? value.flatMap((item) => {
@@ -384,41 +390,22 @@ interface DomMetadataOptions {
readonly nodeName: string;
readonly allowedOrigins: ReadonlySet<string>;
readonly svgElement: boolean;
readonly xmlDocument: boolean;
readonly xlinkHref: string | undefined;
}
// DOMSnapshot carries no namespace URI. Chrome's HTML parser records SVG
// local names (`svg`, `foreignObject`). Descendants of foreignObject are HTML
// until a nested `svg`.
const elementInSvgContext = (
index: number,
nodeNames: readonly number[],
parents: readonly number[],
strings: readonly string[],
contexts: Map<number, boolean>,
): boolean => {
let current = index;
const seen = new Set<number>();
let inSvg = false;
while (current >= 0 && current < nodeNames.length && !seen.has(current)) {
const cached = contexts.get(current);
if (cached !== undefined) {
inSvg = cached;
break;
}
seen.add(current);
if (indexedString(strings, nodeNames[current]) === "svg") {
inSvg = true;
break;
}
const parent = Math.trunc(parents[current] ?? -1);
if (parent < 0 || parent === current) break;
if (indexedString(strings, nodeNames[parent]) === "foreignObject") break;
current = parent;
}
// Cache the entire examined chain, including forward references and cycles,
// so shared ancestors are examined once per document.
for (const node of seen) contexts.set(node, inSvg);
return inSvg;
// Chrome reads an SVG XLink href only when href itself is absent, and an
// empty href still wins. HTML keeps the literal attribute name `xlink:href`.
// XML uses any prefix bound to the XLink namespace. Both are stored as href.
const hrefAttributeValue = (
pairs: ReadonlyMap<string, string>,
attribute: string,
options: DomMetadataOptions,
): string | undefined => {
const declared = pairs.get(attribute);
if (declared !== undefined) return declared;
if (attribute !== "href" || !options.svgElement) return undefined;
return options.xmlDocument ? options.xlinkHref : pairs.get("xlink:href");
};
const domMetadata = (
@@ -435,7 +422,6 @@ const domMetadata = (
nodeIndex,
nodeName,
allowedOrigins,
svgElement,
} = options;
const pairs = new Map<string, string>();
for (let index = 0; index + 1 < attributes.length; index += 2) {
@@ -447,13 +433,7 @@ const domMetadata = (
}
const urls: WebPageInspection["metadata"]["dom_urls"] = [];
for (const attribute of domUrlAttributes) {
// Chrome reads SVG xlink:href only when href itself is absent, and an
// empty href still wins. Report the fallback as href.
const value =
pairs.get(attribute) ??
(attribute === "href" && svgElement
? pairs.get("xlink:href")
: undefined);
const value = hrefAttributeValue(pairs, attribute, options);
if (value === undefined) continue;
const tagName = nodeName.toLowerCase();
// Chrome strips HTML whitespace for FORM.action, while submit controls
+275
View File
@@ -0,0 +1,275 @@
import { numberValue } from "./CdpCaptureValues.js";
const SVG_NAMESPACE = "http://www.w3.org/2000/svg";
const XLINK_NAMESPACE = "http://www.w3.org/1999/xlink";
interface DomSnapshotNodes {
readonly nodeTypes: readonly number[];
readonly nodeNames: readonly number[];
readonly parents: readonly number[];
readonly strings: readonly string[];
readonly attributes: readonly unknown[];
}
interface XmlSvgLink {
readonly svgElement: boolean;
readonly xlinkHref: string | undefined;
}
interface DocumentSvgAncestry {
readonly xml: boolean;
readonly nodes: DomSnapshotNodes;
readonly svgContexts: Map<number, boolean>;
readonly xmlLinks: Map<number, XmlSvgLink>;
}
interface DomSvgLink {
readonly svgElement: boolean;
readonly xmlDocument: boolean;
readonly xlinkHref: string | undefined;
}
const UNRESOLVED: XmlSvgLink = {
svgElement: false,
xlinkHref: undefined,
};
const stringAt = (strings: readonly string[], index: unknown): string => {
const integer = numberValue(index);
return integer === undefined ? "" : (strings[Math.trunc(integer)] ?? "");
};
const numberArray = (value: unknown): readonly number[] =>
Array.isArray(value)
? value.flatMap((item) => {
const number = numberValue(item);
return number === undefined ? [] : [number];
})
: [];
// DOMSnapshot has no namespace URI. HTML documents use uppercase `HTML` and
// the parser's local names. XML documents preserve prefixed names and xmlns
// declarations, including those on standalone SVG roots.
const xmlDocument = (nodes: DomSnapshotNodes): boolean => {
const documentIndex = nodes.nodeTypes.findIndex(
(nodeType) => Math.trunc(nodeType) === 9,
);
if (documentIndex < 0) return false;
for (let index = 0; index < nodes.nodeTypes.length; index += 1) {
if (Math.trunc(nodes.nodeTypes[index] ?? 0) !== 1) continue;
if (Math.trunc(nodes.parents[index] ?? -1) !== documentIndex) continue;
const name = stringAt(nodes.strings, nodes.nodeNames[index]);
if (name === "HTML") return false;
if (name === "html" || name.includes(":")) return true;
// Explicit namespaces also identify standalone SVG and other XML roots.
// Namespace-free synthetic snapshots retain the legacy HTML behavior.
const attributes = numberArray(nodes.attributes[index]);
return attributes.some(
(value, offset) =>
offset % 2 === 0 &&
xmlnsPrefix(stringAt(nodes.strings, value)) !== undefined,
);
}
return false;
};
export const createDocumentSvgAncestry = (
nodes: DomSnapshotNodes,
): DocumentSvgAncestry => {
const xml = xmlDocument(nodes);
return {
xml,
nodes,
svgContexts: new Map(),
xmlLinks: xml ? resolveXmlElements(nodes) : new Map(),
};
};
export const domSvgLink = (
ancestry: DocumentSvgAncestry,
index: number,
): DomSvgLink => {
if (!ancestry.xml) {
return {
svgElement: elementInSvgContext(
index,
ancestry.nodes,
ancestry.svgContexts,
),
xmlDocument: false,
xlinkHref: undefined,
};
}
const resolved = ancestry.xmlLinks.get(index) ?? UNRESOLVED;
return {
svgElement: resolved.svgElement,
xmlDocument: true,
xlinkHref: resolved.xlinkHref,
};
};
// Chrome's HTML parser records SVG local names (`svg`, `foreignObject`).
// Descendants of foreignObject are HTML until a nested `svg`.
const elementInSvgContext = (
index: number,
nodes: DomSnapshotNodes,
contexts: Map<number, boolean>,
): boolean => {
let current = index;
const seen = new Set<number>();
let inSvg = false;
while (
current >= 0 &&
current < nodes.nodeNames.length &&
!seen.has(current)
) {
const cached = contexts.get(current);
if (cached !== undefined) {
inSvg = cached;
break;
}
seen.add(current);
if (stringAt(nodes.strings, nodes.nodeNames[current]) === "svg") {
inSvg = true;
break;
}
const parent = Math.trunc(nodes.parents[current] ?? -1);
if (parent < 0 || parent === current) break;
if (stringAt(nodes.strings, nodes.nodeNames[parent]) === "foreignObject")
break;
current = parent;
}
// Cache the entire examined chain, including forward references and cycles,
// so shared ancestors are examined once per document.
for (const node of seen) contexts.set(node, inSvg);
return inSvg;
};
interface NamespaceRestore {
readonly prefix: string;
readonly previous: string | undefined;
}
type XmlWalkStep =
| { readonly node: number }
| { readonly restore: readonly NamespaceRestore[] };
// Walk parent-before-child with one mutable namespace environment. Restore
// declarations on exit so siblings retain their own scope. Only declarations
// on the active path are retained, rather than cumulative maps at every node.
const resolveXmlElements = (
nodes: DomSnapshotNodes,
): Map<number, XmlSvgLink> => {
const children = new Map<number, number[]>();
const pending: XmlWalkStep[] = [];
for (let node = 0; node < nodes.nodeNames.length; node += 1) {
const parent = Math.trunc(nodes.parents[node] ?? -1);
if (parent < 0) pending.push({ node });
else if (parent < nodes.nodeNames.length && parent !== node) {
const siblings = children.get(parent);
if (siblings === undefined) children.set(parent, [node]);
else siblings.push(node);
}
}
const bindings = new Map<string, string>();
const resolved = new Map<number, XmlSvgLink>();
while (pending.length > 0) {
const step = pending.pop();
if (step === undefined) break;
if ("restore" in step) {
for (const { prefix, previous } of step.restore) {
if (previous === undefined) bindings.delete(prefix);
else bindings.set(prefix, previous);
}
continue;
}
const { node } = step;
const attributes = numberArray(nodes.attributes[node]);
const restore = applyDeclarations(bindings, attributes, nodes.strings);
const svgElement =
Math.trunc(nodes.nodeTypes[node] ?? 0) === 1 &&
elementNamespace(
stringAt(nodes.strings, nodes.nodeNames[node]),
bindings,
) === SVG_NAMESPACE;
resolved.set(node, {
svgElement,
xlinkHref: svgElement
? xlinkHrefValue(attributes, nodes.strings, bindings)
: undefined,
});
pending.push({ restore });
for (const child of children.get(node) ?? []) pending.push({ node: child });
}
// Cycles (including self-parents) and dangling chains cannot reach a root,
// so remain unresolved instead of deriving URLs from an invalid ancestry.
return resolved;
};
const applyDeclarations = (
bindings: Map<string, string>,
attributes: readonly number[],
strings: readonly string[],
): readonly NamespaceRestore[] => {
const restore: NamespaceRestore[] = [];
const seen = new Set<string>();
for (let index = 0; index + 1 < attributes.length; index += 2) {
const prefix = xmlnsPrefix(stringAt(strings, attributes[index]));
if (prefix === undefined || seen.has(prefix)) continue;
seen.add(prefix);
restore.push({ prefix, previous: bindings.get(prefix) });
const value = stringAt(strings, attributes[index + 1]);
if (value.length === 0) bindings.delete(prefix);
else bindings.set(prefix, value);
}
return restore;
};
const xmlnsPrefix = (name: string): string | undefined => {
if (name === "xmlns") return "";
if (!name.startsWith("xmlns:")) return undefined;
const prefix = name.slice("xmlns:".length);
if (prefix.length === 0 || prefix === "xmlns" || prefix.includes(":"))
return undefined;
return prefix;
};
const elementNamespace = (
nodeName: string,
bindings: ReadonlyMap<string, string>,
): string => {
const colon = nodeName.indexOf(":");
if (colon < 0) return bindings.get("") ?? "";
const prefix = nodeName.slice(0, colon);
const local = nodeName.slice(colon + 1);
if (
prefix.length === 0 ||
local.length === 0 ||
local.includes(":") ||
prefix === "xmlns"
)
return "";
return bindings.get(prefix) ?? "";
};
// First prefixed href bound to the XLink namespace. Duplicate names keep the
// earlier value. Plain href is applied by the caller and wins even when empty.
const xlinkHrefValue = (
attributes: readonly number[],
strings: readonly string[],
bindings: ReadonlyMap<string, string>,
): string | undefined => {
const seen = new Set<string>();
for (let index = 0; index + 1 < attributes.length; index += 2) {
const name = stringAt(strings, attributes[index]);
if (seen.has(name)) continue;
seen.add(name);
const colon = name.indexOf(":");
if (colon <= 0) continue;
const prefix = name.slice(0, colon);
const local = name.slice(colon + 1);
if (local !== "href" || bindings.get(prefix) !== XLINK_NAMESPACE) continue;
return stringAt(strings, attributes[index + 1]);
}
return undefined;
};
+268 -1
View File
@@ -3,6 +3,10 @@ import { crc32, deflateSync } from "node:zlib";
import { describe, expect, it } from "vitest";
import { comparePngScreenshots } from "./PngVisualDiff.js";
import {
AnalysisResourceConstraintError,
AnalysisUnsupportedTargetError,
} from "../domain/analysisErrorCore.js";
import {
compareWebScreenshotsInputSchema,
createWebScreenshotArtifact,
@@ -117,7 +121,7 @@ describe("PNG visual diff", () => {
});
it("rejects malformed PNG dimensions after validating image data", () => {
const image = artifact(32_000_001, 1, [0, 0, 0, 255]);
const image = artifact(2, 1, [0, 0, 0, 255]);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
@@ -149,6 +153,256 @@ describe("PNG visual diff", () => {
});
});
describe("PNG comparison memory budget", () => {
it("classifies legal huge dimensions as a memory constraint before inflate", () => {
const image = createWebScreenshotArtifact(headerOnlyPng(0x7fffffff, 1));
const input = compareWebScreenshotsInputSchema.parse({
before: image,
after: image,
});
try {
comparePngScreenshots(input);
throw new Error("expected PNG working-memory rejection");
} catch (cause: unknown) {
expect(cause).toBeInstanceOf(AnalysisResourceConstraintError);
expect(cause).toMatchObject({
resource: "memory",
reportedLimits: {
maximum_working_memory_bytes: 256 * 1024 * 1024,
estimated_working_memory_bytes: "34359738499",
before_dimensions: { width: 0x7fffffff, height: 1 },
after_dimensions: { width: 0x7fffffff, height: 1 },
},
});
}
});
});
describe("PNG chunk integrity", () => {
it.each(["IHDR", "IDAT", "IEND"])(
"rejects a corrupt %s chunk checksum",
(type) => {
const bytes = png(1, 1, [0, 0, 0, 255]);
let offset = 8;
while (offset < bytes.length) {
const length = bytes.readUInt32BE(offset);
if (bytes.subarray(offset + 4, offset + 8).toString("ascii") === type) {
bytes[offset + 8 + length] = (bytes[offset + 8 + length] ?? 0) ^ 1;
break;
}
offset += length + 12;
}
const image = createWebScreenshotArtifact(bytes);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: image,
after: image,
}),
),
).toThrow("Invalid PNG chunk checksum");
},
);
it("reports a CRC-valid but corrupt compressed stream as invalid PNG data", () => {
const header = Buffer.from(png(1, 1, [0, 0, 0, 255]).subarray(16, 29));
const image = createWebScreenshotArtifact(
Buffer.concat([
Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]),
chunk("IHDR", header),
chunk("IDAT", Buffer.from("bad")),
chunk("IEND", Buffer.alloc(0)),
]),
);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({ before: image, after: image }),
),
).toThrow("Invalid PNG image data");
});
it("rejects high-bit bytes in a CRC-valid PNG chunk name", () => {
const bytes = png(1, 1, [0, 0, 0, 255]);
const headerChunk = Buffer.from(bytes.subarray(8, 33));
headerChunk[4] = 0xc9;
headerChunk.writeUInt32BE(crc32(headerChunk.subarray(4, 21)), 21);
const malformed = createWebScreenshotArtifact(
Buffer.concat([bytes.subarray(0, 8), headerChunk, bytes.subarray(33)]),
);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: malformed,
after: malformed,
}),
),
).toThrow("Invalid PNG chunk type");
});
it("rejects duplicate image headers and unknown critical chunks while preserving ancillary chunks", () => {
const bytes = png(1, 1, [0, 0, 0, 255]);
const duplicate = Buffer.concat([
bytes.subarray(0, 33),
bytes.subarray(8, 33),
bytes.subarray(33),
]);
const unknown = Buffer.concat([
bytes.subarray(0, 33),
chunk("ABCD", Buffer.alloc(0)),
bytes.subarray(33),
]);
for (const malformed of [duplicate, unknown]) {
const image = createWebScreenshotArtifact(malformed);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: image,
after: image,
}),
),
).toThrow(/Invalid PNG header order|Unsupported critical PNG chunk/u);
}
const ancillary = createWebScreenshotArtifact(
Buffer.concat([
bytes.subarray(0, 33),
chunk("tEXt", Buffer.from("Comment\0original capture")),
bytes.subarray(33),
]),
);
expect(
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: ancillary,
after: createWebScreenshotArtifact(bytes),
}),
).status,
).toBe("identical");
});
it("rejects bytes after IEND and reports valid unsupported PNG encodings separately", () => {
const bytes = png(1, 1, [0, 0, 0, 255]);
const trailing = createWebScreenshotArtifact(
Buffer.concat([bytes, Buffer.from([0])]),
);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: trailing,
after: trailing,
}),
),
).toThrow("Invalid PNG end chunk");
const grayscaleHeader = Buffer.from(bytes.subarray(16, 29));
grayscaleHeader[9] = 0;
const grayscale = createWebScreenshotArtifact(
Buffer.concat([
Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]),
chunk("IHDR", grayscaleHeader),
chunk("IDAT", deflateSync(Buffer.from([0, 30]))),
chunk("IEND", Buffer.alloc(0)),
]),
);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: grayscale,
after: grayscale,
}),
),
).toThrow(AnalysisUnsupportedTargetError);
});
});
describe("PNG chunk ordering", () => {
it("rejects malformed palettes, interrupted data and misplaced transparency while retaining valid chunks", () => {
const header = Buffer.from(png(1, 1, [10, 20, 30, 255]).subarray(16, 29));
header[9] = 2;
const ihdr = chunk("IHDR", header);
const compressed = deflateSync(Buffer.from([0, 10, 20, 30]));
const idat = chunk("IDAT", compressed);
const iend = chunk("IEND", Buffer.alloc(0));
const palette = chunk("PLTE", Buffer.from([10, 20, 30]));
const transparency = chunk("tRNS", Buffer.from([0, 10, 0, 20, 0, 30]));
const ancillary = chunk("tEXt", Buffer.from("Comment\0retained"));
const image = (...chunks: readonly Buffer[]) =>
createWebScreenshotArtifact(
Buffer.concat([
Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]),
...chunks,
]),
);
const original = image(ihdr, idat, iend);
const cases = [
[ihdr, chunk("PLTE", Buffer.alloc(1)), idat, iend],
[ihdr, palette, palette, idat, iend],
[ihdr, idat, palette, iend],
[ihdr, transparency, palette, idat, iend],
[ihdr, transparency, transparency, idat, iend],
[ihdr, idat, transparency, iend],
[
ihdr,
chunk("IDAT", compressed.subarray(0, 6)),
ancillary,
chunk("IDAT", compressed.subarray(6)),
iend,
],
[ihdr, idat, chunk("IEND", Buffer.from([1]))],
];
for (const chunks of cases) {
const malformed = image(...chunks);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: malformed,
after: original,
}),
),
).toThrow(
/Invalid PNG palette|Invalid PNG transparency order|Nonconsecutive PNG image data|Invalid PNG end chunk/u,
);
}
const indexedHeader = Buffer.from(header);
indexedHeader[8] = 1;
indexedHeader[9] = 3;
const emptyTransparency = image(
chunk("IHDR", indexedHeader),
chunk("PLTE", Buffer.from([10, 20, 30])),
chunk("tRNS", Buffer.alloc(0)),
chunk("IDAT", deflateSync(Buffer.from([0, 0]))),
iend,
);
expect(() =>
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: emptyTransparency,
after: original,
}),
),
).toThrow("Unsupported PNG transparency");
for (const valid of [
image(ihdr, palette, idat, iend),
image(ihdr, ancillary, idat, iend),
image(
ihdr,
chunk("IDAT", compressed.subarray(0, 6)),
chunk("IDAT", compressed.subarray(6)),
iend,
),
]) {
expect(
comparePngScreenshots(
compareWebScreenshotsInputSchema.parse({
before: valid,
after: original,
}),
).status,
).toBe("identical");
}
});
});
const artifact = (width: number, height: number, pixels: readonly number[]) =>
createWebScreenshotArtifact(png(width, height, pixels));
@@ -173,6 +427,19 @@ const png = (
]);
};
const headerOnlyPng = (width: number, height: number): Buffer => {
const header = Buffer.alloc(13);
header.writeUInt32BE(width, 0);
header.writeUInt32BE(height, 4);
header.set([8, 6, 0, 0, 0], 8);
return Buffer.concat([
Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]),
chunk("IHDR", header),
chunk("IDAT", deflateSync(Buffer.from([0]))),
chunk("IEND", Buffer.alloc(0)),
]);
};
const chunk = (type: string, data: Buffer): Buffer => {
const result = Buffer.alloc(12 + data.byteLength);
result.writeUInt32BE(data.byteLength, 0);
+267 -31
View File
@@ -1,10 +1,14 @@
import { inflateSync } from "node:zlib";
import { constants as zlibConstants, crc32, inflateSync } from "node:zlib";
import type {
CompareWebScreenshotsInput,
WebScreenshotArtifact,
WebScreenshotDiff,
} from "../domain/webScreenshot.js";
import {
AnalysisResourceConstraintError,
AnalysisUnsupportedTargetError,
} from "../domain/analysisErrorCore.js";
import { decodeCanonicalBase64 } from "../domain/webScreenshot.js";
interface DecodedPng {
@@ -13,12 +17,32 @@ interface DecodedPng {
readonly rgba: Buffer;
}
interface ParsedPng {
readonly bytes: Buffer;
readonly width: number;
readonly height: number;
readonly channels: number;
readonly compressed: readonly Buffer[];
readonly transparentColor: readonly number[] | undefined;
readonly expectedRawBytes: bigint;
readonly unfilteredBytes: bigint;
readonly rgbaBytes: bigint;
}
/** Conservative allowance for the simultaneously live PNG comparison buffers. */
const PNG_COMPARISON_WORKING_MEMORY_BYTES = 256 * 1024 * 1024;
const PNG_INFLATE_CHUNK_BYTES = zlibConstants.Z_DEFAULT_CHUNK;
/** Compare two validated screenshots using deterministic RGBA metrics. */
export const comparePngScreenshots = (
input: CompareWebScreenshotsInput,
): WebScreenshotDiff => {
const before = decodePng(input.before);
const after = decodePng(input.after);
admitEncodedInputPair(input.before, input.after);
const beforePlan = parsePng(input.before, "before");
const afterPlan = parsePng(input.after, "after");
admitPngPair(beforePlan, afterPlan);
const before = decodePng(beforePlan);
const after = decodePng(afterPlan);
if (before.width !== after.width || before.height !== after.height)
return {
status: "dimension_mismatch",
@@ -73,7 +97,10 @@ export const comparePngScreenshots = (
};
};
const decodePng = (artifact: WebScreenshotArtifact): DecodedPng => {
const parsePng = (
artifact: WebScreenshotArtifact,
field: "before" | "after",
): ParsedPng => {
const bytes = decodeCanonicalBase64(artifact.data_base64);
if (bytes === undefined || !bytes.subarray(0, 8).equals(PNG_SIGNATURE))
throw new TypeError("Invalid PNG signature");
@@ -81,68 +108,277 @@ const decodePng = (artifact: WebScreenshotArtifact): DecodedPng => {
let header: ReturnType<typeof parseHeader> | undefined;
const compressed: Buffer[] = [];
let sawEnd = false;
let sawPalette = false;
let imageDataEnded = false;
let paletteEntries = 0;
let transparentColor: readonly number[] | undefined;
let sawTransparency = false;
let unsupportedReason: string | undefined;
while (offset < bytes.length) {
if (offset + 12 > bytes.length) throw new TypeError("Truncated PNG chunk");
const length = bytes.readUInt32BE(offset);
const type = bytes.subarray(offset + 4, offset + 8).toString("ascii");
if (length > 0x7fffffff) throw new TypeError("Invalid PNG chunk length");
const type = bytes.subarray(offset + 4, offset + 8).toString("latin1");
const dataStart = offset + 8;
const dataEnd = dataStart + length;
if (dataEnd + 4 > bytes.length) throw new TypeError("Truncated PNG data");
const data = bytes.subarray(dataStart, dataEnd);
if (type === "IHDR") header = parseHeader(data);
if (
crc32(bytes.subarray(offset + 4, dataEnd)) !== bytes.readUInt32BE(dataEnd)
)
throw new TypeError("Invalid PNG chunk checksum");
if (!/^[A-Za-z]{4}$/u.test(type) || type[2] !== type[2]?.toUpperCase())
throw new TypeError("Invalid PNG chunk type");
if (type !== "IDAT" && compressed.length > 0) imageDataEnded = true;
if (type === "IHDR") {
if (offset !== 8 || header !== undefined)
throw new TypeError("Invalid PNG header order");
header = parseHeader(data);
} else if (header === undefined)
throw new TypeError("Invalid PNG header order");
else if (type === "tRNS") {
if (header?.channels !== 3 || data.length !== 6)
if (sawTransparency || compressed.length > 0)
throw new TypeError("Invalid PNG transparency order");
sawTransparency = true;
if (header.colorType === 2 && data.length === 6) {
transparentColor = [
data.readUInt16BE(0),
data.readUInt16BE(2),
data.readUInt16BE(4),
];
} else if (
(header.colorType === 0 && data.length === 2) ||
(header.colorType === 3 &&
sawPalette &&
data.length > 0 &&
data.length <= paletteEntries)
) {
unsupportedReason ??=
"PNG transparency for this color type is not supported.";
} else {
throw new TypeError("Unsupported PNG transparency");
transparentColor = [
data.readUInt16BE(0),
data.readUInt16BE(2),
data.readUInt16BE(4),
];
} else if (type === "IDAT") compressed.push(data);
else if (type === "IEND") {
}
} else if (type === "PLTE") {
if (
sawPalette ||
sawTransparency ||
compressed.length > 0 ||
length === 0 ||
length > 768 ||
length % 3 !== 0 ||
(header.colorType === 3 && length / 3 > 2 ** header.bitDepth) ||
(header.colorType !== 2 &&
header.colorType !== 3 &&
header.colorType !== 6)
)
throw new TypeError("Invalid PNG palette");
sawPalette = true;
paletteEntries = length / 3;
} else if (type === "IDAT") {
if (imageDataEnded) throw new TypeError("Nonconsecutive PNG image data");
compressed.push(data);
} else if (type === "IEND") {
if (length !== 0 || dataEnd + 4 !== bytes.length)
throw new TypeError("Invalid PNG end chunk");
sawEnd = true;
break;
} else if (/^[A-Z]/u.test(type)) {
unsupportedReason ??= `Unsupported critical PNG chunk ${type}.`;
}
offset = dataEnd + 4;
}
if (header === undefined || !sawEnd || compressed.length === 0)
throw new TypeError("Incomplete PNG");
const rowBytes = header.width * header.channels;
const expected = (rowBytes + 1) * header.height;
const raw = inflateSync(Buffer.concat(compressed), {
maxOutputLength: expected + 1,
});
if (raw.byteLength !== expected) throw new TypeError("Unexpected PNG size");
if (header.colorType === 3 && !sawPalette)
throw new TypeError("Missing PNG palette");
if (unsupportedReason !== undefined)
throw new AnalysisUnsupportedTargetError(
"compare_web_screenshots",
field,
unsupportedReason,
);
if (!header.supported)
throw new AnalysisUnsupportedTargetError(
"compare_web_screenshots",
field,
"PNG encoding is not supported for pixel comparison.",
);
const pixels = BigInt(header.width) * BigInt(header.height);
const expectedRawBytes =
(BigInt(header.width) * BigInt(header.channels) + 1n) *
BigInt(header.height);
return {
bytes,
width: header.width,
height: header.height,
channels: header.channels,
compressed,
transparentColor,
expectedRawBytes,
unfilteredBytes: pixels * BigInt(header.channels),
rgbaBytes: pixels * 4n,
};
};
const decodePng = (plan: ParsedPng): DecodedPng => {
const expected = Number(plan.expectedRawBytes);
const raw = inflatePng(Buffer.concat(plan.compressed), expected + 1);
if (raw.byteLength !== expected) throw new TypeError("Unexpected PNG size");
return {
width: plan.width,
height: plan.height,
rgba: expandRgba(
unfilter(raw, header.width, header.height, header.channels),
header.channels,
transparentColor,
unfilter(raw, plan.width, plan.height, plan.channels),
plan.channels,
plan.transparentColor,
),
};
};
const admitEncodedInputPair = (
before: WebScreenshotArtifact,
after: WebScreenshotArtifact,
): void => {
const sourceBytes = BigInt(before.bytes) + BigInt(after.bytes);
if (sourceBytes > BigInt(PNG_COMPARISON_WORKING_MEMORY_BYTES))
throw pngWorkingMemoryError(sourceBytes, null, null);
};
const admitPngPair = (before: ParsedPng, after: ParsedPng): void => {
// Input buffers remain live for both plans. For each decode, count the
// concatenated IDAT, rounded zlib output chunks, any concatenated output and
// unfiltered rows, plus RGB-to-RGBA output. The first completed RGBA image
// remains live while the second is decoded.
const inputBytes = BigInt(before.bytes.byteLength + after.bytes.byteLength);
const estimate = (current: ParsedPng, retained: ParsedPng | undefined) => {
const compressedBytes = current.compressed.reduce(
(total, chunk) => total + BigInt(chunk.byteLength),
0n,
);
const rgbaExpansion = current.channels === 3 ? current.rgbaBytes : 0n;
return (
inputBytes +
(retained?.rgbaBytes ?? 0n) +
compressedBytes +
inflateBufferAllowance(current.expectedRawBytes) +
current.unfilteredBytes +
rgbaExpansion
);
};
const estimateBytes = [
estimate(before, undefined),
estimate(after, before),
].reduce((maximum, value) => (value > maximum ? value : maximum), 0n);
if (estimateBytes > BigInt(PNG_COMPARISON_WORKING_MEMORY_BYTES))
throw pngWorkingMemoryError(estimateBytes, before, after);
};
const inflateBufferAllowance = (expectedRawBytes: bigint): bigint => {
const chunkBytes = BigInt(PNG_INFLATE_CHUNK_BYTES);
const maximumOutputBytes = expectedRawBytes + 1n;
const roundUpToChunk = (bytes: bigint): bigint =>
((bytes + chunkBytes - 1n) / chunkBytes) * chunkBytes;
// Node checks maxOutputLength after writing a chunk, so a rejected stream can
// retain one chunk's worth of output past the limit before it throws.
const rejectedOutputChunks = roundUpToChunk(
maximumOutputBytes + chunkBytes - 1n,
);
// Sync zlib retains each output chunk while Buffer.concat creates the final
// result. A full final chunk also allocates the next scratch chunk first.
const returnedOutputChunks = roundUpToChunk(maximumOutputBytes);
const scratchChunk =
expectedRawBytes % chunkBytes === 0n ||
maximumOutputBytes % chunkBytes === 0n
? chunkBytes
: 0n;
const concatenatedOutput =
maximumOutputBytes > chunkBytes ? maximumOutputBytes : 0n;
const returnedOutput =
returnedOutputChunks + scratchChunk + concatenatedOutput;
return rejectedOutputChunks > returnedOutput
? rejectedOutputChunks
: returnedOutput;
};
const pngWorkingMemoryError = (
estimateBytes: bigint,
before: Pick<ParsedPng, "width" | "height"> | null,
after: Pick<ParsedPng, "width" | "height"> | null,
) =>
new AnalysisResourceConstraintError(
"compare_web_screenshots",
"memory",
"Screenshot comparison exceeds REA's 256 MiB estimated working-memory budget.",
{
boundary: "png-screenshot-comparison",
estimated_working_memory_bytes: estimateBytes.toString(),
maximum_working_memory_bytes: PNG_COMPARISON_WORKING_MEMORY_BYTES,
before_dimensions:
before === null ? null : { width: before.width, height: before.height },
after_dimensions:
after === null ? null : { width: after.width, height: after.height },
},
{
remediationAction:
"Compare smaller screenshots or reduce their viewport dimensions before capture.",
},
);
const inflatePng = (compressed: Buffer, maxOutputLength: number): Buffer => {
try {
return inflateSync(compressed, {
maxOutputLength,
chunkSize: PNG_INFLATE_CHUNK_BYTES,
});
} catch (cause: unknown) {
if (
cause instanceof Error &&
"code" in cause &&
(cause.code === "Z_DATA_ERROR" ||
cause.code === "Z_BUF_ERROR" ||
cause.code === "ERR_BUFFER_TOO_LARGE")
)
throw new TypeError("Invalid PNG image data", { cause });
throw cause;
}
};
const parseHeader = (data: Buffer) => {
if (data.byteLength !== 13) throw new TypeError("Invalid PNG header");
const width = data.readUInt32BE(0);
const height = data.readUInt32BE(4);
const bitDepth = data[8];
const colorType = data[9];
const bitDepth = data[8] ?? 0;
const colorType = data[9] ?? -1;
const validDepth =
(colorType === 0 && [1, 2, 4, 8, 16].includes(bitDepth)) ||
(colorType === 2 && [8, 16].includes(bitDepth)) ||
(colorType === 3 && [1, 2, 4, 8].includes(bitDepth)) ||
(colorType === 4 && [8, 16].includes(bitDepth)) ||
(colorType === 6 && [8, 16].includes(bitDepth));
if (
width === 0 ||
height === 0 ||
!Number.isSafeInteger(width * height) ||
bitDepth !== 8 ||
(colorType !== 2 && colorType !== 6) ||
width > 0x7fffffff ||
height > 0x7fffffff ||
!validDepth ||
data[10] !== 0 ||
data[11] !== 0 ||
data[12] !== 0
(data[12] !== 0 && data[12] !== 1)
)
throw new TypeError("Unsupported PNG");
return { width, height, channels: colorType === 6 ? 4 : 3 };
throw new TypeError("Invalid PNG header");
return {
width,
height,
colorType,
bitDepth,
interlace: data[12],
channels: colorType === 6 ? 4 : 3,
supported:
bitDepth === 8 && (colorType === 2 || colorType === 6) && data[12] === 0,
};
};
const unfilter = (
@@ -1,4 +1,4 @@
import { access } from "node:fs/promises";
import { access, stat } from "node:fs/promises";
import { constants } from "node:fs";
import { isAbsolute } from "node:path";
import { fileURLToPath } from "node:url";
@@ -32,6 +32,8 @@ export class MitmproxyCaptureAdapter implements HistoricalCaptureFormatAdapter {
);
try {
await access(command, constants.R_OK | constants.X_OK);
if (!(await stat(command)).isFile())
throw new TypeError("Configured path is not a regular file");
} catch (cause: unknown) {
const reason = `Configured mitmdump executable is unavailable: ${command}.`;
throw new AnalysisCapabilityUnavailableError(
+53
View File
@@ -1,6 +1,12 @@
import { z } from "incur";
import { inspectAnalysisViewValidated } from "../application/analysisView/AnalysisViewService.js";
import { readEvidenceBundle } from "../application/EvidenceBundleFiles.js";
import { inspectEvidenceBundle } from "../application/investigation/InspectEvidenceBundle.js";
import {
getEvidenceBundleInputSchema,
importEvidenceBundleInputSchema,
} from "../contracts/sessionToolSchemas.js";
import { CLI_COMMANDS } from "../cliCommandNames.js";
import { parseCliJsonInput } from "../cliJsonInput.js";
import { logCliCommand } from "../cliLogging.js";
@@ -16,6 +22,53 @@ export const registerAnalysisViewCommands = (
cli: CliInstance,
logger: Logger,
): void => {
cli.command(CLI_COMMANDS.inspectEvidenceBundle, {
description:
"Inspect complete or selected discovery metadata from a local Evidence bundle",
args: z.object({
inputJson: z
.string()
.describe(
"JSON or JSON file with absolute path, detail and optional record filters",
),
}),
run: ({ args }) =>
logCliCommand(logger, CLI_COMMANDS.inspectEvidenceBundle, async () => {
const input = await parseCliJsonInput(
args.inputJson,
CLI_COMMANDS.inspectEvidenceBundle,
);
if (!input.ok) return input.error;
const schema = getEvidenceBundleInputSchema.safeExtend({
path: importEvidenceBundleInputSchema.shape.path,
});
const parsed = schema.safeParse(input.value);
if (!parsed.success)
return {
error: "Evidence bundle inspection failed",
...projectAnalysisError(
analysisInputErrorFromIssues(
CLI_COMMANDS.inspectEvidenceBundle,
parsed.error.issues,
input.value,
),
),
};
const loaded = await readEvidenceBundle(parsed.data.path);
if (!loaded.ok)
return {
error: "Evidence bundle inspection failed",
...projectAnalysisError(loaded.error),
};
const result = inspectEvidenceBundle(loaded.value, parsed.data);
return result.ok
? result.value
: {
error: "Evidence bundle inspection failed",
...projectAnalysisError(result.error),
};
}),
});
cli.command(CLI_COMMANDS.inspectAnalysisView, {
description:
"Project a selected view of completed layout, JavaScript application or native function Evidence JSON",
+1
View File
@@ -43,6 +43,7 @@ export const CLI_COMMANDS = Object.freeze({
inspectBinaryLayout: "inspect-binary-layout",
inspectPeResources: "inspect-pe-resources",
inspectAnalysisView: "inspect-analysis-view",
inspectEvidenceBundle: "inspect-evidence-bundle",
inspectFirmwareRegions: "inspect-firmware-regions",
extractFirmware: "extract-firmware",
inspectAndroidPackage: "inspect-android-package",
+12 -5
View File
@@ -7,7 +7,6 @@ import {
} from "../application/DirectAnalysis.js";
import { runSessionStatus as executeSessionStatus } from "../application/DirectAnalysisStatus.js";
import type { DirectAnalysisDependencies } from "../application/DirectAnalysisDependencies.js";
import { createBinarySession, createManagedBinarySession } from "./binary.js";
/** Bind one-shot analysis and status commands to one caller-selected environment. */
export const createDirectAnalysis = (
@@ -16,9 +15,14 @@ export const createDirectAnalysis = (
const environment = snapshotEnvironment(selectedEnvironment);
const dependencies: DirectAnalysisDependencies = {
readConfiguration: () => parseConfig(environment),
createBinarySession: (config, logger) =>
createBinarySession(config, logger, environment),
createManagedBinarySession,
createBinarySession: async (config, logger) => {
const { createBinarySession } = await import("./binary.js");
return createBinarySession(config, logger, environment);
},
createManagedBinarySession: async () => {
const { createManagedBinarySession } = await import("./binary.js");
return createManagedBinarySession();
},
};
return {
runDirectAnalysis: executeDirectAnalysis.bind(undefined, dependencies),
@@ -34,6 +38,9 @@ export type DirectAnalysis = ReturnType<typeof createDirectAnalysis>;
export const runManagedProviderExecution = executeManagedProvider.bind(
undefined,
{
createManagedBinarySession,
createManagedBinarySession: async () => {
const { createManagedBinarySession } = await import("./binary.js");
return createManagedBinarySession();
},
},
);
+1 -1
View File
@@ -159,7 +159,7 @@ export const SESSION_TOOL_CONTRACTS = [
),
session(
"get_evidence_bundle",
"Return every Evidence record and residual unknown currently retained by this session as one inline bundle for direct inspection or follow-up workflows.",
"Return every retained Evidence record and residual unknown as a complete canonical bundle by default. detail=summary discovers retained records without copying or transferring their payloads; optionally filter by exact Evidence ID, operation, target SHA-256, analysis-profile digest or canonical native procedure address. Metadata distinguishes complete record retention from native value-flow truncation or unavailable/unknown coverage. Use a discovered evidence_id with inspect_analysis_view without restarting a provider. Filters apply only to summary delivery.",
getEvidenceBundleInputSchema,
),
session(
+6 -2
View File
@@ -6,9 +6,13 @@ import { localPathStringSchema } from "../domain/localPath.js";
import { processTraceSpecificationSchema } from "../domain/process/processTraceSpecification.js";
import { evidenceSchema } from "../domain/evidence.js";
import { prefixedDigestSchema } from "./../domain/digests.js";
import { evidenceRecordFiltersSchema } from "../domain/evidenceRecordSummary.js";
/** Return the current canonical Evidence bundle inline. */
export const getEvidenceBundleInputSchema = z.strictObject({});
/** Complete retained Evidence, or selected discovery metadata without payloads. */
export const getEvidenceBundleInputSchema = z.strictObject({
detail: z.enum(["complete", "summary"]).default("complete"),
filters: evidenceRecordFiltersSchema.optional(),
});
/** Optional document selection for volatile navigation context. */
export const navigationContextInputSchema = z.strictObject({
+4
View File
@@ -53,6 +53,10 @@ export const TOOL_EXAMPLE_OVERRIDES: Readonly<
trace_native_ui_action: { action: "buildTapped:" },
open_binary: { path: "/tmp/fixture" },
export_evidence_bundle: { path: "/tmp/evidence.json" },
get_evidence_bundle: {
detail: "summary",
filters: { operation: "analyze_function", procedure_address: "0x1000" },
},
inspect_address_context: { address: "0x1000" },
import_evidence_bundle: { path: "/tmp/evidence.json" },
capture_process_scenario: {
+4 -1
View File
@@ -1,4 +1,5 @@
import { z } from "zod";
import { evidenceBundleSummarySchema } from "../domain/evidenceRecordSummary.js";
import { BINARY_ARCHITECTURES } from "../domain/binaryTargetTypes.js";
import { nativeFunctionAnnotationsSchema } from "../domain/native/nativeFunctionAnnotations.js";
import { nativeLoadImageSchema } from "../domain/native/nativeLoadImage.js";
@@ -470,7 +471,9 @@ export const sessionOutputSchemas = {
unknowns: z.number().int().min(0),
}),
),
get_evidence_bundle: lifecycleResultOf(evidenceBundleSchema),
get_evidence_bundle: lifecycleResultOf(
z.union([evidenceBundleSchema, evidenceBundleSummarySchema]),
),
get_navigation_context: lifecycleResultOf(
z.object({
document: z.string(),
+1 -1
View File
@@ -47,7 +47,7 @@ export const analysisErrorRemediationAction = (
return (
error.remediationAction ??
(error.resource === "transport"
? "Export retained session evidence through export_evidence_bundle to a caller-selected path, or use complete CLI JSON output. Tools that accept retained-evidence references can inspect compatible records. The connection remains usable."
? "Discover retained records with get_evidence_bundle and detail: summary, then inspect compatible records through inspect_analysis_view using an exact Evidence reference. Export complete retained session evidence through export_evidence_bundle to a caller-selected path, or use complete CLI JSON output. The connection remains usable."
: error.resource === "cpu"
? "Review the reported worker CPU limits and observed signal. Retry with sufficient CPU time or a smaller artifact; REA retains tighter inherited limits."
: error.resource === "file-size"
@@ -21,6 +21,12 @@ describe("cooperative canonical JSON digest", () => {
),
{ payload: `${"x".repeat(8191)}𐀀${"\n\ud800".repeat(20_000)}` },
{ [`${"x".repeat(8191)}𐀀${"\n".repeat(20_000)}`]: "value" },
Object.fromEntries(
Array.from({ length: 512 }, (_, index) => [
`escaped\n\"\\𐀀${"x".repeat(32)}${index}`,
{ repeated: index },
]),
),
];
it.each(values.map((value) => ({ value })))(
"preserves complete canonical meaning (%#)",
@@ -55,6 +61,18 @@ describe("cooperative canonical JSON digest", () => {
expect(steps.next().done).toBe(true);
});
it("continues object hashing when a pending property is omitted", () => {
const value: { [key: string]: JsonValue } = {
a: "x".repeat(32_768),
b: 1,
c: { retained: true },
};
const steps = canonicalJsonDigestSteps(value);
expect(steps.next().done).toBe(false);
delete value.b;
expect(complete(steps)).toBe(digestCanonicalValue(value));
});
it("rejects sparse arrays, nonfinite numbers and cycles", () => {
const sparse: JsonValue[] = [];
sparse.length = 2;
+34 -13
View File
@@ -26,6 +26,9 @@ export function* canonicalJsonDigestSteps(
const hash = createHash("sha256");
const ancestors = new Set<object>();
const stack: DigestFrame[] = [{ kind: "value", value }];
const encodedKeys = new Map<string, string>();
// Keep reusable key encodings within one existing digest-buffer quantum.
let remainingKeyCodeUnits = 8192;
let buffered = "";
let flushed = false;
const emit = (part: string): void => {
@@ -69,22 +72,22 @@ export function* canonicalJsonDigestSteps(
stack.push(frame);
continue;
}
let item: JsonValue;
if (frame.kind === "array") {
if (frame.index === frame.length) {
emit("]");
ancestors.delete(frame.value);
continue;
}
const item = frame.value[frame.index];
if (item === undefined)
const element = frame.value[frame.index];
if (element === undefined)
throw new TypeError(
"Canonical JSON digest requires dense JSON arrays",
);
if (frame.index++ > 0) emit(",");
stack.push(frame, { kind: "value", value: item });
continue;
}
if (frame.kind === "object") {
stack.push(frame);
item = element;
} else if (frame.kind === "object") {
if (frame.index === frame.keys.length) {
emit("}");
ancestors.delete(frame.value);
@@ -93,25 +96,42 @@ export function* canonicalJsonDigestSteps(
const key = frame.keys[frame.index++];
if (key === undefined)
throw new TypeError("Canonical JSON object changed during hashing");
const item = frame.value[key];
if (item === undefined) continue;
const property = frame.value[key];
if (property === undefined) {
stack.push(frame);
continue;
}
if (frame.written++ > 0) emit(",");
if (key.length > 8192) {
emit('"');
stack.push(
frame,
{ kind: "value", value: item },
{ kind: "value", value: property },
{ kind: "separator" },
{ kind: "string", value: key, index: 0 },
);
continue;
}
emit(JSON.stringify(key));
let encodedKey = encodedKeys.get(key);
if (encodedKey === undefined) {
encodedKey = JSON.stringify(key);
if (encodedKey.length <= remainingKeyCodeUnits) {
encodedKeys.set(key, encodedKey);
remainingKeyCodeUnits -= encodedKey.length;
}
}
emit(encodedKey);
emit(":");
stack.push(frame, { kind: "value", value: item });
continue;
stack.push(frame);
item = property;
} else {
item = frame.value;
}
// Resume containers directly; scalar members need no temporary value frame.
if (flushed) {
flushed = false;
yield;
}
const item = frame.value;
if (typeof item === "string" && item.length > 8192) {
emit('"');
stack.push({ kind: "string", value: item, index: 0 });
@@ -151,6 +171,7 @@ export function* canonicalJsonDigestSteps(
} finally {
stack.length = 0;
ancestors.clear();
encodedKeys.clear();
buffered = "";
}
}
+55
View File
@@ -0,0 +1,55 @@
import { expect, it } from "vitest";
import { createEvidence } from "./evidence.js";
import { createEvidenceBundle } from "./evidenceBundle.js";
import { summarizeEvidenceBundle } from "./evidenceRecordSummary.js";
it("preserves native identity independently of missing or incompatible value-flow metadata", () => {
const records = [
undefined,
{ available: true, truncated: true },
{ unexpected: "legacy" },
{ available: false, reason: "Decompiler unavailable" },
].map((flow) =>
createEvidence(
{ path: "/fixtures/native.exe", format: "pe", sha256: "a".repeat(64) },
{ id: "provider", name: "Provider", version: "1" },
{
operation: "analyze_function",
parameters: {},
result: {
procedure: { address: "space:0x10", name: "entry" },
...(flow === undefined ? {} : { native_value_flow: flow }),
},
},
),
);
const summary = summarizeEvidenceBundle(createEvidenceBundle(records), {
procedure_address: "space:0x10",
});
expect(summary.matching_records).toBe(4);
const native = summary.records.map((r) => r.native_dossier);
expect(
native.every((r) => r.available && r.procedure_address === "space:0x10"),
).toBe(true);
expect(native).toEqual(
expect.arrayContaining([
expect.objectContaining({ value_flow: { status: "not-recorded" } }),
expect.objectContaining({
value_flow: {
status: "observed",
truncated: true,
omitted_operations_lower_bound: null,
known_omitted_inputs: null,
known_omitted_edges: null,
},
}),
expect.objectContaining({
value_flow: { status: "unknown", reason: expect.any(String) },
}),
expect.objectContaining({
value_flow: { status: "unavailable", reason: "Decompiler unavailable" },
}),
]),
);
});
+166
View File
@@ -0,0 +1,166 @@
import { z } from "zod";
import { digestSchema, prefixedDigestSchema } from "./digests.js";
import { evidenceSchema, type Evidence } from "./evidence.js";
import type { EvidenceBundle } from "./evidenceBundle.js";
/** Exact retained-record selectors; selection never resolves a provider address. */
export const evidenceRecordFiltersSchema = z.strictObject({
evidence_id: prefixedDigestSchema("ev").optional(),
operation: z.string().min(1).optional(),
target_sha256: digestSchema.optional(),
analysis_profile_digest: digestSchema.optional(),
procedure_address: z.string().min(1).optional(),
});
export type EvidenceRecordFilters = z.infer<typeof evidenceRecordFiltersSchema>;
const countSchema = z.number().int().nonnegative();
const nativeMetadataSchema = z.object({
procedure: z.object({ address: z.string(), name: z.string().nullish() }),
native_value_flow: z.unknown().optional(),
});
const valueFlowMetadataSchema = z.union([
z.object({
available: z.literal(true),
truncated: z.boolean(),
omitted_operations_lower_bound: countSchema.optional(),
known_omitted_inputs: countSchema.optional(),
known_omitted_edges: countSchema.optional(),
}),
z.object({ available: z.literal(false), reason: z.string() }),
]);
const nativeSummarySchema = z.discriminatedUnion("available", [
z.strictObject({
available: z.literal(true),
procedure_address: z.string(),
procedure_name: z.string().nullable(),
value_flow: z.discriminatedUnion("status", [
z.strictObject({
status: z.literal("observed"),
truncated: z.boolean(),
omitted_operations_lower_bound: countSchema.nullable(),
known_omitted_inputs: countSchema.nullable(),
known_omitted_edges: countSchema.nullable(),
}),
z.strictObject({ status: z.literal("unavailable"), reason: z.string() }),
z.strictObject({ status: z.literal("not-recorded") }),
z.strictObject({ status: z.literal("unknown"), reason: z.string() }),
]),
}),
z.strictObject({
available: z.literal(false),
reason: z.enum(["different-operation", "not-recorded-or-incompatible"]),
}),
]);
/** Metadata for complete retained records, independent of analysis completeness. */
export const evidenceBundleSummarySchema = z.strictObject({
kind: z.literal("evidence-bundle-summary"),
filters: z.record(z.string(), z.string()),
total_retained_records: countSchema,
matching_records: countSchema,
unknown_revisions: countSchema,
records: z.array(
z.strictObject({
evidence_id: evidenceSchema.shape.evidence_id,
subject: evidenceSchema.shape.subject,
provider: evidenceSchema.shape.provider,
operation: evidenceSchema.shape.operation,
confidence: evidenceSchema.shape.confidence,
authority: evidenceSchema.shape.authority,
analysis_profile_digest: digestSchema.nullable(),
retention: z.literal("complete-record"),
limitation_count: countSchema,
native_dossier: nativeSummarySchema,
}),
),
});
export type EvidenceBundleSummary = z.infer<typeof evidenceBundleSummarySchema>;
const nativeSummary = (
evidence: Evidence,
): z.infer<typeof nativeSummarySchema> => {
if (evidence.operation !== "analyze_function")
return { available: false, reason: "different-operation" };
// Read only producer metadata. Parsing the full dossier would allocate every
// large collection merely to discover its already authenticated parent ID.
const parsed = nativeMetadataSchema.safeParse(evidence.normalized_result);
if (!parsed.success)
return { available: false, reason: "not-recorded-or-incompatible" };
const flow = valueFlowMetadataSchema.safeParse(parsed.data.native_value_flow);
return {
available: true,
procedure_address: parsed.data.procedure.address,
procedure_name: parsed.data.procedure.name ?? null,
value_flow:
parsed.data.native_value_flow == null
? { status: "not-recorded" }
: !flow.success
? {
status: "unknown",
reason: "Value-flow metadata has an incompatible representation.",
}
: flow.data.available
? {
status: "observed",
truncated: flow.data.truncated,
omitted_operations_lower_bound:
flow.data.omitted_operations_lower_bound ?? null,
known_omitted_inputs: flow.data.known_omitted_inputs ?? null,
known_omitted_edges: flow.data.known_omitted_edges ?? null,
}
: { status: "unavailable", reason: flow.data.reason },
};
};
/** Discover retained Evidence without reading or copying its complete payloads. */
export const summarizeEvidenceBundle = (
bundle: EvidenceBundle,
filters: EvidenceRecordFilters = {},
): EvidenceBundleSummary => {
const selectedFilters: Record<string, string> = {};
for (const [field, value] of Object.entries(filters))
if (value !== undefined) selectedFilters[field] = value;
const records: EvidenceBundleSummary["records"] = [];
for (const evidence of bundle.records) {
if (
(filters.evidence_id !== undefined &&
filters.evidence_id !== evidence.evidence_id) ||
(filters.operation !== undefined &&
filters.operation !== evidence.operation) ||
(filters.target_sha256 !== undefined &&
filters.target_sha256 !== evidence.subject?.digest.sha256) ||
(filters.analysis_profile_digest !== undefined &&
filters.analysis_profile_digest !== evidence.analysis_profile?.digest)
)
continue;
const native = nativeSummary(evidence);
if (
filters.procedure_address !== undefined &&
(!native.available ||
filters.procedure_address !== native.procedure_address)
)
continue;
records.push({
evidence_id: evidence.evidence_id,
subject: evidence.subject,
provider: evidence.provider,
operation: evidence.operation,
confidence: evidence.confidence,
authority: evidence.authority,
analysis_profile_digest: evidence.analysis_profile?.digest ?? null,
retention: "complete-record",
limitation_count: evidence.limitations.length,
native_dossier: native,
});
}
return {
kind: "evidence-bundle-summary",
filters: selectedFilters,
total_retained_records: bundle.records.length,
matching_records: records.length,
unknown_revisions: bundle.unknowns.length,
records,
};
};
@@ -33,7 +33,7 @@ export const inspectElectronBrowserNode = (
context: JavaScriptFindingContext,
): void => {
if (t.isNewExpression(node)) inspectBrowserWindow(node, context);
if (!t.isCallExpression(node)) return;
if (!t.isCallExpression(node) && !t.isOptionalCallExpression(node)) return;
inspectContextBridge(node, context);
inspectUtilityProcess(node, context);
};
@@ -162,7 +162,7 @@ const collectWebPreferences = (
};
const inspectContextBridge = (
node: t.CallExpression,
node: t.CallExpression | t.OptionalCallExpression,
context: JavaScriptFindingContext,
): void => {
const path = electronCalleePath(node.callee, context);
@@ -208,7 +208,7 @@ const inspectContextBridge = (
};
const inspectUtilityProcess = (
node: t.CallExpression,
node: t.CallExpression | t.OptionalCallExpression,
context: JavaScriptFindingContext,
): void => {
const path = electronCalleePath(node.callee, context);
@@ -0,0 +1,76 @@
import { describe, expect, it } from "vitest";
import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js";
describe("Electron API identity", () => {
it("recovers APIs through an ES module default import of Electron", () => {
const analysis = analyzeJavaScriptStaticSource(`
import electron from "electron";
const { ipcMain } = electron;
new electron.BrowserWindow({ webPreferences: { preload: "./preload.js" } });
electron.contextBridge.exposeInMainWorld("api", { ping: true });
electron.ipcRenderer.invoke("default:ping");
ipcMain.handle("destructured:ping", () => "pong");
electron.default.ipcMain.handle("nested-default", () => "local");
`);
expect(analysis.electron.browser_windows).toMatchObject([
{ preload_path: "./preload.js" },
]);
expect(analysis.electron.context_bridge_apis).toMatchObject([
{ api_key: "api", members: ["ping"] },
]);
expect(
analysis.electron.ipc.map(({ side, operation, channel }) => ({
side,
operation,
channel,
})),
).toEqual([
{ side: "renderer", operation: "invoke", channel: "default:ping" },
{ side: "main", operation: "handle", channel: "destructured:ping" },
]);
});
it("recovers APIs and sender checks written with optional chaining", () => {
const analysis = analyzeJavaScriptStaticSource(`
const { contextBridge, ipcMain, utilityProcess } = require("electron");
contextBridge?.exposeInMainWorld("api", {});
utilityProcess?.fork("./worker.js");
ipcMain?.handle("optional:ping", (event) => {
if (!event.senderFrame?.url.startsWith("file://")) return null;
return event.sender?.getURL() === "file:///index.html";
});
`);
expect(analysis.electron.context_bridge_apis).toMatchObject([
{ api_key: "api" },
]);
expect(analysis.electron.utility_processes).toMatchObject([
{ module_path: "./worker.js" },
]);
expect(analysis.electron.ipc).toMatchObject([
{ side: "main", operation: "handle", channel: "optional:ping" },
]);
expect(
analysis.electron.sender_validations.map(
({ subject, mechanism, expected }) => ({
subject,
mechanism,
expected: expected.value,
}),
),
).toEqual([
{
subject: "sender-url",
mechanism: "call:startsWith",
expected: "file://",
},
{
subject: "sender-url",
mechanism: "comparison:===",
expected: "file:///index.html",
},
]);
});
});
@@ -26,7 +26,7 @@ export const inspectElectronIpcNode = (
node: t.Node,
context: JavaScriptFindingContext,
): void => {
if (t.isCallExpression(node)) {
if (t.isCallExpression(node) || t.isOptionalCallExpression(node)) {
inspectIpcCall(node, context);
inspectValidationCall(node, context);
}
@@ -34,7 +34,7 @@ export const inspectElectronIpcNode = (
};
const inspectIpcCall = (
node: t.CallExpression,
node: t.CallExpression | t.OptionalCallExpression,
context: JavaScriptFindingContext,
): void => {
const descriptor = ipcDescriptor(electronCalleePath(node.callee, context));
@@ -117,7 +117,7 @@ const inspectValidationComparison = (
};
const inspectValidationCall = (
node: t.CallExpression,
node: t.CallExpression | t.OptionalCallExpression,
context: JavaScriptFindingContext,
): void => {
if (
@@ -225,9 +225,10 @@ const ipcDescriptor = (
const senderSubject = (
node: t.Node,
): ElectronSenderValidationFinding["subject"] | undefined => {
const name = t.isCallExpression(node)
? calleeName(node.callee)
: calleeName(node);
const name =
t.isCallExpression(node) || t.isOptionalCallExpression(node)
? calleeName(node.callee)
: calleeName(node);
if (/(?:^|\.)sender\.getURL$/u.test(name)) return "sender-url";
if (/(?:^|\.)senderFrame\.origin$/u.test(name)) return "sender-origin";
if (/(?:^|\.)senderFrame\.url$/u.test(name)) return "sender-url";
@@ -12,6 +12,16 @@ import { traverseJavaScriptAstSteps } from "./javascriptSemanticTraversal.js";
export const ELECTRON_MODULE =
/^electron(?:\/(?:common|main|renderer|utility))?$/u;
/**
* Electron's CommonJS exports define no `default` member, so a leading
* `default` segment can only denote the module object itself, as an ES module
* default import (`import electron from "electron"`) binds it.
*/
export const electronExportPath = (
importedPath: readonly string[],
): readonly string[] =>
importedPath[0] === "default" ? importedPath.slice(1) : importedPath;
/** Static identity recovery cannot establish the absence of Electron usage. */
export const ELECTRON_IDENTITY_LIMITATION =
"Electron API identity follows proven lexical module aliases. Reassigned bindings, dynamic selections, and visibly overwritten namespace members are unresolved; missing findings do not establish absence. Arbitrary call side effects and runtime registration are not evaluated.";
@@ -0,0 +1,329 @@
import { describe, expect, it } from "vitest";
import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js";
const endpoints = (source: string) =>
analyzeJavaScriptStaticSource(source).endpoints.map(
({ kind, value, mechanism }) => ({ kind, value, mechanism }),
);
const rpcEndpoints = (source: string) =>
endpoints(source).filter(
({ mechanism }) =>
mechanism.startsWith("trpc:") || mechanism.startsWith("graphql:"),
);
describe("tRPC procedure recognition", () => {
it("recognizes React hook procedures on a renamed proxy root", () => {
expect(
rpcEndpoints(`
const a = n.post.byId.useQuery({ id: 1 });
const b = n.post.add.useMutation();
n.feed.useInfiniteQuery({ limit: 10 });
n.onEvent.useSubscription(undefined, { onData() {} });
n.invoice.list.useSuspenseQuery();
`),
).toEqual([
{
kind: "network",
value: "post.byId",
mechanism: "trpc:procedure:useQuery",
},
{
kind: "network",
value: "post.add",
mechanism: "trpc:procedure:useMutation",
},
{
kind: "network",
value: "feed",
mechanism: "trpc:procedure:useInfiniteQuery",
},
{
kind: "network",
value: "onEvent",
mechanism: "trpc:procedure:useSubscription",
},
{
kind: "network",
value: "invoice.list",
mechanism: "trpc:procedure:useSuspenseQuery",
},
]);
});
it("recognizes TanStack React Query integration option factories", () => {
expect(
rpcEndpoints(`
useQuery(e.invoice.list.queryOptions({ page: 1 }));
useMutation(e.invoice.create.mutationOptions());
client.invalidateQueries({ queryKey: e.invoice.list.queryKey() });
`),
).toEqual([
{
kind: "network",
value: "invoice.list",
mechanism: "trpc:procedure:queryOptions",
},
{
kind: "network",
value: "invoice.create",
mechanism: "trpc:procedure:mutationOptions",
},
]);
});
it("recognizes vanilla client calls and link URLs only in sources carrying a tRPC marker", () => {
expect(
rpcEndpoints(`
const client = createTRPCClient({
links: [httpBatchLink({ url: "/api/trpc" })],
});
const bilbo = await client.getUser.query("id_bilbo");
await client.post.add.mutate({ title: "x" });
client.onPost.subscribe(undefined, { onData() {} });
`),
).toEqual([
{
kind: "network",
value: "/api/trpc",
mechanism: "trpc:link:httpBatchLink",
},
{ kind: "network", value: "getUser", mechanism: "trpc:procedure:query" },
{
kind: "network",
value: "post.add",
mechanism: "trpc:procedure:mutate",
},
{
kind: "network",
value: "onPost",
mechanism: "trpc:procedure:subscribe",
},
]);
expect(
rpcEndpoints(`
const rows = await pool.users.query("select 1");
emitter.changes.subscribe(listener);
store.cart.mutate(update);
`),
).toEqual([]);
});
it("requires corroboration for request methods shared with Apollo, RxJS and stores", () => {
// The source carries a tRPC marker, so only the call shape can decide.
expect(
rpcEndpoints(`
class TRPCClientError extends Error {}
this.client.query({ query: q });
e.queryManager.query(options);
this.untypedClient.query("path", input);
this.route.params.subscribe((params) => params);
t.current.subscribe(listener);
r.store.subscribe(() => {});
n.cache.mutate(update);
o.k.endpoints.getPosts.useQuery();
e.api.endpoints.list.useQuery();
this.trpc.invoice.list.useQuery();
`),
).toEqual([]);
expect(
rpcEndpoints(`
class TRPCClientError extends Error {}
e.invoice.list.query({ page: 1 });
e.invoice.onChange.subscribe(undefined, { onData() {} });
`),
).toEqual([
{
kind: "network",
value: "invoice.list",
mechanism: "trpc:procedure:query",
},
{
kind: "network",
value: "invoice.onChange",
mechanism: "trpc:procedure:subscribe",
},
]);
});
it("never reports browser globals or RTK Query endpoints as procedures", () => {
expect(
rpcEndpoints(`
class TRPCClientError extends Error {}
navigator.permissions.query({ name: "geolocation" });
window.navigator.permissions.query({ name: "camera" });
api.endpoints.getPosts.useQuery();
api.useGetPostsQuery();
useQuery({ queryKey: ["x"] });
`),
).toEqual([]);
});
});
describe("tRPC binding identity and bundled syntax", () => {
it("keeps client bindings local to their lexical declaration", () => {
expect(
rpcEndpoints(`
function first() { const c = createTRPCClient({}); c.users.query(); }
function unrelated(c) { c.users.query("select 1"); }
const c = createTRPCClient({});
function closure() { c.posts.query(); }
function shadowed() { c.users.query(); var c = pool; }
{ const c = pool; c.users.query(); }
try {} catch (c) { c.users.query(); }
`),
).toEqual([
{ kind: "network", value: "users", mechanism: "trpc:procedure:query" },
{ kind: "network", value: "posts", mechanism: "trpc:procedure:query" },
]);
});
it("does not corroborate overwritten or destructured client bindings", () => {
expect(
rpcEndpoints(`
let c = createTRPCClient({}); c = pool; c.users.query();
const { client } = createTRPCClient({}); client.users.query();
`),
).toEqual([]);
expect(
rpcEndpoints(`
var c = createTRPCClient({}); c.users.query();
`),
).toEqual([
{ kind: "network", value: "users", mechanism: "trpc:procedure:query" },
]);
});
it("recognizes surviving factory names inside bundled call wrappers", () => {
expect(
rpcEndpoints(`
const c = (0, t.createTRPCClient)({
links: [(0, t.httpBatchLink)({ url: "/api/rpc" })],
});
c.users.query();
const p = (t.createTRPCProxyClient as any)({});
p.posts.query();
(0, t.httpLink)({ url: "/api/other" });
`),
).toEqual([
{
kind: "network",
value: "/api/rpc",
mechanism: "trpc:link:httpBatchLink",
},
{ kind: "network", value: "users", mechanism: "trpc:procedure:query" },
{ kind: "network", value: "posts", mechanism: "trpc:procedure:query" },
{ kind: "network", value: "/api/other", mechanism: "trpc:link:httpLink" },
]);
});
it("preserves procedure paths beyond the former nesting cutoff", () => {
const path = Array.from({ length: 64 }, (_, i) => `router${i}`).join(".");
expect(rpcEndpoints(`root.${path}.useQuery();`)).toEqual([
{ kind: "network", value: path, mechanism: "trpc:procedure:useQuery" },
]);
});
});
describe("GraphQL operation recognition", () => {
it("recognizes named operations in tagged templates and string literals", () => {
expect(
rpcEndpoints(`
const GET_USER = gql\`
# comment
query GetUser($id: ID!) { user(id: $id) { name } }
\`;
const UPDATE = "mutation UpdateInvoice($input: InvoiceInput!) { updateInvoice(input: $input) { id } }";
const ON = \`subscription OnInvoice { invoiceChanged { id } } \${fragment}\`;
`),
).toEqual([
{ kind: "network", value: "GetUser", mechanism: "graphql:query" },
{
kind: "network",
value: "UpdateInvoice",
mechanism: "graphql:mutation",
},
{
kind: "network",
value: "OnInvoice",
mechanism: "graphql:subscription",
},
]);
});
it("retains operations after long ignored prefixes and long headers", () => {
const name = "Operation" + "x".repeat(512);
const text =
"#" +
"comment".repeat(700) +
"\nquery " +
name +
" ".repeat(600) +
"{ users { id } }";
expect(rpcEndpoints(`const doc = ${JSON.stringify(text)};`)).toEqual([
{ kind: "network", value: name, mechanism: "graphql:query" },
]);
});
it("recognizes precompiled GraphQL document objects", () => {
expect(
rpcEndpoints(`
const doc = { kind: "Document", definitions: [{
kind: "OperationDefinition",
operation: "query",
name: { kind: "Name", value: "ListInvoices" },
selectionSet: { kind: "SelectionSet", selections: [] },
}] };
`),
).toEqual([
{ kind: "network", value: "ListInvoices", mechanism: "graphql:query" },
]);
});
it("ignores anonymous operations and ordinary prose", () => {
expect(
rpcEndpoints(`
const a = "query { viewer { id } }";
const b = "query string parameters";
const c = "mutation observer started";
const d = { kind: "OperationDefinition", operation: "query" };
const e = "mutation Observer {";
const f = "query Search(";
const g = "subscription Plan (monthly)";
`),
).toEqual([]);
});
it("reports a precompiled graphql-tag document once, not again from loc.source.body", () => {
expect(
rpcEndpoints(`
const doc = { kind: "Document", definitions: [{
kind: "OperationDefinition",
operation: "query",
name: { kind: "Name", value: "GetUser" },
selectionSet: { kind: "SelectionSet", selections: [] },
}], loc: { start: 0, end: 40, source: {
body: "query GetUser($id: ID!) { user(id: $id) { name } }",
name: "GraphQL request",
locationOffset: { line: 1, column: 1 },
} } };
`),
).toEqual([
{ kind: "network", value: "GetUser", mechanism: "graphql:query" },
]);
});
});
it("keeps existing network endpoint recognition unchanged", () => {
expect(
endpoints(`fetch("/api/invoices"); n.invoice.list.useQuery();`),
).toEqual([
{ kind: "network", value: "/api/invoices", mechanism: "call:fetch" },
{
kind: "network",
value: "invoice.list",
mechanism: "trpc:procedure:useQuery",
},
]);
});
@@ -0,0 +1,322 @@
import * as t from "@babel/types";
import {
unwrapJavaScriptExpression,
semanticStaticPropertyKey,
} from "./javascriptAstValues.js";
import { calleeName } from "./javascriptStaticAnalysisHelpers.js";
/** One RPC operation or transport URL recovered from client-side syntax. */
export interface JavaScriptRpcEndpoint {
readonly value: string;
readonly mechanism: string;
}
/** Per-source RPC recognition state; feed it every AST node in pre-order. */
export interface JavaScriptRpcScan {
inspect(node: t.Node): JavaScriptRpcEndpoint | undefined;
}
// tRPC proxies preserve router and procedure names as property names, so they
// survive identifier minification. Method names follow @trpc/react-query and
// @trpc/tanstack-react-query 11.19 documentation; cache-key helpers such as
// queryKey are excluded because they never issue a request.
const TRPC_HOOK_METHODS: ReadonlySet<string> = new Set([
"useQuery",
"useSuspenseQuery",
"useInfiniteQuery",
"useSuspenseInfiniteQuery",
"usePrefetchQuery",
"usePrefetchInfiniteQuery",
"useMutation",
"useSubscription",
"queryOptions",
"infiniteQueryOptions",
"mutationOptions",
"subscriptionOptions",
]);
// @trpc/client 11.19 clientCallTypeMap methods; common outside tRPC (Apollo,
// database pools, stores), so they need corroborating evidence.
const TRPC_REQUEST_METHODS: ReadonlySet<string> = new Set(["query", "mutate"]);
// @trpc/client 11.19 TRPCSubscriptionObserver callbacks; RxJS observers use
// next/error/complete instead.
const TRPC_SUBSCRIPTION_CALLBACKS: ReadonlySet<string> = new Set([
"onStarted",
"onData",
"onError",
"onStopped",
"onComplete",
"onConnectionStateChange",
]);
// @trpc/client 11.19 client factories (createTRPCProxyClient is the
// deprecated alias of createTRPCClient).
const TRPC_CLIENT_FACTORIES: ReadonlySet<string> = new Set([
"createTRPCClient",
"createTRPCProxyClient",
]);
// Built-in HTTP links documented by tRPC 11.19 that carry a `url` option.
const TRPC_URL_LINKS: ReadonlySet<string> = new Set([
"httpLink",
"httpBatchLink",
"httpBatchStreamLink",
"httpSubscriptionLink",
]);
// Literals retained by @trpc/client 11.19 bundles (`this.name`, the default
// accept header key) or by the conventional transport path.
const TRPC_SOURCE_MARKERS = ["TRPCClientError", "trpc-accept", "/trpc"];
// Browser-owned roots whose methods share tRPC names, such as
// navigator.permissions.query.
const BROWSER_GLOBAL_NAMES: ReadonlySet<string> = new Set([
"window",
"self",
"globalThis",
"document",
"navigator",
"location",
"history",
"console",
"performance",
"screen",
"top",
"parent",
"frames",
]);
const PROCEDURE_SEGMENT = /^[A-Za-z_$][\w$]*$/u;
interface MemberPath {
readonly root: t.Node;
/** Non-computed property names from the root outward. */
readonly properties: readonly string[];
}
/** Read a static member path and the expression it starts from. */
const memberPath = (callee: t.Node): MemberPath | undefined => {
const reversed: string[] = [];
let current: t.Node = callee;
while (
t.isMemberExpression(current) ||
t.isOptionalMemberExpression(current)
) {
const property = semanticStaticPropertyKey(
current.property,
current.computed,
);
if (property === null || !PROCEDURE_SEGMENT.test(property))
return undefined;
reversed.push(property);
current = current.object;
}
return { root: current, properties: reversed.reverse() };
};
const objectKeys = (node: t.Node | undefined): readonly string[] =>
t.isObjectExpression(node)
? node.properties.flatMap((property) => {
if (!t.isObjectProperty(property) && !t.isObjectMethod(property))
return [];
const key = semanticStaticPropertyKey(property.key, property.computed);
return key === null ? [] : [key];
})
: [];
const objectStringProperty = (
node: t.ObjectExpression,
key: string,
): string | undefined => {
for (const property of node.properties)
if (
t.isObjectProperty(property) &&
semanticStaticPropertyKey(property.key, property.computed) === key &&
t.isStringLiteral(property.value)
)
return property.value.value;
return undefined;
};
const objectProperty = (
node: t.ObjectExpression,
key: string,
): t.Node | undefined => {
for (const property of node.properties)
if (
t.isObjectProperty(property) &&
semanticStaticPropertyKey(property.key, property.computed) === key
)
return property.value;
return undefined;
};
const GRAPHQL_OPERATION_TYPES: ReadonlySet<string> = new Set([
"query",
"mutation",
"subscription",
]);
// A named GraphQL operation header (GraphQL October 2021, OperationDefinition):
// type, name, then variable definitions starting with `$`, a directive, or a
// selection set starting with a field name.
const GRAPHQL_OPERATION_HEADER =
/^(query|mutation|subscription)[\s,]+([_A-Za-z][_0-9A-Za-z]*)[\s,]*(?:\([\s,]*\$|@[_A-Za-z]|\{[\s,]*[_A-Za-z])/u;
const isGraphqlIgnoredCharacter = (character: string): boolean =>
character === "," || character === "" || /\s/u.test(character);
/** Recover a named operation from GraphQL document text. */
export const graphqlOperationFromText = (
text: string,
): JavaScriptRpcEndpoint | undefined => {
// Skip GraphQL ignored tokens: whitespace, commas, BOM and # comments.
const limit = text.length;
let index = 0;
while (index < limit) {
const character = text.charAt(index);
if (isGraphqlIgnoredCharacter(character)) index += 1;
else if (character === "#") {
while (index < limit && !/[\n\r]/u.test(text.charAt(index))) index += 1;
} else break;
}
if (index >= limit) return undefined;
const header = GRAPHQL_OPERATION_HEADER.exec(text.slice(index));
const type = header?.[1];
const name = header?.[2];
return type === undefined || name === undefined
? undefined
: { value: name, mechanism: `graphql:${type}` };
};
/** Recover a named operation from a precompiled GraphQL document object. */
export const graphqlOperationFromDocumentNode = (
node: t.ObjectExpression,
): JavaScriptRpcEndpoint | undefined => {
if (objectStringProperty(node, "kind") !== "OperationDefinition")
return undefined;
const type = objectStringProperty(node, "operation");
if (type === undefined || !GRAPHQL_OPERATION_TYPES.has(type))
return undefined;
const name = objectProperty(node, "name");
if (
!t.isObjectExpression(name) ||
objectStringProperty(name, "kind") !== "Name"
)
return undefined;
const value = objectStringProperty(name, "value");
return value === undefined
? undefined
: { value, mechanism: `graphql:${type}` };
};
/** Bundlers use `(0, imported.factory)` to call an import without a receiver. */
const rpcFactoryName = (callee: t.Node): string => {
let current = unwrapJavaScriptExpression(callee).node;
while (t.isSequenceExpression(current)) {
const last = current.expressions.at(-1);
if (last === undefined) return "";
current = unwrapJavaScriptExpression(last).node;
}
return calleeName(current);
};
/** Whether an initializer has the syntax of a documented tRPC client factory. */
export const isJavaScriptRpcClientFactory = (node: t.Node): boolean => {
const call = unwrapJavaScriptExpression(node).node;
return (
(t.isCallExpression(call) || t.isOptionalCallExpression(call)) &&
TRPC_CLIENT_FACTORIES.has(
rpcFactoryName(call.callee).split(".").pop() ?? "",
)
);
};
/** Recover the literal `url` option passed to a documented tRPC HTTP link. */
const trpcLinkUrl = (
node: t.CallExpression | t.OptionalCallExpression,
): JavaScriptRpcEndpoint | undefined => {
const name = rpcFactoryName(node.callee);
const factory = name.slice(name.lastIndexOf(".") + 1);
if (!TRPC_URL_LINKS.has(factory)) return undefined;
const options = node.arguments[0];
if (!t.isObjectExpression(options)) return undefined;
const url = objectStringProperty(options, "url");
return url === undefined
? undefined
: { value: url, mechanism: `trpc:link:${factory}` };
};
/**
* Create the per-source scan. Hook and option-factory methods are distinctive.
* `query`/`mutate` need a root bound to a tRPC client factory, or a tRPC source
* marker with a router path of at least two segments; `subscribe` needs the
* marker and a tRPC observer argument. `this` roots, browser globals and RTK
* Query `endpoints` chains are never procedures.
*/
export const createJavaScriptRpcScan = (
source: string,
clientRoots: ReadonlySet<t.Node>,
): JavaScriptRpcScan => {
let marker: boolean | undefined;
const hasMarker = (): boolean =>
(marker ??= TRPC_SOURCE_MARKERS.some((value) => source.includes(value)));
// graphql-tag documents retain their source text in loc.source.body; that
// text repeats the OperationDefinition already recovered from the object.
const documentSourceBodies = new WeakSet<t.Node>();
const procedure = (
node: t.CallExpression | t.OptionalCallExpression,
): JavaScriptRpcEndpoint | undefined => {
const path = memberPath(node.callee);
if (path === undefined || t.isThisExpression(path.root)) return undefined;
if (t.isIdentifier(path.root) && BROWSER_GLOBAL_NAMES.has(path.root.name))
return undefined;
const method = path.properties[path.properties.length - 1];
const segments = path.properties.slice(0, -1);
const first = segments[0];
if (method === undefined || first === undefined) return undefined;
if (BROWSER_GLOBAL_NAMES.has(first) || segments.includes("endpoints"))
return undefined;
const bound = clientRoots.has(path.root);
const recognized = TRPC_HOOK_METHODS.has(method)
? true
: TRPC_REQUEST_METHODS.has(method)
? bound || (segments.length >= 2 && hasMarker())
: method === "subscribe" &&
(bound || hasMarker()) &&
objectKeys(node.arguments[1]).some((key) =>
TRPC_SUBSCRIPTION_CALLBACKS.has(key),
);
return recognized
? { value: segments.join("."), mechanism: `trpc:procedure:${method}` }
: undefined;
};
return {
inspect: (node) => {
if (t.isCallExpression(node) || t.isOptionalCallExpression(node))
return trpcLinkUrl(node) ?? procedure(node);
if (t.isObjectExpression(node)) {
const body = objectProperty(node, "body");
if (
t.isStringLiteral(body) &&
objectProperty(node, "locationOffset") !== undefined
) {
documentSourceBodies.add(body);
return undefined;
}
return graphqlOperationFromDocumentNode(node);
}
if (t.isStringLiteral(node))
return documentSourceBodies.has(node)
? undefined
: graphqlOperationFromText(node.value);
if (t.isTemplateLiteral(node)) {
// A template recovered without a cooked value stays dynamic.
const cooked = node.quasis[0]?.value.cooked;
return cooked === undefined || cooked === null
? undefined
: graphqlOperationFromText(cooked);
}
return undefined;
},
};
};
@@ -47,6 +47,7 @@ import { collectJavaScriptDerivedSemanticsSteps } from "./javascriptSemanticDeri
import { evaluateSemanticProvenance } from "./javascriptSemanticValues.js";
import {
collectElectronMemberWritesSteps,
electronExportPath,
electronMemberPath,
ELECTRON_MODULE,
} from "./javascriptElectronMemberWrites.js";
@@ -59,6 +60,8 @@ import {
type ParsedJavaScriptSource,
} from "./javascriptSourceParser.js";
import { isJavaScriptRpcClientFactory } from "./javascriptRpcEndpoints.js";
interface BindPatternInput {
readonly pattern: t.Node;
readonly initializer: t.Node | null;
@@ -181,6 +184,40 @@ export function* analyzeParsedJavaScriptSemanticsSteps(
};
}
/** Resolve tRPC client roots by lexical identity, excluding overwritten bindings. */
export function* classifyParsedJavaScriptRpcClientRootsSteps(
file: ParsedJavaScriptSource,
): Generator<void, ReadonlySet<t.Node>> {
const state = createState(file.program);
yield* collectDefinitionsSteps(file.program, state);
const roots = new Set<t.Node>();
const clients = new Map<string, boolean>();
yield* traverseJavaScriptAstSteps(file.program, {
enter: (node) => {
if (!t.isMemberExpression(node) && !t.isOptionalMemberExpression(node))
return;
const root = node.object;
if (!t.isIdentifier(root)) return;
const binding = resolveSemanticBindingState(state, root, root.name);
if (binding === undefined) return;
let client = clients.get(binding.bindingId);
if (client === undefined) {
const [initializer] = binding.initializers;
client =
initializer !== undefined &&
initializer.projection.length === 0 &&
binding.initializers.length === 1 &&
!binding.definitions.some(({ kind }) => kind === "assignment") &&
!state.conditionalInitializers.has(initializer.node) &&
isJavaScriptRpcClientFactory(initializer.node);
clients.set(binding.bindingId, client);
}
if (client) roots.add(root);
},
});
return roots;
}
/** Receiver identity facts for the overloaded `.open` syntax only. */
export type JavaScriptOpenReceiverFact =
| "window"
@@ -349,7 +386,7 @@ export function* classifyParsedJavaScriptElectronBindingsSteps(
}
const origin = semanticRequireOrigin(root, state);
return origin !== undefined && ELECTRON_MODULE.test(origin.specifier)
? origin.importedPath
? electronExportPath(origin.importedPath)
: undefined;
};
const writes = yield* collectElectronMemberWritesSteps(
@@ -400,7 +437,7 @@ const electronExport = (
? provenance.origins[0]
: undefined);
return origin !== undefined && ELECTRON_MODULE.test(origin.specifier)
? origin.importedPath
? electronExportPath(origin.importedPath)
: null;
};
@@ -23,6 +23,7 @@ import {
inspectCall,
inspectRouteProperty,
inspectRoleProperty,
inspectRpcNode,
} from "./javascriptStaticAnalysisCalls.js";
import {
inspectBundlerRegistration,
@@ -37,6 +38,7 @@ import {
import type { JavaScriptStaticAnalysis } from "./javascriptStaticAnalysisTypes.js";
import {
classifyParsedJavaScriptElectronBindingsSteps,
classifyParsedJavaScriptRpcClientRootsSteps,
classifyParsedJavaScriptOpenReceivers,
type JavaScriptOpenReceiverFact,
} from "./javascriptSemanticAnalysis.js";
@@ -45,6 +47,8 @@ import {
type ParsedJavaScriptSource,
} from "./javascriptSourceParser.js";
import { createJavaScriptRpcScan } from "./javascriptRpcEndpoints.js";
/** Parse one JavaScript artifact and recover static structure only. */
export const analyzeJavaScriptStaticSource = (
source: string,
@@ -78,11 +82,13 @@ export function* analyzeParsedJavaScriptStaticSourceSteps(
const accumulator = createJavaScriptAnalysisAccumulator(source.length);
const electronBindings =
yield* classifyParsedJavaScriptElectronBindingsSteps(file);
const clientRoots = yield* classifyParsedJavaScriptRpcClientRootsSteps(file);
yield* traverseStaticSourceSteps(file, {
source,
accumulator,
...(openReceiverFacts === undefined ? {} : { openReceiverFacts }),
electronBindings,
rpcScan: createJavaScriptRpcScan(source, clientRoots),
});
addSourceMapDirectives(source, file.comments ?? [], accumulator);
return finalizeStaticAnalysis(source, file, accumulator);
@@ -223,4 +229,5 @@ const inspectNode = (
inspectRouteProperty(node, findings);
inspectRoleProperty(node, findings);
}
inspectRpcNode(node, findings);
};
@@ -201,6 +201,13 @@ const inspectEndpointCall = (
});
};
/** Record tRPC procedures, tRPC link URLs and named GraphQL operations. */
export const inspectRpcNode = (node: t.Node, context: FindingContext): void => {
const rpc = context.rpcScan.inspect(node);
if (rpc !== undefined)
addEndpoint(context, { node, kind: "network", ...rpc });
};
export const inspectRouteProperty = (
node: t.ObjectProperty,
context: FindingContext,
@@ -1,4 +1,5 @@
import type * as t from "@babel/types";
import type { JavaScriptRpcScan } from "./javascriptRpcEndpoints.js";
import type {
JavaScriptBundlerRegistration,
@@ -57,6 +58,7 @@ export interface JavaScriptAnalysisAccumulator {
/** Shared source and accumulator for helper inspections. */
export interface JavaScriptFindingContext {
readonly rpcScan: JavaScriptRpcScan;
readonly source: string;
readonly accumulator: JavaScriptAnalysisAccumulator;
readonly openReceiverFacts?: ReadonlyMap<
@@ -0,0 +1,176 @@
import { describe, expect, it } from "vitest";
import type { NativeAotParserLimits } from "./nativeAotPe.js";
import {
NATIVE_AOT_MAX_REPORT_BYTES,
parseNativeAotPe,
} from "./nativeAotPe.js";
import {
nativeAotPeDigest,
nativeAotPeFixture,
} from "../../../tests/fixtures/nativeaotPe.js";
const REPORT_BUDGET = NATIVE_AOT_MAX_REPORT_BYTES;
describe("NativeAOT parser resource budgets", () => {
it("reports source-cap rejection without hashing or inventing omitted-byte coverage", () => {
const bytes = nativeAotPeFixture();
const sha = nativeAotPeDigest(bytes);
const limits: NativeAotParserLimits = {
sourceBytes: bytes.length - 1,
workUnits: 1024,
workingMemoryBytes: 1024,
};
const report = parseNativeAotPe(
bytes,
sha,
REPORT_BUDGET,
undefined,
limits,
).summary;
expect(report).toMatchObject({
status: "partial",
analysis_artifact_sha256: sha,
header_address: null,
coverage: {
source_bytes_examined: 0,
source_byte_budget_bytes: bytes.length - 1,
truncation_reason: "source-byte-budget",
},
});
expect(report.coverage.source_bytes_omitted).toBeUndefined();
});
});
describe("NativeAOT parser report admission", () => {
it("never returns an over-budget report from early exits", () => {
const bytes = nativeAotPeFixture();
const sha = nativeAotPeDigest(bytes);
expect(() =>
parseNativeAotPe(bytes, sha, 0, undefined, {
sourceBytes: bytes.length - 1,
workUnits: 1024,
workingMemoryBytes: 1024,
}),
).toThrow(/minimum truthful report/iu);
const nonPe = Buffer.from("not a PE image");
expect(() => parseNativeAotPe(nonPe, nativeAotPeDigest(nonPe), 0)).toThrow(
/minimum truthful report/iu,
);
});
it("does not claim a missing RTR header when signature coverage hits the work limit", () => {
const bytes = nativeAotPeFixture();
const limits: NativeAotParserLimits = {
sourceBytes: bytes.length,
workUnits: 8,
workingMemoryBytes: 1024,
};
const report = parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
REPORT_BUDGET,
undefined,
limits,
).summary;
expect(report).toMatchObject({
status: "partial",
header_address: null,
truncated: true,
coverage: {
rtr_signature_scan_bytes_examined: 8,
rtr_signature_scan_bytes_omitted: expect.any(Number),
truncation_reason: "work-unit-budget",
},
});
expect(report.reason).not.toMatch(/No supported ReadyToRun directory/iu);
});
it("does not claim a missing root when table discovery is incomplete", () => {
const bytes = nativeAotPeFixture();
const report = parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
REPORT_BUDGET,
undefined,
{
sourceBytes: bytes.length,
workUnits: 10_000,
workingMemoryBytes: 1024,
},
).summary;
expect(report.truncated).toBe(true);
expect(report.reason).toMatch(/before determining whether.*root exists/iu);
expect(report.reason).not.toMatch(
/no unique System\.Object.*fully examined/iu,
);
});
});
describe("NativeAOT detail and summary byte accounting", () => {
it("admits serialized MethodTable memory descriptors before emitting them", () => {
const bytes = nativeAotPeFixture();
const sha = nativeAotPeDigest(bytes);
const detail = (budget: number) =>
parseNativeAotPe(bytes, sha, REPORT_BUDGET).readTypeDetail(
"0x140002280",
budget,
);
const complete = detail(REPORT_BUDGET);
expect(complete?.derived_memory).not.toBeNull();
let low = 0;
let high = Buffer.byteLength(JSON.stringify(complete));
while (low < high) {
const middle = Math.floor((low + high) / 2);
if (detail(middle)?.derived_memory != null) high = middle;
else low = middle + 1;
}
const atBoundary = detail(low);
const belowBoundary = detail(low - 1);
expect(atBoundary?.derived_memory).not.toBeNull();
expect(Buffer.byteLength(JSON.stringify(atBoundary))).toBeLessThanOrEqual(
low,
);
expect(belowBoundary?.derived_memory ?? null).toBeNull();
expect(belowBoundary?.diagnostics.join(" ")).toContain(
"omitted because the final serialized report exceeded its byte budget",
);
expect(
belowBoundary === undefined
? 0
: Buffer.byteLength(JSON.stringify(belowBoundary)),
).toBeLessThanOrEqual(low - 1);
});
it("truncates report rows before exceeding its JSON byte budget", () => {
const bytes = nativeAotPeFixture();
const sha = nativeAotPeDigest(bytes);
const complete = parseNativeAotPe(bytes, sha, REPORT_BUDGET);
const fullBytes = Buffer.byteLength(JSON.stringify(complete.summary));
let budget = fullBytes - 1;
let summary: ReturnType<typeof parseNativeAotPe>["summary"] | undefined;
while (budget > 0) {
try {
const candidate = parseNativeAotPe(bytes, sha, budget);
if (candidate.summary.truncated) {
summary = candidate.summary;
break;
}
} catch {
// Budgets below the fixed report header cannot carry an honest summary.
}
budget--;
}
if (summary === undefined)
throw new Error("No partial summary budget found");
expect(summary.truncated).toBe(true);
expect(
(summary.types_omitted ?? 0) +
(summary.frozen_strings_omitted ?? 0) +
(summary.coverage.scan_bytes_omitted ?? 0) +
(summary.coverage.frozen_scan_bytes_omitted ?? 0),
).toBeGreaterThan(0);
expect(Buffer.byteLength(JSON.stringify(summary))).toBeLessThanOrEqual(
budget,
);
});
});
+161
View File
@@ -0,0 +1,161 @@
import { describe, expect, it } from "vitest";
import { nativeMetadataRecoverySummarySchema } from "./nativeMetadataRecovery.js";
import {
NATIVE_AOT_MAX_REPORT_BYTES,
parseNativeAotPe,
} from "./nativeAotPe.js";
import {
nativeAotPeDigest,
nativeAotPeFixture,
} from "../../../tests/fixtures/nativeaotPe.js";
const SUMMARY_BUDGET = NATIVE_AOT_MAX_REPORT_BYTES;
describe("read-only NativeAOT PE parser", () => {
it("recovers a bounded RTR9.1 type graph and frozen literal from captured bytes", () => {
const bytes = nativeAotPeFixture();
const before = Buffer.from(bytes);
const parsed = parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
SUMMARY_BUDGET,
"0x140000000",
);
expect(bytes).toEqual(before);
expect(parsed.summary).toMatchObject({
format: "dotnet-nativeaot",
status: "complete",
analysis_mode: "read-only-derived-overlay",
format_major: 9,
format_minor: 1,
method_tables: 3,
frozen_strings: [
{ address: "0x140002300", value: "REA_NATIVEAOT_FROZEN" },
],
derived_overlay: { address: "0x140002380", size_bytes: 1 },
coverage: {
basis: "raw-image-and-rehydrated-pointer-scan",
method_tables_recovered: 3,
frozen_string_candidates: 1,
},
});
expect(parsed.readTypeDetail("0x140002280", SUMMARY_BUDGET)).toMatchObject({
source: "read-only-derived-overlay",
method_table_address: "0x140002280",
related_type: { address: "0x140002200", type: null },
virtual_slots: [{ slot: 0, target_address: "0x140001040" }],
});
expect(nativeMetadataRecoverySummarySchema.parse(parsed.summary)).toEqual(
parsed.summary,
);
});
it("fails closed on snapshot identity mismatch", () => {
const bytes = nativeAotPeFixture();
expect(() =>
parseNativeAotPe(bytes, "0".repeat(64), SUMMARY_BUDGET),
).toThrow(/digest/iu);
});
it("reports a detected but unsupported RTR version as partial without guessed types", () => {
const bytes = nativeAotPeFixture(10, 0);
const parsed = parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
SUMMARY_BUDGET,
);
expect(parsed.summary).toMatchObject({
status: "partial",
format_major: 10,
format_minor: 0,
method_tables: 0,
types: [],
});
});
it("does not discover an RTR directory in raw padding beyond VirtualSize", () => {
const bytes = nativeAotPeFixture();
const sectionTable = 0x98 + 0xf0;
const dataSection = sectionTable + 40;
bytes.writeUInt32LE(0x20, dataSection + 8);
const parsed = parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
SUMMARY_BUDGET,
);
expect(parsed.summary).toMatchObject({
status: "not_applicable",
reason:
"No supported ReadyToRun directory signature was found in initialized non-executable PE sections.",
method_tables: 0,
});
});
it("uses raw size as the virtual extent when VirtualSize is zero", () => {
const bytes = nativeAotPeFixture();
const sectionTable = 0x98 + 0xf0;
bytes.writeUInt32LE(0, sectionTable + 40 + 8);
const parsed = parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
SUMMARY_BUDGET,
);
expect(parsed.summary).toMatchObject({
status: "complete",
method_tables: 3,
derived_overlay: { address: "0x140002380", size_bytes: 1 },
});
});
it("accepts executable-section targets in the zero-filled virtual tail", () => {
const bytes = nativeAotPeFixture();
const sectionTable = 0x98 + 0xf0;
bytes.writeUInt32LE(0x300, sectionTable + 8);
const slotOffset = 0x600 + (0x2280 - 0x2000) + 24;
// RVA 0x1200 is executable-section zero fill: virtual size 0x300, raw size 0x200.
bytes.writeBigUInt64LE(0x140001200n, slotOffset);
const parsed = parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
SUMMARY_BUDGET,
"0x140000000",
);
expect(parsed.summary).toMatchObject({
status: "complete",
method_tables: 3,
});
expect(parsed.readTypeDetail("0x140002280", SUMMARY_BUDGET)).toMatchObject({
virtual_slots: [{ slot: 0, target_address: "0x140001200" }],
});
});
it("does not claim an image mapping when Ghidra rebased the image", () => {
const bytes = nativeAotPeFixture();
expect(
parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
SUMMARY_BUDGET,
"0x180000000",
).summary.reason,
).toMatch(/preferred image base/iu);
});
it("reports an unsupported hydration opcode as partial without mutating the snapshot", () => {
const bytes = nativeAotPeFixture();
bytes.writeUInt8(6, 0x704);
const before = Buffer.from(bytes);
const parsed = parseNativeAotPe(
bytes,
nativeAotPeDigest(bytes),
SUMMARY_BUDGET,
);
expect(bytes).toEqual(before);
expect(parsed.summary).toMatchObject({
status: "partial",
derived_overlay: null,
diagnostics: [
expect.stringContaining("Unsupported DEHYDRATED_DATA command"),
],
});
});
});
File diff suppressed because it is too large Load Diff
+85 -11
View File
@@ -6,6 +6,8 @@ const relatedType = z.strictObject({ address, type: z.string().nullable() });
/** Runtime-format metadata recovered in an analysis image, with explicit inference boundaries. */
export const nativeMetadataRecoverySchema = z.strictObject({
format: z.literal("dotnet-nativeaot"),
/** Identifies parser-only facts without claiming Ghidra defined a DataType. */
source: z.enum(["analysis-database", "read-only-derived-overlay"]).optional(),
status: z.enum(["complete", "partial"]),
header_address: address,
format_major: z.number().int().nonnegative(),
@@ -25,19 +27,29 @@ export const nativeMetadataRecoverySchema = z.strictObject({
basis: z.literal("method-table-pointer"),
}),
),
derived_memory: z.strictObject({
address,
size_bytes: z.number().int().nonnegative(),
sha256: z.string().regex(/^[a-f0-9]{64}$/u),
file_offset: z.null(),
}),
derived_memory: z
.strictObject({
address,
size_bytes: z.number().int().nonnegative(),
sha256: z.string().regex(/^[a-f0-9]{64}$/u),
file_offset: z.null(),
})
.nullable(),
truncated: z.boolean().optional(),
virtual_slots_omitted: z.number().int().nonnegative().optional(),
interfaces_omitted: z.number().int().nonnegative().optional(),
output_budget_bytes: z.number().int().nonnegative().optional(),
diagnostics: z.array(z.string()),
limitations: z.array(z.string()),
});
export type NativeMetadataRecovery = z.infer<
typeof nativeMetadataRecoverySchema
>;
/** Discover a recovered runtime format and reusable type identities from the loaded image. */
export const nativeMetadataRecoverySummarySchema = z.strictObject({
format: z.literal("dotnet-nativeaot"),
analysis_mode: z.literal("read-only-derived-overlay").optional(),
status: z.enum(["complete", "partial", "not_applicable"]),
reason: z.string().nullable(),
analysis_artifact_sha256: z.string().regex(/^[a-f0-9]{64}$/u),
@@ -47,14 +59,76 @@ export const nativeMetadataRecoverySummarySchema = z.strictObject({
format_minor: z.number().int().nonnegative().nullable(),
method_tables: z.number().int().nonnegative(),
types: z.array(relatedType),
derived_memory: nativeMetadataRecoverySchema.shape.derived_memory.nullable(),
frozen_strings: z
.array(z.strictObject({ address, value: z.string() }))
.optional(),
truncated: z.boolean().optional(),
types_omitted: z.number().int().nonnegative().optional(),
frozen_strings_omitted: z.number().int().nonnegative().optional(),
output_budget_bytes: z.number().int().nonnegative().optional(),
derived_overlay: z
.strictObject({
address,
size_bytes: z.number().int().nonnegative(),
sha256: z.string().regex(/^[a-f0-9]{64}$/u),
})
.nullable()
.optional(),
derived_memory: nativeMetadataRecoverySchema.shape.derived_memory,
coverage: z
.strictObject({
frozen_object_candidates: z.number().int().nonnegative(),
frozen_objects_annotated: z.number().int().nonnegative(),
basis: z.literal(
frozen_object_candidates: z.number().int().nonnegative().optional(),
frozen_objects_annotated: z.number().int().nonnegative().optional(),
pointer_candidates_examined: z.number().int().nonnegative().optional(),
pointer_slots_examined: z.number().int().nonnegative().optional(),
scan_bytes_examined: z.number().int().nonnegative().optional(),
scan_bytes_omitted: z.number().int().nonnegative().optional(),
rtr_signature_scan_bytes_examined: z
.number()
.int()
.nonnegative()
.optional(),
rtr_signature_scan_bytes_omitted: z
.number()
.int()
.nonnegative()
.optional(),
method_table_validation_items_examined: z
.number()
.int()
.nonnegative()
.optional(),
work_units_examined: z.number().int().nonnegative().optional(),
method_tables_recovered: z.number().int().nonnegative().optional(),
frozen_string_candidates: z.number().int().nonnegative().optional(),
frozen_scan_bytes_examined: z.number().int().nonnegative().optional(),
frozen_scan_bytes_omitted: z.number().int().nonnegative().optional(),
frozen_string_data_bytes_examined: z
.number()
.int()
.nonnegative()
.optional(),
source_bytes_examined: z.number().int().nonnegative().optional(),
source_bytes_omitted: z.number().int().nonnegative().optional(),
source_byte_budget_bytes: z.number().int().nonnegative().optional(),
work_budget_units: z.number().int().nonnegative().optional(),
working_memory_budget_bytes: z.number().int().nonnegative().optional(),
working_memory_bytes_peak: z.number().int().nonnegative().optional(),
truncation_reason: z
.enum([
"source-byte-budget",
"work-unit-budget",
"working-memory-budget",
"response-budget",
])
.optional(),
basis: z.enum([
"rehydrated-pointer-candidates-and-committed-instance-types",
),
"raw-image-pointer-scan",
"raw-image-and-rehydrated-pointer-scan",
"unsupported-layout",
"no-rtr-header",
]),
})
.nullable(),
diagnostics: z.array(z.string()),
+53
View File
@@ -603,3 +603,56 @@ it("retains direct and literal computed call names", () => {
expect.objectContaining({ name: "literal-tool" }),
]);
});
it("recognizes tRPC procedures and named GraphQL operations in captured scripts", () => {
const result = analyzeCapturedWebBundle(
inspection(`
const n = createTRPCClient({ links: [httpBatchLink({ url: "/api/trpc" })] });
e.invoice.list.useQuery({ competencia: "2026-09" });
n.empresa.byCnpj.query("00000000000191");
const q = "query ListarNotas($competencia: String!) { notas { id } }";
this.client.query({ query: q });
`),
);
expect(
result.observations.endpoints.map(({ value, mechanism }) => ({
value,
mechanism,
})),
).toEqual([
{ value: "/api/trpc", mechanism: "trpc:link:httpBatchLink" },
{ value: "invoice.list", mechanism: "trpc:procedure:useQuery" },
{ value: "empresa.byCnpj", mechanism: "trpc:procedure:query" },
{ value: "ListarNotas", mechanism: "graphql:query" },
]);
});
it("keeps RPC lexical identity and bundled factory recognition in captured scripts", () => {
const longOperation = "Operation" + "x".repeat(512);
const document =
"#" +
"comment".repeat(700) +
"\nquery " +
longOperation +
" { users { id } }";
const result = analyzeCapturedWebBundle(
inspection(`
function first() { const c = createTRPCClient({}); }
function unrelated(c) { c.users.query("select 1"); }
const c = (0, t.createTRPCClient)({ links: [(0, t.httpBatchLink)({ url: "/api/rpc" })] });
c.posts.query();
const doc = ${JSON.stringify(document)};
`),
);
expect(
result.observations.endpoints.map(({ value, mechanism }) => ({
value,
mechanism,
})),
).toEqual([
{ value: "/api/rpc", mechanism: "trpc:link:httpBatchLink" },
{ value: "posts", mechanism: "trpc:procedure:query" },
{ value: longOperation, mechanism: "graphql:query" },
]);
expect(result.completeness.status).toBe("complete");
});
+23 -2
View File
@@ -5,8 +5,14 @@ import { sanitizeEndpointCandidate } from "./browserObservation.js";
import { compositeKey } from "./unicodeCodePointOrder.js";
import type { WebPageInspection } from "./browserObservationSchemas.js";
import type { WebBundleAnalysis } from "./webBundleAnalysis.js";
import { traverseJavaScriptAst } from "./javascript/javascriptSemanticTraversal.js";
import { classifyParsedJavaScriptOpenReceivers } from "./javascript/javascriptSemanticAnalysis.js";
import {
completeSemanticSteps,
traverseJavaScriptAst,
} from "./javascript/javascriptSemanticTraversal.js";
import {
classifyParsedJavaScriptOpenReceivers,
classifyParsedJavaScriptRpcClientRootsSteps,
} from "./javascript/javascriptSemanticAnalysis.js";
import type { JavaScriptOpenReceiverFact } from "./javascript/javascriptSemanticAnalysis.js";
import { semanticStaticPropertyName } from "./javascript/javascriptAstValues.js";
import {
@@ -22,6 +28,7 @@ import {
resolveSpecifier,
isUrlLikeModuleSpecifier,
} from "./webBundleAnalyzerAst.js";
import { createJavaScriptRpcScan } from "./javascript/javascriptRpcEndpoints.js";
type BundleObservations = WebBundleAnalysis["observations"];
type ChunkEdge = BundleObservations["chunks"]["edges"][number];
@@ -66,10 +73,24 @@ export const analyzeScript = (
accumulator.parsedScripts += 1;
detectVendorFingerprints(script, accumulator);
const openReceiverFacts = classifyParsedJavaScriptOpenReceivers(file);
const rpc = createJavaScriptRpcScan(
script.source.artifact.text,
completeSemanticSteps(classifyParsedJavaScriptRpcClientRootsSteps(file)),
);
traverseJavaScriptAst(file, {
enter: (node) => {
accumulator.visitedNodes += 1;
inspectNode(script, node, accumulator, openReceiverFacts);
const endpoint = rpc.inspect(node);
if (endpoint !== undefined)
addFinding({
collection: accumulator.endpoints,
script,
rawValue: endpoint.value,
mechanism: endpoint.mechanism,
node,
accumulator,
});
},
});
};
+1 -1
View File
@@ -1,7 +1,7 @@
/** Generated from package.json and package-lock.json; do not edit. */
export const PACKAGE_METADATA = {
name: "rea-agents",
version: "6.3.0", // x-release-please-version
version: "6.4.0", // x-release-please-version
serverSdkVersion: "2.3.1",
clientSdkVersion: "2.3.1",
coreSdkVersion: "2.3.1",
+14
View File
@@ -17,6 +17,13 @@ import {
ghidraMipsProfileParameters,
ghidraMipsUnsupportedReason,
} from "./GhidraMipsProfile.js";
import {
NATIVE_AOT_MAX_REPORT_BYTES,
NATIVE_AOT_MAX_SOURCE_BYTES,
NATIVE_AOT_MAX_WORKING_MEMORY_BYTES,
NATIVE_AOT_MAX_WORK_UNITS,
NATIVE_AOT_PROFILE_CONTRACT_REVISION,
} from "../domain/native/nativeAotPe.js";
/** Resolve version-bound, deterministic semantics before Ghidra imports a target. */
export const resolveGhidraAnalysisProfile = (
@@ -69,6 +76,13 @@ export const resolveGhidraAnalysisProfile = (
? {
executable_role: target.executableRole ?? null,
managed: target.managed ?? null,
native_aot_metadata: {
contract_revision: NATIVE_AOT_PROFILE_CONTRACT_REVISION,
source_bytes: NATIVE_AOT_MAX_SOURCE_BYTES,
work_units: NATIVE_AOT_MAX_WORK_UNITS,
working_memory_bytes: NATIVE_AOT_MAX_WORKING_MEMORY_BYTES,
report_bytes: NATIVE_AOT_MAX_REPORT_BYTES,
},
}
: {}),
architecture: target.architecture ?? null,
+95 -3
View File
@@ -43,6 +43,21 @@ import {
} from "./GhidraSessionValues.js";
import { createGhidraTargetSnapshot } from "./GhidraTargetSnapshot.js";
import { readFile } from "node:fs/promises";
import { readBoundedFileBytes } from "../process/BoundedFileBytes.js";
import {
openRegularFile,
RegularFileChangedError,
sameRegularFileState,
} from "../filesystem/RegularFile.js";
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
export type GhidraTargetSnapshot =
| { readonly kind: "captured"; readonly bytes: Buffer }
| {
readonly kind: "over-capacity";
readonly sourceBytesAtLeast: number;
readonly maximumBytes: number;
};
import { connectGhidraSocket } from "./GhidraSocketConnection.js";
import {
createGhidraEndpoint,
@@ -79,6 +94,7 @@ export class GhidraClient {
#socketRoot: PrivateRuntimeRoot | undefined;
#endpointPath: string | undefined;
#snapshotPath: string | undefined;
readonly #snapshotOwners = new Set<OwnedFileHandle>();
#targetAdmission: JsonValue | undefined;
#token: string | undefined;
// Retain authentication identities for diagnostics from late request settlement.
@@ -227,7 +243,10 @@ export class GhidraClient {
}
/** Read the private immutable source for independent import verification. */
async readTargetSnapshot(): Promise<Result<Buffer, GhidraSessionError>> {
async readTargetSnapshot(
maximumBytes?: number,
signal?: AbortSignal,
): Promise<Result<GhidraTargetSnapshot, GhidraSessionError>> {
if (this.#snapshotPath === undefined)
return err(
this.#failure(
@@ -236,11 +255,80 @@ export class GhidraClient {
),
);
try {
return ok(await readFile(this.#snapshotPath));
if (maximumBytes !== undefined) {
signal?.throwIfAborted();
if (!Number.isSafeInteger(maximumBytes) || maximumBytes < 0)
throw new RangeError("Snapshot byte limit must be a safe integer");
const handle = await openRegularFile(this.#snapshotPath, {
symlinks: "reject",
});
const owner = new OwnedFileHandle(handle);
this.#snapshotOwners.add(owner);
let result: GhidraTargetSnapshot;
let readFailure: unknown;
try {
const metadata = await handle.stat();
signal?.throwIfAborted();
if (metadata.size > maximumBytes)
result = {
kind: "over-capacity",
sourceBytesAtLeast: metadata.size,
maximumBytes,
};
else {
const bytes = await readBoundedFileBytes(
handle,
maximumBytes,
signal,
metadata.size,
);
signal?.throwIfAborted();
const after = await handle.stat();
if (
bytes !== undefined &&
bytes.length === metadata.size &&
sameRegularFileState(metadata, after)
) {
result = { kind: "captured", bytes };
} else if (
bytes === undefined &&
after.isFile() &&
after.size > maximumBytes
) {
result = {
kind: "over-capacity",
sourceBytesAtLeast: after.size,
maximumBytes,
};
} else {
throw new RegularFileChangedError(this.#snapshotPath);
}
}
} catch (cause: unknown) {
readFailure = cause;
}
try {
await owner.close();
this.#snapshotOwners.delete(owner);
} catch (closeFailure: unknown) {
throw new AggregateError(
readFailure === undefined
? [closeFailure]
: [readFailure, closeFailure],
"Ghidra target snapshot read or descriptor cleanup failed",
);
}
if (readFailure !== undefined) throw readFailure;
return ok(result!);
}
return ok({
kind: "captured",
bytes: await readFile(this.#snapshotPath),
});
} catch (cause: unknown) {
return err(
this.#failure(
"protocol",
signal?.aborted === true ? "cancelled" : "protocol",
"Ghidra target snapshot could not be read for load-image verification",
cause,
),
@@ -602,6 +690,10 @@ export class GhidraClient {
this.#lastDiagnostics = this.#diagnostics();
this.#process = undefined;
this.#launch = undefined;
for (const owner of this.#snapshotOwners) {
await owner.close();
this.#snapshotOwners.delete(owner);
}
try {
await this.#removeRuntimeRoots();
} catch (cause: unknown) {
+87 -9
View File
@@ -1,4 +1,5 @@
import type { AnalysisOperation } from "../application/AnalysisProvider.js";
import { AnalysisCancelledError } from "../domain/analysisErrorCore.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
@@ -12,23 +13,91 @@ import {
attestGhidraDosComLoadImage,
attestGhidraDosLoadImage,
} from "./GhidraLoadImageValues.js";
import { type NativeAotPeResult } from "../domain/native/nativeAotPe.js";
import type { GhidraTargetSnapshot } from "./GhidraClient.js";
import type { GhidraSessionError } from "./GhidraSessionError.js";
type LoadImageSnapshotClient = {
readonly readTargetSnapshot?: () => Promise<
Result<Buffer, GhidraSessionError>
>;
readonly readTargetSnapshot?: (
maximumBytes?: number,
signal?: AbortSignal,
) => Promise<Result<GhidraTargetSnapshot, GhidraSessionError>>;
};
type NativeLoadImageAttestationInput = {
readonly target: BinaryTarget;
readonly operation: AnalysisOperation;
readonly measured: JsonValue;
readonly client: LoadImageSnapshotClient;
readonly mapSessionError: (failure: GhidraSessionError) => AnalysisError;
readonly nativeAotResult?: NativeAotPeResult | undefined;
readonly signal?: AbortSignal | undefined;
};
const isSignalAborted = (signal: AbortSignal | undefined): boolean =>
signal?.aborted === true;
/** Independently verify measured load-image observations for admitted DOS targets. */
export const attestGhidraNativeLoadImage = async (
target: BinaryTarget,
operation: AnalysisOperation,
measured: JsonValue,
client: LoadImageSnapshotClient,
mapSessionError: (failure: GhidraSessionError) => AnalysisError,
input: NativeLoadImageAttestationInput,
): Promise<Result<JsonValue, AnalysisError>> => {
const {
target,
operation,
measured,
client,
mapSessionError,
nativeAotResult,
signal,
} = input;
if (isSignalAborted(signal))
return err(new AnalysisCancelledError(operation));
const observations = nativeLoadImageObservationSchema.parse(measured);
if (target.format === "pe") {
if (nativeAotResult === undefined)
return err(
new ProviderAdapterError("ghidra", operation, {
diagnostics: {
reason:
"The Ghidra provider did not produce read-only NativeAOT metadata for this admitted PE snapshot.",
},
}),
);
if (isSignalAborted(signal))
return err(new AnalysisCancelledError(operation));
try {
return ok(
jsonValueSchema.parse(
nativeLoadImageSchema.parse({
status: "unsupported",
reason:
"Independent load-image verification currently supports DOS MZ and explicit DOS COM targets only.",
observations: {
...observations,
metadata_recovery: [
...(observations.metadata_recovery ?? []),
nativeAotResult.summary,
],
},
limitations: [
"Load-image verification is not performed for PE targets; NativeAOT metadata observations are derived separately from the exact admitted snapshot.",
],
}),
),
);
} catch (cause: unknown) {
return err(
new ProviderAdapterError("ghidra", operation, {
diagnostics: {
reason:
cause instanceof Error
? cause.message
: "NativeAOT metadata could not be represented in the load-image result",
},
}),
);
}
}
if (target.format !== "dos-mz" && target.format !== "dos-com") {
return ok(
jsonValueSchema.parse(
@@ -55,11 +124,20 @@ export const attestGhidraNativeLoadImage = async (
);
const snapshot = await client.readTargetSnapshot();
if (!snapshot.ok) return err(mapSessionError(snapshot.error));
if (snapshot.value.kind !== "captured")
return err(
new ProviderAdapterError("ghidra", operation, {
diagnostics: {
reason:
"The uncapped DOS load-image snapshot was unexpectedly not captured.",
},
}),
);
const attested = (
target.format === "dos-com"
? attestGhidraDosComLoadImage
: attestGhidraDosLoadImage
)(snapshot.value, target.sha256 ?? "", observations);
)(snapshot.value.bytes, target.sha256 ?? "", observations);
if (!attested.ok)
return err(
new ProviderAdapterError("ghidra", operation, {
+522 -1
View File
@@ -10,16 +10,22 @@ import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { describe, expect, it } from "vitest";
import { fixtureDosLoadImage } from "./GhidraLoadImage.fixture.js";
import {
nativeAotPeDigest,
nativeAotPeFixture,
} from "../../tests/fixtures/nativeaotPe.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
import { parseConfig } from "../config/parseConfig.js";
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
import { parseExecutableHeader } from "../domain/binaryTarget.js";
import { GhidraProvider } from "./GhidraProvider.js";
import type { GhidraTargetSnapshot } from "./GhidraClient.js";
import type { GhidraProviderClientFactory } from "./GhidraProviderClient.js";
import type { GhidraInstallationHost } from "./GhidraInstallation.js";
import { GHIDRA_SESSION_CAPABILITIES } from "./GhidraSessionValues.js";
import { err, ok } from "../domain/result.js";
import type { Result } from "../domain/result.js";
import { GhidraSessionError } from "./GhidraSessionError.js";
import { silentLogger } from "../logger.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
@@ -219,10 +225,25 @@ describe("Ghidra platform support", () => {
expect(profile.parameters).toMatchObject({
executable_role: target.executableRole,
managed: target.managed,
native_aot_metadata: {
contract_revision: "rtr-9.1-x64-pe-read-only-v3",
source_bytes: 128 * 1024 * 1024,
work_units: 64 * 1024 * 1024,
working_memory_bytes: 64 * 1024 * 1024,
report_bytes: 4 * 1024 * 1024,
},
});
digests.add(profile.digest);
}
expect(digests.size).toBe(3);
const nonPe = await ghidra.resolveAnalysisProfile(
executableTarget("elf", "x86_64"),
);
if (!nonPe.ok || nonPe.value.profile === null)
throw new Error("Expected a non-PE analysis profile");
expect(nonPe.value.profile.parameters).not.toHaveProperty(
"native_aot_metadata",
);
});
it("keeps Windows annotation mutation unavailable independently of native controls", () => {
@@ -717,7 +738,8 @@ describe("Ghidra measured load-image projection", () => {
...(state === "missing-snapshot"
? {}
: {
readTargetSnapshot: () => Promise.resolve(ok(fixture.bytes)),
readTargetSnapshot: () =>
Promise.resolve(ok({ kind: "captured", bytes: fixture.bytes })),
}),
});
const ghidra = provider(installationHost(), factory);
@@ -757,6 +779,505 @@ describe("Ghidra measured load-image projection", () => {
);
});
it("returns read-only NativeAOT metadata inline and drills down by derived MethodTable address", async () => {
const bytes = nativeAotPeFixture();
const observations = fixtureDosLoadImage().observation;
observations.image_base = "0x140000000";
const factory: GhidraProviderClientFactory = () => ({
start: () =>
Promise.resolve(
ok({
...sessionInfo(),
target: { ...sessionInfo().target, image_base: "0x140000000" },
}),
),
callTool: (operation) =>
Promise.resolve(
ok(
jsonValueSchema.parse(
operation === "inspect_native_load_image"
? observations
: {
status: "unavailable",
reason: "No Ghidra DataType is defined at this address.",
id: null,
name: null,
kind: "unavailable",
source: "analysis-database",
source_archive: null,
address: null,
size_bytes: null,
alignment_bytes: null,
packing_enabled: null,
referenced_type: null,
array_count: null,
array_stride_bytes: null,
fields: [],
members: [],
total_fields: 0,
truncated: false,
limitations: [],
},
),
),
),
readTargetSnapshot: () => Promise.resolve(ok({ kind: "captured", bytes })),
close: () => Promise.resolve(ok(null)),
});
const ghidra = provider(installationHost(), factory);
const target = { ...peTarget("x86_64"), sha256: nativeAotPeDigest(bytes) };
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("Expected admitted PE profile");
const client = ghidra.createClient(target, profile.value.profile);
const image = await client.execute("inspect_native_load_image", {});
expect(image).toMatchObject({
ok: true,
value: {
result: {
status: "unsupported",
observations: {
metadata_recovery: [
{
status: "complete",
analysis_mode: "read-only-derived-overlay",
method_tables: 3,
frozen_strings: [{ value: "REA_NATIVEAOT_FROZEN" }],
},
],
},
},
rawResult: observations,
},
});
const detail = await client.execute("inspect_native_data_type", {
address: "0x140002280",
});
expect(detail).toMatchObject({
ok: true,
value: {
result: {
status: "unavailable",
source: "analysis-database",
metadata_recovery: {
source: "read-only-derived-overlay",
method_table_address: "0x140002280",
related_type: { address: "0x140002200", type: null },
},
},
},
});
await client.close();
});
it("derives PE metadata for a direct type-address query without a load-image call", async () => {
const bytes = nativeAotPeFixture();
const operations: string[] = [];
const factory: GhidraProviderClientFactory = () => ({
start: () =>
Promise.resolve(
ok({
...sessionInfo(),
target: { ...sessionInfo().target, image_base: "0x140000000" },
}),
),
callTool: (operation) => {
operations.push(operation);
return Promise.resolve(
ok(
jsonValueSchema.parse({
status: "unavailable",
reason: "No Ghidra DataType is defined at this address.",
id: null,
name: null,
kind: "unavailable",
source: "analysis-database",
source_archive: null,
address: null,
size_bytes: null,
alignment_bytes: null,
packing_enabled: null,
referenced_type: null,
array_count: null,
array_stride_bytes: null,
fields: [],
members: [],
total_fields: 0,
truncated: false,
limitations: [],
}),
),
);
},
readTargetSnapshot: () => Promise.resolve(ok({ kind: "captured", bytes })),
close: () => Promise.resolve(ok(null)),
});
const ghidra = provider(installationHost(), factory);
const target = { ...peTarget("x86_64"), sha256: nativeAotPeDigest(bytes) };
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("Expected admitted PE profile");
const client = ghidra.createClient(target, profile.value.profile);
const detail = await client.execute("inspect_native_data_type", {
address: "0x140002280",
});
expect(detail).toMatchObject({
ok: true,
value: {
result: {
metadata_recovery: {
source: "read-only-derived-overlay",
method_table_address: "0x140002280",
related_type: { address: "0x140002200", type: null },
},
},
},
});
expect(operations).toEqual(["inspect_native_data_type"]);
await client.close();
});
it("shares one session snapshot across callers while isolating cancellation and rejecting a changed base", async () => {
const bytes = nativeAotPeFixture();
const baseObservation = fixtureDosLoadImage().observation;
baseObservation.image_base = "0x140000000";
let calls = 0;
let snapshotReads = 0;
let snapshotSignal: AbortSignal | undefined;
let releaseSnapshot:
| ((value: Result<GhidraTargetSnapshot, GhidraSessionError>) => void)
| undefined;
let markSnapshotStarted: (() => void) | undefined;
const snapshotStarted = new Promise<void>((resolve) => {
markSnapshotStarted = resolve;
});
const snapshotResult = new Promise<
Result<GhidraTargetSnapshot, GhidraSessionError>
>((resolve) => {
releaseSnapshot = resolve;
});
const factory: GhidraProviderClientFactory = () => ({
start: () => Promise.resolve(ok(sessionInfo())),
callTool: (operation) => {
calls += 1;
const observation = {
...baseObservation,
source_files: [
{ ...baseObservation.source_files[0]!, name: `caller-${calls}` },
],
};
return Promise.resolve(
ok(
jsonValueSchema.parse(
operation === "inspect_native_load_image"
? observation
: {
status: "unavailable",
reason: "No Ghidra DataType is defined at this address.",
id: null,
name: null,
kind: "unavailable",
source: "analysis-database",
source_archive: null,
address: null,
size_bytes: null,
alignment_bytes: null,
packing_enabled: null,
referenced_type: null,
array_count: null,
array_stride_bytes: null,
fields: [],
members: [],
total_fields: 0,
truncated: false,
limitations: [],
},
),
),
);
},
readTargetSnapshot: (_maximumBytes, signal) => {
snapshotReads += 1;
snapshotSignal = signal;
markSnapshotStarted?.();
return snapshotResult;
},
close: () => Promise.resolve(ok(null)),
});
const ghidra = provider(installationHost(), factory);
const target = { ...peTarget("x86_64"), sha256: nativeAotPeDigest(bytes) };
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("Expected admitted PE profile");
const client = ghidra.createClient(target, profile.value.profile);
const firstCaller = new AbortController();
const secondCaller = new AbortController();
const first = client.execute(
"inspect_native_load_image",
{},
{
signal: firstCaller.signal,
},
);
const second = client.execute(
"inspect_native_load_image",
{},
{
signal: secondCaller.signal,
},
);
await snapshotStarted;
expect(snapshotReads).toBe(1);
expect(snapshotSignal).toBeDefined();
expect(snapshotSignal).not.toBe(firstCaller.signal);
expect(snapshotSignal).not.toBe(secondCaller.signal);
firstCaller.abort();
await expect(first).resolves.toMatchObject({
ok: false,
error: { _tag: "AnalysisCancelledError" },
});
releaseSnapshot?.(ok({ kind: "captured", bytes }));
const secondResult = await second;
expect(secondResult).toMatchObject({
ok: true,
value: {
rawResult: {
source_files: [{ name: "caller-2" }],
},
result: {
observations: {
source_files: [{ name: "caller-2" }],
},
},
},
});
expect(snapshotReads).toBe(1);
expect(calls).toBe(2);
baseObservation.image_base = "0x150000000";
const conflictingBase = await client.execute("inspect_native_load_image", {});
expect(conflictingBase).toMatchObject({
ok: false,
error: {
_tag: "ProviderAdapterError",
diagnostics: {
reason: expect.stringContaining("different loaded image base"),
},
},
});
expect(snapshotReads).toBe(1);
await client.close();
expect(snapshotSignal?.aborted).toBe(true);
});
it("retries a failed NativeAOT snapshot read without retaining a failed derivation", async () => {
const bytes = nativeAotPeFixture();
const observations = fixtureDosLoadImage().observation;
observations.image_base = "0x140000000";
let snapshotReads = 0;
const factory: GhidraProviderClientFactory = () => ({
start: () => Promise.resolve(ok(sessionInfo())),
callTool: () => Promise.resolve(ok(jsonValueSchema.parse(observations))),
readTargetSnapshot: () => {
snapshotReads += 1;
return Promise.resolve(
snapshotReads === 1
? err(
new GhidraSessionError(
"process",
"temporary snapshot read failure",
),
)
: ok({ kind: "captured", bytes }),
);
},
close: () => Promise.resolve(ok(null)),
});
const ghidra = provider(installationHost(), factory);
const target = { ...peTarget("x86_64"), sha256: nativeAotPeDigest(bytes) };
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("Expected admitted PE profile");
const client = ghidra.createClient(target, profile.value.profile);
await expect(
client.execute("inspect_native_load_image", {}),
).resolves.toMatchObject({
ok: false,
error: { _tag: "ProviderAdapterError" },
});
await expect(
client.execute("inspect_native_load_image", {}),
).resolves.toMatchObject({
ok: true,
value: {
result: {
observations: {
metadata_recovery: [{ status: "complete" }],
},
},
},
});
expect(snapshotReads).toBe(2);
await client.close();
});
it("rejects all waiters when the loaded image base changes during derivation", async () => {
const bytes = nativeAotPeFixture();
const observation = fixtureDosLoadImage().observation;
let calls = 0;
let resolveSnapshot:
| ((value: Result<GhidraTargetSnapshot, GhidraSessionError>) => void)
| undefined;
let markSnapshotStarted: (() => void) | undefined;
const snapshotStarted = new Promise<void>((resolve) => {
markSnapshotStarted = resolve;
});
const snapshot = new Promise<
Result<GhidraTargetSnapshot, GhidraSessionError>
>((resolve) => {
resolveSnapshot = resolve;
});
const factory: GhidraProviderClientFactory = () => ({
start: () => Promise.resolve(ok(sessionInfo())),
callTool: () => {
calls += 1;
return Promise.resolve(
ok(
jsonValueSchema.parse({
...observation,
image_base: calls === 1 ? "0x140000000" : "0x150000000",
}),
),
);
},
readTargetSnapshot: () => {
markSnapshotStarted?.();
return snapshot;
},
close: () => Promise.resolve(ok(null)),
});
const ghidra = provider(installationHost(), factory);
const target = { ...peTarget("x86_64"), sha256: nativeAotPeDigest(bytes) };
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("Expected admitted PE profile");
const client = ghidra.createClient(target, profile.value.profile);
const originalBase = client.execute("inspect_native_load_image", {});
await snapshotStarted;
const changedBase = await client.execute("inspect_native_load_image", {});
expect(changedBase).toMatchObject({
ok: false,
error: {
_tag: "ProviderAdapterError",
diagnostics: {
observed_image_base: "0x150000000",
measured_observation: { image_base: "0x150000000" },
},
},
});
resolveSnapshot?.(ok({ kind: "captured", bytes }));
await expect(originalBase).resolves.toMatchObject({
ok: false,
error: {
_tag: "ProviderAdapterError",
diagnostics: {
reason: expect.stringContaining("conflicting loaded image bases"),
measured_observation: { image_base: "0x140000000" },
conflicting_observation: { image_base: "0x150000000" },
},
},
});
expect(calls).toBe(2);
await client.close();
});
it("cancels an in-flight NativeAOT waiter when its owning session closes", async () => {
const bytes = nativeAotPeFixture();
const observations = fixtureDosLoadImage().observation;
observations.image_base = "0x140000000";
let resolveSnapshot:
| ((value: Result<GhidraTargetSnapshot, GhidraSessionError>) => void)
| undefined;
let snapshotSignal: AbortSignal | undefined;
let markSnapshotStarted: (() => void) | undefined;
const snapshotStarted = new Promise<void>((resolve) => {
markSnapshotStarted = resolve;
});
const snapshot = new Promise<
Result<GhidraTargetSnapshot, GhidraSessionError>
>((resolve) => {
resolveSnapshot = resolve;
});
const factory: GhidraProviderClientFactory = () => ({
start: () => Promise.resolve(ok(sessionInfo())),
callTool: () => Promise.resolve(ok(jsonValueSchema.parse(observations))),
readTargetSnapshot: (_maximumBytes, signal) => {
snapshotSignal = signal;
markSnapshotStarted?.();
return snapshot;
},
close: () => Promise.resolve(ok(null)),
});
const ghidra = provider(installationHost(), factory);
const target = { ...peTarget("x86_64"), sha256: nativeAotPeDigest(bytes) };
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("Expected admitted PE profile");
const client = ghidra.createClient(target, profile.value.profile);
const pending = client.execute("inspect_native_load_image", {});
await snapshotStarted;
await client.close();
expect(snapshotSignal?.aborted).toBe(true);
resolveSnapshot?.(ok({ kind: "captured", bytes }));
await expect(pending).resolves.toMatchObject({
ok: false,
error: { _tag: "AnalysisCancelledError" },
});
});
describe("Ghidra NativeAOT snapshot budget", () => {
it("reports snapshot-capacity omissions inline through the load-image result", async () => {
const observations = fixtureDosLoadImage().observation;
const target = peTarget("x86_64");
const factory: GhidraProviderClientFactory = () => ({
start: () => Promise.resolve(ok(sessionInfo())),
callTool: () => Promise.resolve(ok(jsonValueSchema.parse(observations))),
readTargetSnapshot: () =>
Promise.resolve(
ok({
kind: "over-capacity",
sourceBytesAtLeast: 128 * 1024 * 1024 + 1,
maximumBytes: 128 * 1024 * 1024,
}),
),
close: () => Promise.resolve(ok(null)),
});
const ghidra = provider(installationHost(), factory);
const profile = await ghidra.resolveAnalysisProfile(target);
if (!profile.ok || profile.value.profile === null)
throw new Error("Expected admitted PE profile");
const client = ghidra.createClient(target, profile.value.profile);
const image = await client.execute("inspect_native_load_image", {});
expect(image).toMatchObject({
ok: true,
value: {
result: {
observations: {
metadata_recovery: [
{
status: "partial",
truncated: true,
coverage: { truncation_reason: "source-byte-budget" },
},
],
},
},
},
});
await client.close();
});
});
const sessionInfo = () => ({
name: "REA Ghidra bridge" as const,
run_id: "11111111-1111-4111-8111-111111111111",
+337 -6
View File
@@ -8,7 +8,6 @@ import {
type AnalysisOperation,
} from "../application/AnalysisProvider.js";
import type { AppConfig } from "../config/types.js";
import type { JsonValue } from "../domain/jsonValue.js";
import {
ghidraSeedCommitmentSchema,
ghidraSeedFailure,
@@ -30,6 +29,7 @@ import {
AnalysisTimeoutError,
} from "../domain/analysisErrorCore.js";
import { type AnalysisError } from "../domain/analysisErrorBase.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { err, ok, type Result } from "../domain/result.js";
import type { Logger } from "pino";
@@ -52,6 +52,18 @@ import {
import { unverifiedGhidraBuildLimitation } from "./GhidraInstallationPolicy.js";
import { GhidraHeadlessLauncher } from "./GhidraLauncher.js";
import { attestGhidraNativeLoadImage } from "./GhidraLoadImageAttest.js";
import {
NATIVE_AOT_MAX_REPORT_BYTES,
NATIVE_AOT_MAX_SOURCE_BYTES,
nativeAotSnapshotOverCapacity,
parseNativeAotPe,
type NativeAotPeResult,
} from "../domain/native/nativeAotPe.js";
import {
ABORTED,
waitForAbortable,
} from "../application/binary/AbortablePromise.js";
import { nativeLoadImageObservationSchema } from "../domain/native/nativeLoadImage.js";
import {
GHIDRA_PROVIDER_IDENTITY,
healthLimitations,
@@ -77,6 +89,250 @@ export type GhidraProviderClientFactory = (
) => Pick<GhidraClient, "start" | "callTool" | "close"> &
Partial<Pick<GhidraClient, "runtimeLineage" | "readTargetSnapshot">>;
type GhidraProviderClientInstance = ReturnType<GhidraProviderClientFactory>;
type NativeAotSnapshotOwner = {
readonly derive: (
imageBase: string,
operation: AnalysisOperation,
callerSignal: AbortSignal | undefined,
observations: JsonValue,
) => Promise<Result<NativeAotPeResult, AnalysisError>>;
readonly readTypeDetail: (
address: string,
) => ReturnType<NativeAotPeResult["readTypeDetail"]>;
readonly close: () => void;
};
/** Own one session's captured PE snapshot and immutable NativeAOT derivation. */
const createNativeAotSnapshotOwner = (input: {
readonly client: GhidraProviderClientInstance;
readonly target: BinaryTarget;
readonly startupTimeoutMs: number;
}): NativeAotSnapshotOwner => {
const { client, target, startupTimeoutMs } = input;
const controller = new AbortController();
let identity: string | undefined;
let result: NativeAotPeResult | undefined;
let pending:
| {
readonly identity: string;
readonly promise: Promise<Result<NativeAotPeResult, AnalysisError>>;
}
| undefined;
let closed = false;
let identityConflict = false;
let identityConflictObservation:
| { readonly imageBase: string; readonly observations: JsonValue }
| undefined;
const identityError = (
operation: AnalysisOperation,
reason: string,
observedBase: string,
observations: JsonValue,
): AnalysisError =>
new ProviderAdapterError("ghidra", operation, {
diagnostics: {
reason,
artifact_sha256: target.sha256,
bound_image_base:
identity === undefined
? null
: `0x${identity.slice(target.sha256.length + 1)}`,
observed_image_base: observedBase,
measured_observation: observations,
conflicting_image_base: identityConflictObservation?.imageBase ?? null,
conflicting_observation:
identityConflictObservation?.observations ?? null,
},
});
const identityConflictFailure = (
operation: AnalysisOperation,
observedBase: string,
observations: JsonValue,
): Result<NativeAotPeResult, AnalysisError> => {
return err(
identityError(
operation,
"The Ghidra session reported conflicting loaded image bases; reopen the analysis session to establish a new identity.",
observedBase,
observations,
),
);
};
const derive = async (
imageBase: string,
operation: AnalysisOperation,
callerSignal: AbortSignal | undefined,
observations: JsonValue,
): Promise<Result<NativeAotPeResult, AnalysisError>> => {
if (callerSignal?.aborted || closed)
return err(new AnalysisCancelledError(operation));
if (identityConflict)
return identityConflictFailure(operation, imageBase, observations);
if (controller.signal.aborted)
return err(new AnalysisCancelledError(operation));
let normalizedBase: string;
try {
normalizedBase = BigInt(imageBase).toString(16);
} catch (cause: unknown) {
return err(
new ProviderAdapterError("ghidra", operation, {
diagnostics: {
reason:
"Ghidra returned an invalid loaded image base for NativeAOT identity binding.",
artifact_sha256: target.sha256,
observed_image_base: imageBase,
measured_observation: observations,
},
cause,
}),
);
}
const key = `${target.sha256}:${normalizedBase}`;
if (identity !== undefined && identity !== key) {
identityConflict = true;
identityConflictObservation = { imageBase, observations };
result = undefined;
controller.abort();
return err(
identityError(
operation,
"The Ghidra session reported a different loaded image base after NativeAOT metadata was bound to its captured snapshot; reopen the analysis session to establish a new identity.",
imageBase,
observations,
),
);
}
identity ??= key;
if (result !== undefined) {
const settled = await waitForAbortable(
Promise.resolve(ok(result)),
callerSignal,
);
if (closed) return err(new AnalysisCancelledError(operation));
if (identityConflict)
return identityConflictFailure(operation, imageBase, observations);
if (controller.signal.aborted)
return err(new AnalysisCancelledError(operation));
return settled === ABORTED
? err(new AnalysisCancelledError(operation))
: settled;
}
if (pending !== undefined && pending.identity !== key)
return err(
identityError(
operation,
"A NativeAOT snapshot derivation is already bound to a different loaded image base.",
imageBase,
observations,
),
);
if (pending === undefined) {
const promise: Promise<Result<NativeAotPeResult, AnalysisError>> =
(async () => {
try {
if (client.readTargetSnapshot === undefined)
return err(
new ProviderAdapterError("ghidra", operation, {
diagnostics: {
reason:
"The Ghidra client does not expose its immutable target snapshot for NativeAOT metadata inspection.",
},
}),
);
const snapshot = await client.readTargetSnapshot(
NATIVE_AOT_MAX_SOURCE_BYTES,
controller.signal,
);
if (!snapshot.ok)
return err(
projectSessionError(
operation,
snapshot.error,
startupTimeoutMs,
),
);
if (closed) return err(new AnalysisCancelledError(operation));
if (identityConflict)
return identityConflictFailure(
operation,
imageBase,
observations,
);
if (controller.signal.aborted)
return err(new AnalysisCancelledError(operation));
const parsed =
snapshot.value.kind === "over-capacity"
? {
summary: nativeAotSnapshotOverCapacity(
target.sha256,
snapshot.value.sourceBytesAtLeast,
NATIVE_AOT_MAX_REPORT_BYTES,
),
readTypeDetail: () => undefined,
}
: parseNativeAotPe(
snapshot.value.bytes,
target.sha256,
NATIVE_AOT_MAX_REPORT_BYTES,
imageBase,
);
if (closed) return err(new AnalysisCancelledError(operation));
if (identityConflict)
return identityConflictFailure(
operation,
imageBase,
observations,
);
if (controller.signal.aborted)
return err(new AnalysisCancelledError(operation));
return ok(parsed);
} catch (cause: unknown) {
return err(
new ProviderAdapterError("ghidra", operation, {
diagnostics: {
reason:
cause instanceof Error
? cause.message
: "NativeAOT metadata could not be derived from the captured target snapshot.",
},
cause,
}),
);
}
})();
const current = { identity: key, promise };
pending = current;
void promise.then((settled) => {
if (pending !== current) return;
pending = undefined;
if (settled.ok && !closed && !identityConflict) result = settled.value;
});
}
const settled = await waitForAbortable(pending.promise, callerSignal);
if (closed) return err(new AnalysisCancelledError(operation));
if (identityConflict)
return identityConflictFailure(operation, imageBase, observations);
if (controller.signal.aborted)
return err(new AnalysisCancelledError(operation));
return settled === ABORTED
? err(new AnalysisCancelledError(operation))
: settled;
};
return {
derive,
readTypeDetail: (address) =>
result?.readTypeDetail(address, NATIVE_AOT_MAX_REPORT_BYTES),
close: () => {
closed = true;
controller.abort();
result = undefined;
pending = undefined;
},
};
};
/** Build one AnalysisClient for an admitted Ghidra target and profile. */
export const createGhidraProviderClient = (input: {
readonly config: AppConfig;
@@ -258,6 +514,14 @@ export const createGhidraProviderClient = (input: {
...seedLimitations,
...(releaseLimitation === undefined ? [] : [releaseLimitation]),
];
const nativeAotOwner =
target.format === "pe"
? createNativeAotSnapshotOwner({
client,
target,
startupTimeoutMs: config.ghidraStartupTimeoutMs,
})
: undefined;
return {
execute: async (operation, parameters, options) => {
if (extensionFailure !== undefined) return err(extensionFailure);
@@ -334,19 +598,83 @@ export const createGhidraProviderClient = (input: {
if (!result.ok) return result;
normalized = result.value;
}
if (operation === "inspect_native_data_type") {
const address = input.value.address;
if (typeof address === "string") {
if (target.format === "pe" && nativeAotOwner !== undefined) {
const started = await client.start(options?.signal);
if (!started.ok)
return err(
projectSessionError(
operation,
started.error,
config.ghidraStartupTimeoutMs,
),
);
const derived = await nativeAotOwner.derive(
started.value.target.image_base,
operation,
options?.signal,
normalized,
);
if (!derived.ok) return derived;
}
let metadata:
| ReturnType<NativeAotPeResult["readTypeDetail"]>
| undefined;
metadata = nativeAotOwner?.readTypeDetail(address);
if (
metadata !== undefined &&
normalized !== null &&
typeof normalized === "object" &&
!Array.isArray(normalized)
) {
const nativeResult = normalized as Record<string, JsonValue>;
if (nativeResult.metadata_recovery === undefined)
normalized = jsonValueSchema.parse({
...nativeResult,
metadata_recovery: metadata,
});
}
}
}
if (operation === "inspect_native_load_image") {
const attested = await attestGhidraNativeLoadImage(
let derived: NativeAotPeResult | undefined;
if (target.format === "pe") {
const observations =
nativeLoadImageObservationSchema.parse(normalized);
if (nativeAotOwner === undefined)
return err(
new ProviderAdapterError("ghidra", operation, {
diagnostics: {
reason:
"The admitted PE session has no read-only NativeAOT snapshot owner.",
},
}),
);
const result = await nativeAotOwner.derive(
observations.image_base,
operation,
options?.signal,
normalized,
);
if (!result.ok) return result;
derived = result.value;
}
const attested = await attestGhidraNativeLoadImage({
target,
operation,
normalized,
measured: normalized,
client,
(failure) =>
mapSessionError: (failure) =>
projectSessionError(
operation,
failure,
config.ghidraStartupTimeoutMs,
),
);
nativeAotResult: derived,
signal: options?.signal,
});
if (!attested.ok) return attested;
normalized = attested.value;
}
@@ -368,7 +696,10 @@ export const createGhidraProviderClient = (input: {
? []
: [{ provider: committedProfile.provider, observation }];
},
close: () => client.close(),
close: async () => {
nativeAotOwner?.close();
return client.close();
},
};
};
+9
View File
@@ -44,6 +44,15 @@ describe("ghidraSessionRoot", () => {
).toBe(posixBase("Users", "operator"));
});
it("can select a filesystem root when the inherited temp starts with a dot-prefixed element", () => {
expect(
ghidraSessionRoot({
base: posixBase(".Tmp", "scratch"),
platform: "win32",
}),
).toBe(resolve(sep));
});
it.skipIf(!POSIX)("prefers the platform temp directory by default", () => {
expect(
ghidraSessionRoot({ base: posixBase("home", "operator", ".cache") }),
+3 -1
View File
@@ -1,6 +1,6 @@
import { randomUUID } from "node:crypto";
import { constants } from "node:fs";
import { access, realpath } from "node:fs/promises";
import { access, realpath, stat } from "node:fs/promises";
import { isAbsolute } from "node:path";
import {
AnalysisCancelledError,
@@ -87,6 +87,8 @@ export const resolveWakaruCommand = async (
if (!isAbsolute(limiter))
throw new TypeError("REA_JAVASCRIPT_PRLIMIT_COMMAND must be absolute");
await access(limiter, constants.X_OK);
if (!(await stat(limiter)).isFile())
throw new TypeError(`Resource limiter is not a regular file: ${limiter}`);
return { command, limiter, ...fingerprint };
} catch (cause: unknown) {
const reason = `Configured Wakaru or util-linux prlimit is unavailable: ${recoveryFailureMessage(cause)}`;
+12 -5
View File
@@ -28,6 +28,12 @@ interface Section {
readonly size: number;
}
// RVA ownership follows the declared virtual extent. Raw size describes
// initialized file bytes and can include file-alignment padding; loaders may
// map page-rounded tails, but those bytes are outside the declared section.
const virtualExtent = (section: Section): number =>
section.virtualSize === 0 ? section.size : section.virtualSize;
/** Header admission and unambiguous file-backed RVA mapping, independent of section names. */
export const readPeResourceLayout = (bytes: Buffer) => {
requirePeRange(bytes, 0, 64);
@@ -64,7 +70,7 @@ export const readPeResourceLayout = (bytes: Buffer) => {
offset: bytes.readUInt32LE(at + 20),
size: bytes.readUInt32LE(at + 16),
};
if (section.rva + Math.max(section.virtualSize, section.size) > 0x100000000)
if (section.rva + virtualExtent(section) > 0x100000000)
peFailure("PE section RVA range overflows.");
if (section.size > 0) {
requirePeRange(bytes, section.offset, section.size);
@@ -90,15 +96,14 @@ export const readPeResourceLayout = (bytes: Buffer) => {
if (rva + size > 0x100000000) peFailure("Resource RVA range overflows.");
const candidates = sections.filter(
(section) =>
rva >= section.rva &&
rva - section.rva < Math.max(section.size, section.virtualSize),
rva >= section.rva && rva - section.rva < virtualExtent(section),
);
if (rva < headerSize) {
if (
sections.some(
(section) =>
section.rva < rva + size &&
section.rva + Math.max(section.size, section.virtualSize) > rva,
section.rva + virtualExtent(section) > rva,
) ||
size > headerSize - rva
)
@@ -111,6 +116,8 @@ export const readPeResourceLayout = (bytes: Buffer) => {
const section = candidates[0];
if (section === undefined) return peFailure("Missing resource section.");
const within = rva - section.rva;
if (size > virtualExtent(section) - within)
peFailure("Resource range leaves its section's virtual size.");
if (within > section.size || size > section.size - within)
peFailure("Resource range is not entirely file-backed in one section.");
// A second section starting inside the requested interval is ambiguous too.
@@ -119,7 +126,7 @@ export const readPeResourceLayout = (bytes: Buffer) => {
(other) =>
other !== section &&
other.rva < rva + size &&
other.rva + Math.max(other.size, other.virtualSize) > rva,
other.rva + virtualExtent(other) > rva,
)
)
peFailure("Resource interval intersects overlapping virtual sections.");
+59
View File
@@ -259,3 +259,62 @@ describe("PE header admission and bounded traversal", () => {
await expect(parsing).rejects.toThrow();
});
});
describe("PE resource section virtual extent", () => {
const sectionSizes = (fixture: ReturnType<typeof peResourceFixture>) => {
const virtualSize = fixture.bytes.readUInt32LE(fixture.sectionAt + 8);
const rawSize = fixture.bytes.readUInt32LE(fixture.sectionAt + 16);
expect(rawSize - virtualSize).toBeGreaterThanOrEqual(8);
return { virtualSize, rawSize };
};
it.each([
[0, "Resource RVA has missing or overlapping section mappings."],
[-2, "Resource range leaves its section's virtual size."],
])(
"rejects a payload reaching raw alignment padding past VirtualSize (start %i)",
async (start, message) => {
const fixture = peResourceFixture();
const { virtualSize } = sectionSizes(fixture);
const rva = 0x2000 + virtualSize + start;
fixture.bytes.writeUInt32LE(rva, fixture.dataOffsets[0]);
fixture.bytes.writeUInt32LE(4, (fixture.dataOffsets[0] ?? 0) + 4);
await expect(parse(fixture.bytes)).rejects.toMatchObject({
reason: "format",
message,
});
},
);
it("maps raw bytes when VirtualSize is zero, as the image loader does", async () => {
const fixture = peResourceFixture();
fixture.bytes.writeUInt32LE(0, fixture.sectionAt + 8);
expect(await parse(fixture.bytes)).toMatchObject({
coverage: { status: "complete", resources: 5 },
});
});
it("selects a later section that starts inside earlier raw padding", async () => {
const fixture = peResourceFixture();
const { virtualSize } = sectionSizes(fixture);
const laterRva = 0x2000 + Math.ceil(virtualSize / 4) * 4;
const rawOffset = fixture.bytes.length;
const payload = Buffer.from("LATE");
const bytes = Buffer.concat([fixture.bytes, payload]);
bytes.writeUInt16LE(2, 0x86);
const later = fixture.sectionAt + 40;
bytes.writeUInt32LE(payload.length, later + 8);
bytes.writeUInt32LE(laterRva, later + 12);
bytes.writeUInt32LE(payload.length, later + 16);
bytes.writeUInt32LE(rawOffset, later + 20);
bytes.writeUInt32LE(laterRva, fixture.dataOffsets[0]);
bytes.writeUInt32LE(payload.length, (fixture.dataOffsets[0] ?? 0) + 4);
const result = peResourcesSchema.parse(await parse(bytes));
expect(result.resources[0]?.payload).toMatchObject({
rva: laterRva,
location: { offset: rawOffset, bytes: payload.length },
sha256: createHash("sha256").update(payload).digest("hex"),
});
});
});
+66 -41
View File
@@ -5,7 +5,9 @@ import type { ExecutionOptions } from "../../application/AnalysisProvider.js";
import { ArtifactOperationError } from "../../domain/artifactOperationError.js";
import {
AnalysisAccessDeniedError,
AnalysisArtifactChangedError,
AnalysisCancelledError,
AnalysisInputError,
AnalysisResourceConstraintError,
} from "../../domain/analysisErrorCore.js";
import type { InspectPeResourcesInput } from "../../domain/native/peResources.js";
@@ -37,47 +39,70 @@ export class PeResourcesProvider implements PeResourcesPort {
),
);
} catch (cause) {
if (options?.signal?.aborted)
return err(
new AnalysisCancelledError("inspect_pe_resources", { cause }),
);
if (cause instanceof ArtifactReaderFailure && cause.reason === "limit")
return err(
new AnalysisResourceConstraintError(
"inspect_pe_resources",
"memory",
cause.message,
{
max_file_bytes: input.max_file_bytes,
max_entries: input.max_entries,
},
{ cause },
),
);
if (
cause instanceof Error &&
"code" in cause &&
(cause.code === "EACCES" || cause.code === "EPERM")
)
return err(
new AnalysisAccessDeniedError(
"inspect_pe_resources",
input.path,
cause.code,
{ cause },
),
);
return err(
new ArtifactOperationError(
"inspect_pe_resources",
cause instanceof ArtifactReaderFailure ? cause.reason : "io",
undefined,
cause instanceof Error
? cause.message
: "PE artifact inspection failed.",
{ cause },
),
);
return err(peResourcesFailure(cause, input, options?.signal));
}
}
}
const peResourcesFailure = (
cause: unknown,
input: InspectPeResourcesInput,
signal: AbortSignal | undefined,
) => {
const operation = "inspect_pe_resources";
if (signal?.aborted) return new AnalysisCancelledError(operation, { cause });
if (cause instanceof ArtifactReaderFailure) {
if (cause.reason === "limit")
return new AnalysisResourceConstraintError(
operation,
"memory",
cause.message,
{
max_file_bytes: input.max_file_bytes,
max_entries: input.max_entries,
},
{ cause },
);
if (cause.reason === "integrity")
return new AnalysisArtifactChangedError(
operation,
input.path,
cause.message,
{ cause },
);
if (cause.reason === "path" || cause.reason === "format")
return new AnalysisInputError(operation, { cause }, [
{
path: ["path"],
reason: "invalid_format",
message: cause.message,
},
]);
if (cause.reason === "cancelled")
return new AnalysisCancelledError(operation, { cause });
}
if (cause instanceof Error && "code" in cause) {
if (cause.code === "EACCES" || cause.code === "EPERM")
return new AnalysisAccessDeniedError(operation, input.path, cause.code, {
cause,
});
if (cause.code === "ENOENT" || cause.code === "ENOTDIR")
return new AnalysisInputError(operation, { cause }, [
{
path: ["path"],
reason: "invalid_value",
message: `Selected PE image could not be read (${cause.code}): ${input.path}.`,
},
]);
}
return new ArtifactOperationError(
operation,
cause instanceof ArtifactReaderFailure ? cause.reason : "io",
undefined,
cause instanceof Error ? cause.message : "PE artifact inspection failed.",
{ cause },
);
};
+33 -7
View File
@@ -1,27 +1,53 @@
import type { FileHandle } from "node:fs/promises";
/** Read through EOF, returning undefined on overflow; consume at most maxBytes + 1 bytes without closing the handle. */
/** Read through EOF, returning undefined on overflow; leave the handle open. */
export const readBoundedFileBytes = async (
handle: FileHandle,
maxBytes: number,
signal?: AbortSignal,
expectedSize?: number,
): Promise<Buffer | undefined> => {
if (!Number.isSafeInteger(maxBytes) || maxBytes < 0)
throw new RangeError("File byte limit must be a nonnegative safe integer");
const chunks: Buffer[] = [];
const buffer = Buffer.allocUnsafe(Math.min(64 * 1_024, maxBytes + 1));
if (
expectedSize !== undefined &&
(!Number.isSafeInteger(expectedSize) ||
expectedSize < 0 ||
expectedSize > maxBytes)
)
throw new RangeError("Expected file size must fit the byte limit");
if (expectedSize === undefined) {
const chunks: Buffer[] = [];
const buffer = Buffer.allocUnsafe(Math.min(64 * 1_024, maxBytes + 1));
let total = 0;
for (;;) {
signal?.throwIfAborted();
const { bytesRead } = await handle.read(
buffer,
0,
Math.min(buffer.length, maxBytes - total + 1),
null,
);
if (bytesRead === 0) return Buffer.concat(chunks, total);
total += bytesRead;
if (total > maxBytes) return undefined;
chunks.push(Buffer.from(buffer.subarray(0, bytesRead)));
}
}
const buffer = Buffer.allocUnsafe(expectedSize);
if (expectedSize === 0) return buffer;
let total = 0;
for (;;) {
signal?.throwIfAborted();
const { bytesRead } = await handle.read(
buffer,
0,
Math.min(buffer.length, maxBytes - total + 1),
total,
expectedSize - total,
null,
);
if (bytesRead === 0) return Buffer.concat(chunks, total);
if (bytesRead === 0) return buffer.subarray(0, total);
total += bytesRead;
if (total > maxBytes) return undefined;
chunks.push(Buffer.from(buffer.subarray(0, bytesRead)));
if (total === expectedSize) return buffer;
}
};
+60 -15
View File
@@ -339,11 +339,61 @@ interface OwnedCleanupPlan {
readonly unverified: readonly OwnershipSweepUnverifiedProcess[];
}
const createOwnedCleanupPlan = async (
// A process-table read can race exit and retain the old state with an altered
// command. Only a fresh table proving that PID is no longer live can settle
// that mismatch; surviving group members still require normal token checks.
const settleLauncherExit = async (
ownership: OwnedProcessGroup,
processes: readonly ProcessTableEntry[],
host: ProcessOwnershipHost,
): Promise<
| readonly ProcessTableEntry[]
| Extract<ProcessCleanupResult, { readonly cleaned: false }>
> => {
const launcher = processes.find(({ pid }) => pid === ownership.leaderPid);
if (launcher === undefined) return processes;
const reason = launcherIdentityFailure(launcher, ownership);
if (reason === null) return processes;
const refreshed = await readLiveProcessTable(host);
if (
refreshed.available &&
!refreshed.processes.some(({ pid }) => pid === launcher.pid)
) {
// Refreshing must not discard a surviving detached descendant's lineage,
// including a child that changed groups while the launcher exited.
const descendants = descendantsOf([launcher.pid], processes);
const descendantPids = new Set(descendants.map(({ pid }) => pid));
const descendantGroupIds = new Set(
descendants.map(({ processGroupId }) => processGroupId),
);
const refreshedDescendantPids = new Set(
descendantsOf(
descendants.map(({ pid }) => pid),
refreshed.processes,
).map(({ pid }) => pid),
);
if (
refreshed.processes.some(
(process) =>
process.processGroupId !== ownership.processGroupId &&
(refreshedDescendantPids.has(process.pid) ||
descendantPids.has(process.pid) ||
descendantGroupIds.has(process.processGroupId)),
)
)
return { cleaned: false, reason };
return refreshed.processes;
}
return { cleaned: false, reason };
};
const createOwnedCleanupPlan = async (
ownership: OwnedProcessGroup,
initialProcesses: readonly ProcessTableEntry[],
host: ProcessOwnershipHost,
): Promise<OwnedCleanupPlan | ProcessCleanupResult> => {
const processes = await settleLauncherExit(ownership, initialProcesses, host);
if ("cleaned" in processes) return processes;
const sampledProcessGroupIds = new Set(
(ownership.sampledProcessGroupIds ?? []).filter(
(processGroupId) =>
@@ -397,10 +447,7 @@ const createOwnedCleanupPlan = async (
};
let descendants: readonly ProcessTableEntry[] = [];
if (launcher !== undefined) {
const identityFailure = launcherIdentityFailure(launcher, ownership);
if (identityFailure !== null)
return { cleaned: false, reason: identityFailure };
descendants = descendantsOf(launcher.pid, processes);
descendants = descendantsOf([launcher.pid], processes);
}
const descendantPids = new Set(descendants.map(({ pid }) => pid));
const processGroupIds = new Set<number>([ownership.processGroupId]);
@@ -496,7 +543,13 @@ const revalidateOwnedProcessGroup = async (
): Promise<{ readonly empty: boolean } | ProcessCleanupResult> => {
let members: readonly ProcessTableEntry[];
try {
members = liveProcesses(await host.listProcesses()).filter(
const processes = await settleLauncherExit(
ownership,
liveProcesses(await host.listProcesses()),
host,
);
if ("cleaned" in processes) return processes;
members = processes.filter(
({ processGroupId: observedGroupId }) =>
observedGroupId === processGroupId,
);
@@ -513,14 +566,6 @@ const revalidateOwnedProcessGroup = async (
expectedIdentities,
);
if (failures.length > 0) return cleanupValidationFailure(failures);
if (processGroupId === ownership.processGroupId) {
const launcher = members.find(({ pid }) => pid === ownership.leaderPid);
if (launcher !== undefined) {
const identityFailure = launcherIdentityFailure(launcher, ownership);
if (identityFailure !== null)
return { cleaned: false, reason: identityFailure };
}
}
if (
processGroupId !== ownership.processGroupId &&
members.length > 0 &&
@@ -850,7 +895,7 @@ const scanTokenOwnedProcesses = async (
const relatedGroupIds = new Set<number>();
if (relation !== undefined) {
relatedPids.add(relation.leaderPid);
for (const descendant of descendantsOf(relation.leaderPid, processes))
for (const descendant of descendantsOf([relation.leaderPid], processes))
relatedPids.add(descendant.pid);
relatedGroupIds.add(relation.processGroupId);
for (const processGroupId of relation.sampledProcessGroupIds ?? [])
+187 -22
View File
@@ -3,6 +3,7 @@ import {
cleanupOwnedProcessGroup,
cleanupWindowsProcessTree,
type ProcessOwnershipHost,
type ProcessTableEntry,
type WindowsProcessTreeHost,
} from "./ProcessOwnership.js";
import {
@@ -11,18 +12,20 @@ import {
} from "./ProcessOwnershipObservation.js";
import { host, ownership } from "./ProcessOwnership.fixture.js";
const launcher: ProcessTableEntry = {
pid: 100,
parentPid: 1,
processGroupId: 100,
state: "S",
command: "fixture",
};
describe("owned process-group cleanup validation: ownership and lineage", () => {
it("fails closed when a descendant in another process group lacks the token", async () => {
const adapter: ProcessOwnershipHost = {
listProcesses: () =>
Promise.resolve([
{
pid: 100,
parentPid: 1,
processGroupId: 100,
state: "S",
command: "fixture",
},
launcher,
{
pid: 101,
parentPid: 100,
@@ -211,16 +214,9 @@ describe("owned process-group cleanup liveness rechecks", () => {
);
it("accepts a member that exits during ownership revalidation", async () => {
const liveLauncher = {
pid: 100,
parentPid: 1,
processGroupId: 100,
state: "S",
command: "fixture",
};
const listProcesses = vi
.fn<ProcessOwnershipHost["listProcesses"]>()
.mockResolvedValueOnce([liveLauncher])
.mockResolvedValueOnce([launcher])
.mockResolvedValue([]);
const signalGroup = vi.fn();
const adapter: ProcessOwnershipHost = {
@@ -239,6 +235,181 @@ describe("owned process-group cleanup liveness rechecks", () => {
});
});
describe("owned process-group cleanup validation: refreshed descendant lineage", () => {
it.each(["planning", "signaling"])(
"retains cleanup uncertainty for newly detached descendants during %s",
async (phase) => {
const child = { ...launcher, pid: 101, parentPid: 100 };
const stale = { ...launcher, command: "[MainThread]" };
const signalGroup = vi.fn();
let reads = 0;
const adapter: ProcessOwnershipHost = {
listProcesses: () => {
reads += 1;
if (phase === "signaling" && reads === 1)
return Promise.resolve([launcher, child]);
if (reads === (phase === "planning" ? 1 : 2))
return Promise.resolve([stale, child]);
return Promise.resolve([
{ ...child, parentPid: 1 },
{ ...child, pid: 102, parentPid: 101 },
{ ...child, pid: 103, parentPid: 102, processGroupId: 103 },
]);
},
environment: () =>
Promise.resolve({ REA_PROCESS_RUN_ID: ownership.runId }),
signalGroup,
};
await expect(
cleanupOwnedProcessGroup(
{ ...ownership, expectedCommand: "fixture" },
adapter,
),
).resolves.toMatchObject({ cleaned: false });
expect(signalGroup).not.toHaveBeenCalled();
},
);
});
describe("owned process-group cleanup validation: launcher exit races", () => {
it.each([101, 999])(
"retains cleanup uncertainty for a detached descendant in group %s after launcher exit",
async (processGroupId) => {
const child = {
...launcher,
pid: 101,
parentPid: 100,
processGroupId: 101,
};
const signalGroup = vi.fn();
const listProcesses = vi
.fn<ProcessOwnershipHost["listProcesses"]>()
.mockResolvedValueOnce([
{ ...launcher, command: "[MainThread]" },
child,
])
.mockResolvedValue([{ ...child, parentPid: 1, processGroupId }]);
await expect(
cleanupOwnedProcessGroup(
{
...ownership,
expectedCommand: "fixture",
},
{
listProcesses,
environment: () =>
Promise.resolve({ REA_PROCESS_RUN_ID: "run-token" }),
signalGroup,
},
),
).resolves.toMatchObject({ cleaned: false });
expect(signalGroup).not.toHaveBeenCalled();
},
);
it.each(["planning", "signaling"])(
"settles a stale launcher command during %s without skipping surviving group ownership",
async (phase) => {
for (const remaining of ["absent", "zombie", "owned", "foreign"]) {
const stale = { ...launcher, command: "[MainThread]" };
const child = { ...launcher, pid: 101, command: "child" };
let reads = 0;
const signalGroup = vi.fn();
const adapter: ProcessOwnershipHost = {
listProcesses: () => {
reads += 1;
if (phase === "signaling" && reads === 1)
return Promise.resolve([launcher]);
if (reads === (phase === "planning" ? 1 : 2))
return Promise.resolve([stale]);
return Promise.resolve(
remaining === "absent"
? []
: remaining === "zombie"
? [{ ...stale, state: "Z" }]
: [child],
);
},
environment: () =>
Promise.resolve({
REA_PROCESS_RUN_ID:
remaining === "foreign" && reads > 1
? "other-run"
: "run-token",
}),
signalGroup,
};
const result = await cleanupOwnedProcessGroup(
{ ...ownership, expectedCommand: "fixture" },
adapter,
);
if (remaining === "foreign")
expect(result).toMatchObject({
cleaned: false,
reason: "process tree contains an unowned or PID-reused process",
});
else
expect(result).toEqual({
cleaned: true,
signaled: remaining === "owned",
});
if (remaining === "owned")
expect(signalGroup).toHaveBeenCalledWith(100, "SIGKILL");
else expect(signalGroup).not.toHaveBeenCalled();
}
},
);
it("fails closed when a mismatched launcher's exit cannot be inspected", async () => {
const { adapter, signalGroup } = host({
100: { REA_PROCESS_RUN_ID: "run-token" },
});
const listProcesses = vi
.fn<ProcessOwnershipHost["listProcesses"]>()
.mockResolvedValueOnce(await adapter.listProcesses())
.mockRejectedValue(new Error("process table unavailable"));
await expect(
cleanupOwnedProcessGroup(
{ ...ownership, expectedCommand: "/owned/hopper" },
{ ...adapter, listProcesses },
),
).resolves.toMatchObject({
cleaned: false,
reason: expect.stringContaining("command identity did not match"),
});
expect(signalGroup).not.toHaveBeenCalled();
});
it.each(["planning", "signaling"])(
"rejects a surviving launcher PID in another group during %s",
async (phase) => {
let reads = 0;
const signalGroup = vi.fn();
const adapter: ProcessOwnershipHost = {
listProcesses: () => {
reads += 1;
return Promise.resolve([
phase === "signaling" && reads === 1
? launcher
: reads === (phase === "planning" ? 1 : 2)
? { ...launcher, command: "[MainThread]" }
: { ...launcher, processGroupId: 999 },
]);
},
environment: () => Promise.resolve({ REA_PROCESS_RUN_ID: "run-token" }),
signalGroup,
};
await expect(
cleanupOwnedProcessGroup(
{ ...ownership, expectedCommand: "fixture" },
adapter,
),
).resolves.toMatchObject({
cleaned: false,
reason: expect.stringContaining("command identity did not match"),
});
expect(signalGroup).not.toHaveBeenCalled();
},
);
});
describe("owned process-group cleanup validation: exited members", () => {
it("preserves cancellation during process-table observation instead of reporting unverifiable ownership", async () => {
const controller = new AbortController();
@@ -293,13 +464,7 @@ describe("owned process-group cleanup validation: exited members", () => {
const adapter: ProcessOwnershipHost = {
listProcesses: () =>
Promise.resolve([
{
pid: 100,
parentPid: 1,
processGroupId: 100,
state: "S",
command: "fixture",
},
launcher,
{
pid: 101,
parentPid: 100,
+1 -1
View File
@@ -116,7 +116,7 @@ export const observeOwnedProcessLineage = async (
const identityFailure = launcherIdentityFailure(launcher, ownership);
if (identityFailure !== null)
return unavailableLineage(ownership, identityFailure);
const descendants = descendantsOf(launcher.pid, processes);
const descendants = descendantsOf([launcher.pid], processes);
const verifiedDescendants: ProcessTableEntry[] = [];
for await (const { process: member, observation } of readProcessRunTokens(
host,
+4 -4
View File
@@ -6,9 +6,9 @@ export const liveProcesses = (
): readonly ProcessTableEntry[] =>
members.filter(({ state }) => !state.startsWith("Z"));
/** Return every live-table descendant of a launcher in breadth-first order. */
/** Return every live-table descendant of the selected roots in breadth-first order. */
export const descendantsOf = (
launcherPid: number,
rootPids: readonly number[],
processes: readonly ProcessTableEntry[],
): ProcessTableEntry[] => {
const childrenByParent = new Map<number, ProcessTableEntry[]>();
@@ -18,8 +18,8 @@ export const descendantsOf = (
childrenByParent.set(process.parentPid, siblings);
}
const descendants: ProcessTableEntry[] = [];
const pending = [...(childrenByParent.get(launcherPid) ?? [])];
const visited = new Set<number>([launcherPid]);
const pending = rootPids.flatMap((pid) => childrenByParent.get(pid) ?? []);
const visited = new Set<number>(rootPids);
for (const process of pending) {
if (visited.has(process.pid)) continue;
visited.add(process.pid);
+3 -2
View File
@@ -8,6 +8,7 @@ import {
import { toolContract } from "../contracts/toolContracts.js";
import type { EvidenceBundle } from "../domain/evidenceBundle.js";
import { ok } from "../domain/result.js";
import { inspectEvidenceBundle } from "../application/investigation/InspectEvidenceBundle.js";
import { toolRegistrationOptions } from "./toolRegistrationOptions.js";
interface EvidenceToolRegistration {
@@ -72,9 +73,9 @@ const registerSnapshotEvidenceTool = ({
server.registerTool(
snapshotContract.name,
toolRegistrationOptions(snapshotContract),
() =>
(input) =>
server.delivery.toCallToolResult(
ok(bundleForSerialization(session)),
inspectEvidenceBundle(bundleForSerialization(session), input),
snapshotContract,
),
);
@@ -0,0 +1,120 @@
import { mkdir, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect } from "vitest";
import { z } from "zod";
import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { cliTest } from "../../support/cli/cliFixture.js";
cliTest.for(["CLI", "stdio MCP"] as const)(
"keeps dependency manifests out of application roots and Electron entries through %s",
async (surface, { cli }) => {
const root = await createTestTempDirectory("rea-application-manifest-");
const dependency = join(root, "node_modules", "dep");
const nested = join(root, "a");
await Promise.all([mkdir(dependency, { recursive: true }), mkdir(nested)]);
await Promise.all([
writeFile(
join(nested, "package.json"),
JSON.stringify({ name: "nested" }),
),
writeFile(
join(root, "package.json"),
JSON.stringify({ name: "app", main: "main.js" }),
),
writeFile(
join(root, "main.js"),
"throw new Error('source must remain inert');",
),
writeFile(
join(dependency, "package.json"),
JSON.stringify({
name: "dep",
main: "index.js",
browser: "browser.js",
}),
),
writeFile(join(dependency, "index.js"), "module.exports = {};"),
writeFile(join(dependency, "browser.js"), "export const browser = true;"),
]);
let document: unknown;
if (surface === "CLI") {
const response = await cli.run({
arguments: ["analyze-javascript-application", root, "--json"],
});
expect(response.exitCode).toBe(0);
document = response.json;
} else {
const { call } = await connectLocalToolsMcp();
const response = await call("analyze_javascript_application", {
input_path: root,
});
expect(response.isError, JSON.stringify(response)).not.toBe(true);
document = response.structuredContent;
}
const { graph } = z
.object({ normalized_result: javascriptApplicationAnalysisResultSchema })
.parse(document).normalized_result;
expect(
graph.nodes
.filter(({ node_id }) => graph.root_node_ids.includes(node_id))
.flatMap(({ observations }) => observations.map(({ label }) => label)),
).toEqual(["app"]);
expect(
graph.nodes
.filter(({ kind }) => kind === "electron-main")
.flatMap(({ observations }) =>
observations.map(({ properties }) => properties.declared_path),
),
).toEqual(["main.js"]);
expect(
graph.nodes.filter(({ kind }) => kind === "electron-renderer"),
).toEqual([]);
expect(
graph.nodes
.filter(({ kind }) => kind === "package")
.flatMap(({ observations }) => observations.map(({ label }) => label))
.sort(),
).toEqual(["app", "dep", "nested"]);
},
);
cliTest(
"keeps the artifact root when only dependency manifests are present",
async ({ cli }) => {
const root = await createTestTempDirectory("rea-dependency-only-");
const dependency = join(root, "node_modules", "dep");
await mkdir(dependency, { recursive: true });
await writeFile(
join(dependency, "package.json"),
JSON.stringify({ name: "dep", main: "index.js" }),
);
await writeFile(join(dependency, "index.js"), "module.exports = {};");
const response = await cli.run({
arguments: ["analyze-javascript-application", root, "--json"],
});
expect(response.exitCode, response.stderr).toBe(0);
const { graph } = z
.object({ normalized_result: javascriptApplicationAnalysisResultSchema })
.parse(response.json).normalized_result;
expect(
graph.nodes
.filter(({ node_id }) => graph.root_node_ids.includes(node_id))
.map(({ kind }) => kind),
).toEqual(["artifact"]);
expect(
graph.nodes
.filter(({ kind }) => kind === "package")
.flatMap(({ observations }) => observations.map(({ label }) => label)),
).toEqual(["dep"]);
expect(
graph.nodes.some(
({ kind }) => kind === "electron-main" || kind === "electron-renderer",
),
).toBe(false);
},
);
@@ -6,7 +6,7 @@ import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { describe, expect, it } from "vitest";
import { afterAll, describe, expect, it } from "vitest";
import { z } from "zod";
import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js";
@@ -18,6 +18,27 @@ const requireFixture = createRequire(import.meta.url);
const comparisonEnvelope = z.object({
normalized_result: javaScriptExportShapeComparisonResultSchema,
});
let mcpComparisonConnection:
| {
readonly client: Client;
readonly transport: StdioClientTransport;
readonly ready: Promise<void>;
}
| undefined;
afterAll(async () => {
if (mcpComparisonConnection === undefined) return;
try {
await mcpComparisonConnection.ready;
} catch {
// Keep the failed connection owned until both resources are closed below.
}
try {
await mcpComparisonConnection.client.close();
} finally {
await mcpComparisonConnection.transport.close();
}
});
describe("exported function binding resolution through the CLI", () => {
it.each([
@@ -289,6 +310,113 @@ describe("CommonJS assignment-chain receiver identity through the CLI", () => {
});
});
describe("CommonJS export identity and scope regressions", () => {
it("does not export a named write through the original exports alias after replacement", async () => {
const fixture = await analyzeVersions(
(count) =>
`const VERSION = ${String(count)};\nexports.parse = (module.exports = () => ({ count: VERSION }));`,
"parse",
"app.cjs",
);
for (const path of [fixture.leftPath, fixture.rightPath]) {
const value = requireFixture(path);
expect(typeof value).toBe("function");
expect(Reflect.ownKeys(value)).toEqual(["length", "name"]);
}
const cli = await compareThroughCli(fixture);
expect(cli).toMatchObject({
left: { status: "missing" },
right: { status: "missing" },
summary: { added: 0, removed: 0, changed: 0 },
});
expect(await compareThroughMcp(fixture.input)).toEqual(cli);
});
it("keeps a dead var shadow of the alias-chain RHS unavailable", async () => {
const fixture = await analyzeVersions(
(count) =>
`function parse() { return { count: ${String(count)} }; }\nfunction configure() { if (false) { var parse = () => ({ count: 7 }); } module.exports = exports = parse; }\nconfigure();`,
"default",
"app.cjs",
);
expect(requireFixture(fixture.leftPath)).toBeUndefined();
expect(requireFixture(fixture.rightPath)).toBeUndefined();
const cli = await compareThroughCli(fixture);
expectUncertainComparison(cli);
expect(await compareThroughMcp(fixture.input)).toEqual(cli);
});
it.each(["+=", "||=", "??="] as const)(
"does not treat module.exports %s as a definite function export",
async (operator) => {
const fixture = await analyzeVersions(
(count) =>
`module.exports ${operator} (() => ({ kind: "result", count: ${String(count)} }));`,
"default",
"app.cjs",
);
const left = requireFixture(fixture.leftPath);
const right = requireFixture(fixture.rightPath);
if (operator === "+=") {
expect(typeof left).toBe("string");
expect(typeof right).toBe("string");
} else {
expect(left).toEqual({});
expect(right).toEqual({});
}
const cli = await compareThroughCli(fixture);
expectUncertainComparison(cli);
expect(await compareThroughMcp(fixture.input)).toEqual(cli);
},
);
it("keeps a parameter-default mutation unknown in a CommonJS return shape", async () => {
const fixture = await analyzeVersions(
(count) =>
`const source = { value: 0 }; module.exports = exports = function current(arg = null, set = (arg = source)) { var arg; arg.value = ${String(count)}; return { kind: "result", count: source.value }; };`,
"default",
"app.cjs",
);
expect(requireFixture(fixture.leftPath)()).toEqual({
kind: "result",
count: 1,
});
expect(requireFixture(fixture.rightPath)()).toEqual({
kind: "result",
count: 2,
});
for (const evidence of [fixture.input.left, fixture.input.right]) {
const analysis = z
.object({
normalized_result: javascriptApplicationAnalysisResultSchema,
})
.parse(evidence).normalized_result;
const shapes = analysis.graph.nodes
.flatMap(({ observations }) => observations)
.find(
({ properties }) =>
properties.semantic_role === "export-return-shapes" &&
properties.exported_name === "default",
)?.properties.static_return_shapes;
expect(shapes).toEqual([
expect.objectContaining({
fields: expect.arrayContaining([
expect.objectContaining({ path: "/count", state: "unknown" }),
]),
}),
]);
}
const cli = await compareThroughCli(fixture);
expect(cli).toMatchObject({
left: { status: "selected" },
right: { status: "selected" },
summary: { added: 0, removed: 0, changed: 0 },
changes: [],
});
expect(await compareThroughMcp(fixture.input)).toEqual(cli);
});
});
describe("stable exported function bindings through the CLI", () => {
it.each([
{
@@ -798,34 +926,37 @@ const compareThroughCli = async (
};
const compareThroughMcp = async (input: Record<string, unknown>) => {
const transport = new StdioClientTransport({
command: process.execPath,
args: [resolve("scripts/rea.mjs"), "mcp"],
cwd: process.cwd(),
env: { PATH: process.env.PATH ?? "", REA_LOG_LEVEL: "silent" },
stderr: "pipe",
const client = await getMcpComparisonClient();
const response = await client.callTool({
name: "compare_javascript_export_shapes",
arguments: input,
});
const client = new Client({
name: "exported-function-bindings",
version: "1",
});
try {
await client.connect(transport);
const response = await client.callTool({
name: "compare_javascript_export_shapes",
arguments: input,
expect(response.isError).not.toBe(true);
await client.ping();
return comparisonEnvelope.parse(response.structuredContent).normalized_result;
};
const getMcpComparisonClient = async (): Promise<Client> => {
if (mcpComparisonConnection === undefined) {
const transport = new StdioClientTransport({
command: process.execPath,
args: [resolve("scripts/rea.mjs"), "mcp"],
cwd: process.cwd(),
env: { PATH: process.env.PATH ?? "", REA_LOG_LEVEL: "silent" },
stderr: "pipe",
});
expect(response.isError).not.toBe(true);
await client.ping();
return comparisonEnvelope.parse(response.structuredContent)
.normalized_result;
} finally {
try {
await client.close();
} finally {
await transport.close();
}
const client = new Client({
name: "exported-function-bindings",
version: "1",
});
mcpComparisonConnection = {
client,
transport,
ready: client.connect(transport),
};
}
await mcpComparisonConnection.ready;
return mcpComparisonConnection.client;
};
const runtimeExport = async (
@@ -8,7 +8,87 @@ import { toolContract } from "../../../src/contracts/toolContracts.js";
import { peResourcesSchema } from "../../../src/domain/native/peResources.js";
import { peResourceFixture } from "../../../src/native/pe/PeResources.fixture.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { cliTest } from "../../support/cli/cliFixture.js";
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
import { cliTest, type TestCli } from "../../support/cli/cliFixture.js";
const expectPeInputFailureAcrossAdapters = async (input: {
readonly cli: TestCli;
readonly client: Client;
readonly toolName: string;
readonly path: string;
readonly reason: "invalid_format" | "invalid_value";
}) => {
const { cli, client, toolName, path, reason } = input;
const result = await cli.run({
arguments: ["inspect-pe-resources", path, "--json"],
});
expect(result.exitCode).toBe(1);
expect(result.json).toMatchObject({
code: "invalid_request",
category: "invalid_input",
details: {
operation: "inspect_pe_resources",
issues: [{ path: ["path"], reason }],
},
});
const response = await client.callTool({
name: toolName,
arguments: { path },
});
expect(response.isError).toBe(true);
expect(parseMcpToolError(response)).toEqual({ error: result.json });
};
const expectPePaddingFailureAcrossAdapters = async (input: {
readonly cli: TestCli;
readonly client: Client;
readonly toolName: string;
readonly path: string;
}) => {
const { cli, client, toolName, path } = input;
const fixture = peResourceFixture();
const virtualSize = fixture.bytes.readUInt32LE(fixture.sectionAt + 8);
fixture.bytes.writeUInt32LE(
0x2000 + virtualSize,
fixture.dataOffsets[0] ?? 0,
);
fixture.bytes.writeUInt32LE(4, (fixture.dataOffsets[0] ?? 0) + 4);
await writeFile(path, fixture.bytes);
const cliResult = await cli.run({
arguments: ["inspect-pe-resources", path, "--json"],
});
expect(cliResult.exitCode).not.toBe(0);
const mcpResult = await client.callTool({
name: toolName,
arguments: { path },
});
expect(mcpResult.isError).toBe(true);
expect(peResourcesSchema.safeParse(mcpResult.structuredContent).success).toBe(
false,
);
};
const expectPeInputFailuresAcrossAdapters = async (input: {
readonly cli: TestCli;
readonly client: Client;
readonly toolName: string;
readonly root: string;
}) => {
const { cli, client, toolName, root } = input;
const malformedPath = join(root, "malformed.exe");
await writeFile(malformedPath, Buffer.from("not a PE image"));
for (const [path, reason] of [
[join(root, "missing.exe"), "invalid_value"],
[malformedPath, "invalid_format"],
] as const)
await expectPeInputFailureAcrossAdapters({
cli,
client,
toolName,
path,
reason,
});
};
cliTest(
"inspects PE32/PE32+ resources through compiled CLI and real stdio MCP without an active target",
@@ -92,6 +172,18 @@ cliTest(
).toBe(evidence.evidence_id);
expect(await readFile(path)).toEqual(fixture.bytes);
}
await expectPePaddingFailureAcrossAdapters({
cli,
client,
toolName: contract.name,
path: join(root, "raw-padding.exe"),
});
await expectPeInputFailuresAcrossAdapters({
cli,
client,
toolName: contract.name,
root,
});
const absent = peResourceFixture();
absent.bytes.fill(0, absent.directoryAt, absent.directoryAt + 8);
const path = join(root, "absent.exe");
+158 -2
View File
@@ -30,8 +30,9 @@ const chromeExecutable = [
);
// Issue 1586's page, plus the HTML, image, use, and foreign-origin cases the
// real snapshot must distinguish. The test is skipped when no Chrome-family
// executable is installed.
// real snapshot must distinguish. The XHTML page is the prefixed-svg gap
// reported on PR 1602. The tests are skipped when no Chrome-family executable
// is installed.
const pageHtml = `<!doctype html><title>x</title>
<svg width="10" height="10">
<a id="s2" xlink:href="/xl-s2h"><text>s2</text></a>
@@ -159,6 +160,161 @@ it.skipIf(chromeExecutable === undefined)(
},
);
const xhtmlPage = `<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<head><title>x</title></head>
<body>
<svg:svg width="10" height="10">
<svg:a id="pa1" xlink:href="/xl-pa1"><svg:text>p</svg:text></svg:a>
<svg:image id="img" xlink:href="/xl-image"/>
<svg:foreignObject width="10" height="10">
<a id="fo-a" xlink:href="/xl-html-fo">no</a>
<h:a xmlns:h="http://www.w3.org/1999/xhtml" id="fo-h" xlink:href="/xl-h">no</h:a>
<svg:a id="fo-svg" xlink:href="/xl-fo-svg"><svg:text>y</svg:text></svg:a>
</svg:foreignObject>
</svg:svg>
<svg:a id="orphan" xlink:href="/xl-orphan"><svg:text>o</svg:text></svg:a>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:x="http://www.w3.org/1999/xlink" width="10" height="10">
<a id="custom" x:href="/xl-custom"><text>c</text></a>
<a id="both" href="/href-wins" x:href="/xlink-loses"><text>b</text></a>
<a id="empty" href="" x:href="/xlink-empty"><text>e</text></a>
</svg>
<svg:a id="out" xlink:href="https://cdn.example.test/out"><svg:text>out</svg:text></svg:a>
<bogus:svg xmlns:bogus="http://example.com/not-svg" id="bogus" xlink:href="/xl-bogus"/>
<a id="xhtml-a" xlink:href="/xl-xhtml-a">no</a>
</body>
</html>
`;
for (const { format, page, contentType, nodeNames, link, destinations } of [
{
format: "XHTML",
page: xhtmlPage,
contentType: "application/xhtml+xml",
nodeNames: ["html", "svg:a", "bogus:svg"],
link: { name: "svg:a", attribute: "xlink:href" },
destinations: [
"/xl-pa1",
"/xl-image",
"/xl-fo-svg",
"/xl-orphan",
"/xl-custom",
"/href-wins",
"",
null,
],
},
{
format: "SVG",
page: `<svg xmlns="http://www.w3.org/2000/svg" xmlns:p="http://www.w3.org/1999/xlink"><a p:href="/linked"><text>link</text></a></svg>`,
contentType: "image/svg+xml",
nodeNames: ["svg", "a"],
link: { name: "a", attribute: "p:href" },
destinations: ["/linked"],
},
]) {
it.skipIf(chromeExecutable === undefined)(
`reports namespaced SVG links from a standalone ${format} document through headless Chrome`,
async ({ processes }) => {
const executable = chromeExecutable;
if (executable === undefined) return;
const profile = await createTestTempDirectory("rea-svg-xhtml-");
const server = createServer((_request, response) => {
response.writeHead(200, {
"content-type": `${contentType}; charset=utf-8`,
});
response.end(page);
});
await new Promise<void>((resolve) => {
server.listen(0, "127.0.0.1", resolve);
});
const address = server.address();
if (address === null || typeof address === "string")
throw new Error("XHTML fixture server did not bind a TCP port");
const origin = `http://127.0.0.1:${String(address.port)}`;
const pageUrl = `${origin}/page.xhtml`;
let stderr = "";
const child = processes.spawn(executable, [
"--headless=new",
...(process.env.REA_BROWSER_NO_SANDBOX === "true"
? ["--no-sandbox"]
: []),
"--remote-debugging-address=127.0.0.1",
"--remote-debugging-port=0",
`--user-data-dir=${profile}`,
"--no-first-run",
"--no-default-browser-check",
"--disable-background-networking",
"--disable-component-update",
"--disable-dev-shm-usage",
"--disable-sync",
"--metrics-recording-only",
pageUrl,
]);
child.stdout.on("data", () => undefined);
child.stderr.on("data", (chunk: Buffer) => {
if (stderr.length < 8_192) stderr += chunk.toString("utf8");
});
try {
const port = await waitForBrowserDevtoolsPort({
child,
executable,
activePortPath: join(profile, "DevToolsActivePort"),
stderr: () => stderr,
timeoutMs: 20_000,
});
const endpoint = `http://127.0.0.1:${String(port)}`;
const provider = new CdpBrowserProvider();
const targetId = await pageTarget(provider, endpoint, origin, pageUrl);
const result = await provider.inspectPage(
inspectWebPageInputSchema.parse({
cdp_endpoint: endpoint,
allowed_origins: [origin],
target_id: targetId,
observation_ms: 0,
}),
);
if (!result.ok) throw result.error;
const urls = result.value.metadata.dom_urls;
for (const name of nodeNames) {
expect(
result.value.dom.nodes.some((node) => node.node_name === name),
).toBe(true);
}
expect(
result.value.dom.nodes.some(
(node) =>
node.node_name === link.name &&
node.attribute_names.includes(link.attribute),
),
).toBe(true);
expect(urls).toEqual(
destinations.map((destination) =>
expect.objectContaining({
attribute: "href",
url:
destination === null
? null
: destination === ""
? pageUrl
: `${origin}${destination}`,
destination_scope:
destination === null ? "outside_policy" : "approved",
}),
),
);
} finally {
child.kill("SIGTERM");
if (!(await waitForExit(child, 2_000))) child.kill("SIGKILL");
await waitForExit(child, 2_000);
await new Promise<void>((resolve) => {
server.close(() => resolve());
});
}
},
);
}
const pageTarget = async (
provider: CdpBrowserProvider,
endpoint: string,
@@ -2,8 +2,10 @@ import { PrivateRuntimeRoot } from "../../../src/process/PrivateRuntimeRoot.js";
import {
access,
chmod,
mkdir,
mkdtemp,
rm,
symlink,
truncate,
writeFile,
} from "node:fs/promises";
@@ -11,6 +13,7 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it, onTestFinished } from "vitest";
import { HistoricalCaptureDecoder } from "../../../src/browser/history/HistoricalCaptureDecoder.js";
import { MitmproxyCaptureAdapter } from "../../../src/browser/history/MitmproxyCaptureAdapter.js";
import { HAR_CAPTURE_PROVIDER_IDENTITY } from "../../../src/browser/history/CaptureRelease.js";
import {
inspectWebNetworkCaptureInputSchema,
@@ -20,6 +23,41 @@ import { historicalHar } from "../../fixtures/historicalHar.js";
import { decodeHarCapture } from "../../../src/browser/history/HarCapture.js";
import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js";
const linux = it.skipIf(process.platform !== "linux");
linux("refuses a directory configured as the mitmdump executable", async () => {
const root = await mkdtemp(join(tmpdir(), "rea-mitmdump-command-"));
onTestFinished(() => rm(root, { recursive: true, force: true }));
const command = join(root, "mitmdump");
await mkdir(command);
await expect(
new MitmproxyCaptureAdapter({ REA_MITMDUMP_COMMAND: command }).command(
join(root, "request.json"),
join(root, "runtime"),
),
).rejects.toMatchObject({
_tag: "AnalysisCapabilityUnavailableError",
reason: expect.stringContaining(command),
});
});
linux("allows a symlink to a regular mitmdump executable", async () => {
const root = await mkdtemp(join(tmpdir(), "rea-mitmdump-command-symlink-"));
onTestFinished(() => rm(root, { recursive: true, force: true }));
const command = join(root, "mitmdump");
const target = join(root, "mitmdump-bin");
await writeFile(target, "#!/bin/sh\n", { mode: 0o700 });
await symlink(target, command);
await expect(
new MitmproxyCaptureAdapter({ REA_MITMDUMP_COMMAND: command }).command(
join(root, "request.json"),
join(root, "runtime"),
),
).resolves.toMatchObject({ command });
});
it.each([true, false])(
"binds decoder replies to the complete selected provider identity (matching name: %s)",
async (matching) => {
@@ -9,6 +9,53 @@ import { cliTest } from "../../support/cli/cliFixture.js";
import { analysisViewLayoutEvidence } from "../../fixtures/analysisView.js";
import { createEvidence } from "../../../src/domain/evidence.js";
import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture.js";
import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js";
import { inspectEvidenceBundle } from "../../../src/application/investigation/InspectEvidenceBundle.js";
cliTest(
"discovers native records from a complete portable bundle using the shared workflow",
async ({ cli }) => {
const root = await createTestTempDirectory("rea-evidence-discovery-cli-");
const parent = createEvidence(
{ path: "/fixtures/native.exe", format: "pe", sha256: "a".repeat(64) },
{ id: "ghidra", name: "Ghidra", version: "12.1.4" },
{
operation: "analyze_function",
parameters: {},
result: ghidraFunctionDossier(),
},
);
const bundle = createEvidenceBundle([parent]);
const path = join(root, "evidence.json");
await writeFile(path, JSON.stringify(bundle));
const options = {
detail: "summary" as const,
filters: { evidence_id: parent.evidence_id },
};
const result = await cli.run({
arguments: [
"inspect-evidence-bundle",
JSON.stringify({ path, ...options }),
"--json",
],
environment: { HOME: root, XDG_CONFIG_HOME: root, XDG_CACHE_HOME: root },
});
expect(result.exitCode).toBe(0);
const expected = inspectEvidenceBundle(bundle, options);
if (!expected.ok) throw expected.error;
expect(result.json).toEqual(expected.value);
const complete = await cli.run({
arguments: [
"inspect-evidence-bundle",
JSON.stringify({ path }),
"--json",
],
environment: { HOME: root, XDG_CONFIG_HOME: root, XDG_CACHE_HOME: root },
});
expect(complete.exitCode).toBe(0);
expect(complete.json).toEqual(bundle);
},
);
cliTest(
"projects one section from a JSON file of inline layout Evidence",
async ({ cli }) => {
@@ -97,6 +97,14 @@ describe("browser comparison JSON file inputs", () => {
INTEGRATION_TEST_TIMEOUT_MS,
);
it(
"reports malformed PNG chunks as invalid input and valid unsupported PNGs as unsupported targets",
async () => {
await assertPngInputErrorProjections();
},
INTEGRATION_TEST_TIMEOUT_MS,
);
it(
"identifies a missing capture file instead of reporting malformed JSON",
async () => {
@@ -123,3 +131,105 @@ describe("browser comparison JSON file inputs", () => {
INTEGRATION_TEST_TIMEOUT_MS,
);
});
const assertPngInputErrorProjections = async (): Promise<void> => {
const damagedBytes = noisePng(1, 1);
damagedBytes[29] = (damagedBytes[29] ?? 0) ^ 1;
const damaged = JSON.stringify(createWebScreenshotArtifact(damagedBytes));
expect(
await runCli(["compare-web-screenshots", damaged, damaged, "--json"]),
).toMatchObject({
code: "invalid_request",
details: {
operation: "compare_web_screenshots",
issues: [
{ path: [], reason: "invalid_format", message: expect.any(String) },
],
},
});
const header = Buffer.alloc(13);
header.writeUInt32BE(1, 0);
header.writeUInt32BE(1, 4);
header.set([8, 2, 0, 0, 0], 8);
const malformedDeflate = JSON.stringify(
createWebScreenshotArtifact(
Buffer.concat([
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
pngChunk("IHDR", header),
pngChunk("IDAT", Buffer.from("bad")),
pngChunk("IEND", Buffer.alloc(0)),
]),
),
);
expect(
await runCli([
"compare-web-screenshots",
malformedDeflate,
malformedDeflate,
"--json",
]),
).toMatchObject({
code: "invalid_request",
details: {
operation: "compare_web_screenshots",
issues: [{ path: [], reason: "invalid_format" }],
},
});
const grayscaleHeader = Buffer.alloc(13);
grayscaleHeader.writeUInt32BE(1, 0);
grayscaleHeader.writeUInt32BE(1, 4);
grayscaleHeader.set([8, 0, 0, 0, 0], 8);
const grayscale = JSON.stringify(
createWebScreenshotArtifact(
Buffer.concat([
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
pngChunk("IHDR", grayscaleHeader),
pngChunk("IDAT", deflateSync(Buffer.from([0, 30]))),
pngChunk("IEND", Buffer.alloc(0)),
]),
),
);
expect(
await runCli(["compare-web-screenshots", grayscale, grayscale, "--json"]),
).toMatchObject({
code: "unsupported_target",
details: { operation: "compare_web_screenshots" },
});
const oversizedHeader = Buffer.alloc(13);
oversizedHeader.writeUInt32BE(0x7fffffff, 0);
oversizedHeader.writeUInt32BE(1, 4);
oversizedHeader.set([8, 6, 0, 0, 0], 8);
const hugeDimensions = JSON.stringify(
createWebScreenshotArtifact(
Buffer.concat([
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
pngChunk("IHDR", oversizedHeader),
pngChunk("IDAT", deflateSync(Buffer.from([0]))),
pngChunk("IEND", Buffer.alloc(0)),
]),
),
);
expect(
await runCli([
"compare-web-screenshots",
hugeDimensions,
hugeDimensions,
"--json",
]),
).toMatchObject({
code: "resource_constraint",
details: {
operation: "compare_web_screenshots",
resource: "memory",
reported_limits: {
maximum_working_memory_bytes: 256 * 1024 * 1024,
estimated_working_memory_bytes: "34359738499",
before_dimensions: { width: 0x7fffffff, height: 1 },
after_dimensions: { width: 0x7fffffff, height: 1 },
},
},
});
};
+58 -8
View File
@@ -1,5 +1,5 @@
import { execFile } from "node:child_process";
import { mkdir } from "node:fs/promises";
import { mkdir, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { promisify } from "node:util";
@@ -9,20 +9,34 @@ import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
const execute = promisify(execFile);
const MARKER = "REA_TEST_RESOLVED ";
// Dependencies that only one command uses; startup must not resolve them.
const DEFERRED_PACKAGES = ["playwright-core", "isomorphic-git"] as const;
// Dependencies and composition modules used by one command family; startup
// must not resolve them just to register the complete command catalog.
const DEFERRED_IMPORTS = [
{
name: "playwright-core",
path: "/node_modules/playwright-core/",
},
{
name: "isomorphic-git",
path: "/node_modules/isomorphic-git/",
},
{
name: "binary-composition",
path: "/dist/composition/binary.js",
},
] as const;
// Report each resolved deferred package on stderr without changing resolution.
const RESOLUTION_HOOK = `data:text/javascript,${encodeURIComponent(`
import { registerHooks } from "node:module";
const packages = ${JSON.stringify(DEFERRED_PACKAGES)};
const imports = ${JSON.stringify(DEFERRED_IMPORTS)};
registerHooks({
resolve(specifier, context, next) {
const resolved = next(specifier, context);
const name = packages.find((item) =>
resolved.url.includes("/node_modules/" + item + "/"),
const name = imports.find((item) =>
resolved.url.includes(item.path),
);
if (name !== undefined)
process.stderr.write(${JSON.stringify(MARKER)} + name + "\\n");
process.stderr.write(${JSON.stringify(MARKER)} + name.name + "\\n");
return resolved;
},
});
@@ -68,9 +82,45 @@ describe("CLI startup imports", () => {
).resolves.toContain("isomorphic-git");
});
it("does not load command-specific packages to register commands", async () => {
it("does not load command-specific code to register the command catalog", async () => {
await expect(
resolvedPackages(["scripts/rea.mjs", "--help"]),
).resolves.toEqual(new Set());
});
it("does not load native provider composition for JavaScript application analysis", async () => {
const source = await createTestTempDirectory("rea-cli-js-startup-");
await mkdir(join(source, "src"));
await writeFile(
join(source, "package.json"),
JSON.stringify({
name: "startup-fixture",
version: "1.0.0",
main: "src/index.js",
}),
);
await writeFile(
join(source, "src/index.js"),
"export const answer = 42;\n",
);
await expect(
resolvedPackages([
"scripts/rea.mjs",
"analyze-javascript-application",
source,
"--json",
]),
).resolves.toEqual(new Set());
});
it("loads native provider composition when a provider operation is called", async () => {
await expect(
resolvedPackages(
evaluate(
'const { createDirectAnalysis } = await import("./dist/composition/directAnalysis.js"); await createDirectAnalysis({}).runSessionStatus();',
),
),
).resolves.toContain("binary-composition");
});
});

Some files were not shown because too many files have changed in this diff Show More