Compare commits

...
Author SHA1 Message Date
morluto 9ec8267bee docs(readme): remove CI status badge 2026-10-11 19:09:11 +08:00
morluto d3a39a9376 fix(javascript): preserve source identity in multi-input recovery 2026-10-11 18:29:19 +08:00
JingHao-Leon 5ddf8766b8 feat(javascript): recover code-split bundles from multiple inputs
recover_javascript_sources accepted a single path, and
validateModuleSet bound every module's provenance to that one snapshot.
A code-split app (runtime + chunks, or an Electron main/preload/renderer
set) had to be recovered one file per call, and modules recovered alone
could not resolve imports across chunks: cross-chunk require(id)
references stayed as numeric ids with property reads.

Add an optional additional_paths input. The listed files are
concatenated, in order after path, into the single engine snapshot, so
one Wakaru run sees every module and cross-chunk imports resolve.
validateModuleSet's single-snapshot binding is unchanged. Module
provenance is attributed back to the input file its first extraction
range starts in, with byte ranges rebased to that file's own offsets;
a module whose ranges cross an input boundary keeps the concatenated
snapshot identity and offsets, recorded as an explicit limitation.
Single-path behavior is byte-for-byte unchanged (one segment,
offset 0).

Fixes #1605
2026-10-11 11:24:09 +08:00
morluto e5bc4e85ec fix(ci): close speculative browser fixture connections 2026-10-11 11:05:08 +08:00
38221cb12e fix(native): bound PE resource RVAs by section virtual size (#1678)
The PE resource reader treated each section as covering
`max(VirtualSize, SizeOfRawData)` bytes, the mapping #1596 removed from
managed RVA reads. A resource payload in the raw alignment padding past
VirtualSize was reported with a hash of bytes the image loader never maps.
A later section whose virtual range starts inside that padding was refused
as an overlapping mapping.

Containment and overlap now use the loader's virtual extent: VirtualSize,
or SizeOfRawData when VirtualSize is zero. Ranges must still be file-backed.

Refs #1596

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:01:45 -07:00
3749ccec47 feat(ghidra): recover signed AArch64 relative jump tables (#1592)
* feat(ghidra): recover signed AArch64 relative jump tables

Generalise the AArch64 relative-table matcher to signed byte/halfword entries, UXTW indexes, byte or LSL #2 offsets, B.CS/B.HS bounds and a CBZ route for zero entries. Cross-check it against Ghidra's own load table when one exists, and add a fixture modelled on the RenderScript blend kernel.

Refs #487

* fix(ghidra): preserve typed labels for normalized switch selectors

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 10:45:35 +08:00
a67dbf6a48 fix: reject directory paths for the Wakaru limiter and mitmdump
`REA_JAVASCRIPT_PRLIMIT_COMMAND` and `REA_MITMDUMP_COMMAND` were admitted
after an access check alone, which a searchable directory passes. Admission
then succeeded, and the failure surfaced later as a launch-time adapter
error instead of the capability refusal that names the configured path.

Require a regular file, as #751 did for the firmware engine and limiter.
Linux-only tests mirror that change; both checks only run on Linux hosts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
morlutoandClaude Opus 5.5 bd3a9bbbaf fix(native): classify PE resource input failures
Report missing and non-file selections as input issues, malformed PE images as format errors, and unstable reads as artifact changes. Keep existing limit, permission, cancellation, and other I/O classifications.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
morlutoandClaude Opus 5.5 74050978be fix(javascript): keep dependency manifests out of application roots
Select the shallowest inventoried nondependency manifest and retain dependency package evidence without assigning Electron entry roles. Keep the artifact root when no application manifest is observed.

Adapted from PR #1392, with compiled CLI and stdio MCP regressions for root selection and dependency-only inputs.

Co-authored-by: Marnix Pluim <marnixpluim@users.noreply.github.com>

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
b212e92f9b ci: select semantic tests and fixture lanes for their guarded inputs (#1675)
Edits to `release.yml` or `real-ghidra-windows.yml` selected only the
workflow lane, so the shards holding releaseWorkflow, releaseCheckpoint
and realGhidraWindowsWorkflow tests were skipped. Actionlint checks
syntax only, so a PR could drop the package-before-publish gate, the
registry canary or the default-branch-only self-hosted runner restriction
without CI noticing.

Conformance fixture sources under tests/ set only the code lane, so the
conformance-fixtures and reconstruction-readiness jobs, the only callers
of `verify:fixtures` and `verify:readiness`, never ran for them. Route
the directories that build-conformance-fixtures.mjs compiles to both
lanes, and readiness applications to the readiness lane.

Both were raised by the Codex review of #1660 after it merged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:41:40 -07:00
morluto 990e1ad8c1 fix(browser): clear losing storage verifier timers 2026-10-11 10:35:42 +08:00
morluto 76d8f68cb9 feat(ghidra): inspect NativeAOT PE metadata read-only 2026-10-11 10:31:49 +08:00
fe574c0c9f fix(javascript): recover Electron APIs through default imports and optional calls (#1673)
`import electron from "electron"` bound the module with a leading `default`
export segment, so `electron.ipcMain.handle(...)`, `new
electron.BrowserWindow(...)` and destructuring from that binding matched no
Electron API. Electron's CommonJS exports define no `default` member, so the
segment can only denote the module object; drop it when classifying
Electron bindings.

The binding classifier already recorded optional-call roots, but the IPC,
context-bridge, utility-process and sender-validation visitors accepted only
plain calls. `ipcMain?.handle(...)`, `contextBridge?.exposeInMainWorld(...)`
and the common `event.senderFrame?.url.startsWith(...)` check were dropped
without a limitation.

Refs #1656

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:31:08 +08:00
3a7a3074e3 feat(javascript): recognize tRPC procedures and GraphQL operations (#1670)
* feat(javascript): recognize tRPC procedures and GraphQL operations

Record RPC client syntax as network endpoint candidates in both
analyze_javascript_application and analyze_web_bundle, distinguished by
mechanism:

- trpc:procedure:<method> keeps the static member path of
  root.<path>.<method>(...), which tRPC proxies preserve through
  identifier minification. React hooks and TanStack option factories are
  recognized directly. Vanilla query/mutate need a root bound to
  createTRPCClient/createTRPCProxyClient, or a tRPC source marker with a
  router path of at least two segments; subscribe needs the marker and a
  tRPC observer argument. `this` roots, browser globals and RTK Query
  `endpoints` chains are never procedures; cache-key helpers are excluded.
- trpc🔗<link> records the literal url of documented HTTP links.
- graphql:<type> records named operations from string and cooked template
  literals with a valid operation header, and from precompiled
  OperationDefinition objects; a graphql-tag loc.source.body copy is not
  reported twice.

Method, callback and factory names follow @trpc/client and React/TanStack
integration sources and documentation for 11.19. Carrier segments before
the router path, useUtils helpers and minifier-renamed links remain
documented limitations.

Validation: typecheck; domain tests for recognition and false positives
(Apollo, RxJS, stores, RTK Query, browser globals, prose); web bundle
analyzer test; end-to-end analysis of a minified fixture through the
built MCP server on Windows 11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): preserve RPC binding identity and bundled calls

* style(reference-source): fix CI formatting

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 10:28:52 +08:00
morluto 83e52559b8 style(reference-source): format loader binding changes 2026-10-11 10:20:04 +08:00
morluto 824ecf109d feat(release): add contributor-aware release notes workflow 2026-10-11 10:19:16 +08:00
morlutoandLucien 8b4c3de3b3 fix(browser): seed storage by browser partition
Co-authored-by: Lucien <1428823446@qq.com>
2026-10-11 10:17:53 +08:00
morlutoandRudy Celekli f998fcb1cc fix(reference-source): respect require bindings
Reuse lexical ownership to exclude shadowed require calls while retaining native CommonJS and direct createRequire bindings. Preserve candidate edges when JavaScript or TypeScript runtime mode is unknown.

Co-authored-by: Rudy Celekli <rudy@gradiahq.com>
2026-10-11 10:10:59 +08:00
morluto f88c99021b fix(ghidra): normalize Mach-O imported symbol names 2026-10-11 09:59:27 +08:00
You Wishandmorluto 49e6f00ed0 fix(ghidra): recover returning import callers 2026-10-11 09:59:27 +08:00
af88184eee feat(ghidra): configurable import language and analysis seeds (#1667)
* feat(ghidra): configurable import language and analysis seeds

REA_GHIDRA_LANGUAGE_ID and REA_GHIDRA_COMPILER_SPEC_ID set the import
language when the header cannot identify the compiler (for example,
Borland C++ PE images).

REA_GHIDRA_SEED_FILE seeds functions, code and labels before
auto-analysis, for code reached only through computed jumps or data.

Both are validated before launch, recorded in the analysis profile and
verified in the bridge handshake.

* fix(ghidra): parse language XML and preserve seed ordering

* test(ghidra): use explicit platform paths for seed launch checks

* fix(javascript): retain cleanup failure reason in error causes

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 09:57:38 +08:00
c1efa6467d feat(process): bound a finalization interval before forced capture timeout
Add optional finalization_ms to keep observing a captured root after SIGTERM, record the actual finalizer exit and each signal attempt, and escalate once when needed. Bind the initial root identity to captured ownership and preserve observations through cleanup uncertainty.

Fixes #1253.

Co-authored-by: Mykola Dementiev <16836724+nikolay-dementiev@users.noreply.github.com>
Co-authored-by: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com>
2026-10-11 09:55:43 +08:00
morluto 1e51c732bb fix(javascript): preserve CommonJS assignment result and receiver uncertainty
Resolve values returned by simple assignment chains at the export boundary. Suppress property links proven to target a detached receiver; retain unresolved links when reassignment may preserve receiver identity. Identify root assignments from their AST target rather than the displayed export name.

Validation: 25 semantic tests, seven targeted compiled CLI/MCP scenarios with Node runtime controls, build, and check:fast passed. Existing full acceptance suite passed before receiver refinements; the final changes were checked with targeted scenarios rather than repeating its 130-second run.
2026-10-11 09:47:41 +08:00
morluto 7956934965 fix(javascript): preserve getter receiver mutation aliases
Resolve this, private fields and super reads through receiver-aware references. Separate instance fields from prototype descriptors, preserve alternative inheritance branches and allow finite repeated getter paths.

Add domain precision regressions and compiled CLI/MCP coverage for the four reported getter shapes.

Fixes #1665
2026-10-11 09:46:53 +08:00
morluto db4e360846 fix(browser)!: preserve WebSocket connections and comparable payload coverage
Retain source URLs, empty connections, and ordered frame streams. Compare same-URL streams without relying on transient request IDs, and compare selected shapes and byte counts only through common known coverage. Malformed binary frames retain known metadata with an unknown byte count.

BREAKING CHANGE: inspect_web_page network.websocket_events is replaced by network.websocket_connections, whose records contain request_id, url, and events. Frame payload_bytes is nullable when decoding cannot establish its size. No legacy alias is retained.

Validation: 78 domain/browser/MCP focused tests, check:fast, docs:check, and real Chrome 154 CDP capture passed. Real capture retained six connections and twelve frames with source URLs and selected shapes. Full browser verifier was interrupted during an unrelated slow scenario stage; its assertions were exercised by the bounded capture proof.
2026-10-11 09:43:25 +08:00
morluto 7bb4ac219c fix(ghidra): reject conflicting address aliases in batches 2026-10-11 09:35:22 +08:00
morluto 3df67c640a fix(ghidra): isolate optional nativeaot script sources 2026-10-11 09:35:22 +08:00
morluto 044aa34ce6 perf(native): index call-binding parameter and return nodes
Build ordered per-procedure indices from admitted nodes instead of scanning every node twice for each call binding. Preserve edge order, capacity limits, omitted-parameter unknowns, and surviving return bindings.

Validation: five focused service tests and check:fast passed. Four paired Flameox blocks at 4096 call sites passed all eight full-result digest oracles. Median paired reduction was 183.02 ms (71.4%) on this synthetic fan-out workload; real-provider performance is unverified.

Flameox evidence: 5a4c98ed7d224587c10a0f4596c8680d3a0330eb112a629014448ff1ea4634e7.
2026-10-11 09:33:36 +08:00
morluto 31bfff6fdf perf(domain): order JSON shapes through their structural tree
Traverse sorted siblings instead of comparing complete shared path prefixes. Return the typed producer result directly while preserving independent path segments and schema validation at unknown-input boundaries.

Validation: 12 domain/CDP boundary tests, build, and check:fast passed. Six paired Flameox blocks retained identical full-output digests across deep, mixed, Unicode, and wide controls. Depth-512 inference median fell from 10.78 ms to 1.51 ms (120 samples per variant); whole harness median fell from 518.6 ms to 327.5 ms.

Flameox evidence: f4a93424f34e5bfea254ada4ef278dc9bbb6c21047c977e20da130e11a8899e9. Timings describe this bounded workload only.
2026-10-11 09:30:05 +08:00
morluto 026126eafe fix(artifacts): serialize ASAR snapshot ownership
Serialize preparation and close for each reader so concurrent callers share one owned snapshot. Preserve per-call cancellation and expected digest checks, cleanup retry ownership, and reuse after close.

Validation: 34 ASAR boundary tests, one fork-isolated snapshot race regression, test discovery, and check:fast passed.
2026-10-11 09:28:28 +08:00
YsopwnableandGitHub 29cc1b8746 fix(javascript): exclude overwritten Electron API identities (#1664)
* fix(javascript): exclude overwritten Electron API identities

* test(javascript): bind Electron in file-base path fixture

* test(electron): bind BrowserWindow in preload override fixtures

* test(mcp): allow cold catalog validation in runtime probe

* test(electron): bind APIs in option reconstruction fixtures
2026-10-11 09:25:32 +08:00
LuisandGitHub ad41afa2b5 fix(setup): recognize Omarchy as an Arch-based host (#1662) 2026-10-11 09:23:39 +08:00
morluto 69a8b3a954 fix(ghidra): expose terminal-call boundary assumptions
Preserve observed terminal-call sites and callee flags across function views, raw bridge responses, Evidence, and snapshot replay. Version analysis profiles so older unqualified results cannot be reused.

Qualify Ghidra function extents without changing external no-return flags or claiming recovery of excluded code.

Validation: 76 focused tests, check:fast, docs:check, formatting, and the full Ghidra 12.1.4 host lane pass. Real CLI and all five stdio MCP function views retain the terminal-call evidence; returning puts calls remain unflagged. The reported stripped Rust ELF and Windows host remain unverified.

Fixes #1658
2026-10-11 09:08:01 +08:00
morluto 4733fc810a fix(ghidra): qualify defined-string search coverage
Carry defined-data scope through list and search Evidence and bind it into the analysis profile so older snapshots cannot replay unqualified results. Clarify the tool contract and document known-address byte/reference inspection.

Validation: 36 focused tests, check:fast, docs:check, and real Ghidra 12.1.4 CLI/stdio MCP on macOS arm64. The Rust Mach-O fixture contains the phrase and Ghidra defines it; the reported Linux ELF omission was not reproduced.

Fixes #1659
2026-10-11 08:51:02 +08:00
morluto d21132fc97 fix(ci): format Electron IPC verification assertion 2026-10-11 08:34:22 +08:00
morlutoandGitHub 66c344e289 ci: scope provider and native verification by changed inputs (#1660) 2026-10-11 08:31:27 +08:00
morluto 28bd0d990a test(electron): verify aliased APIs against a real IPC round trip
Extend the existing real Electron public-tool fixture with assignment aliases for BrowserWindow, contextBridge, ipcMain, and ipcRenderer. Assert exact static recovery counts and observe the handler invocation after a real click.

Verified the public CLI/stdio MCP fixture, IPC invocation, cancellation, and owned-process cleanup against checksum-verified Electron 43.3.0 on macOS ARM64. Build and check:fast pass. The separate readiness reload/crash scenario still fails on a destroyed execution context; this commit does not claim the full verify:electron lane passes.
2026-10-11 08:30:09 +08:00
morluto 2612e474b6 fix(javascript): recover Electron API identity through aliases
Reuse semantic module provenance for aliases and keep property path segments intact through Electron matching. Reject shadowed names and nested or dotted lookalikes. Reassigned and conditionally initialized module origins remain ambiguous, with their source origins retained.

Add a compiled CLI/stdio MCP regression for windows, preloads, context bridges, IPC operations, pairing, and source ranges. Validate 771 JavaScript domain tests, seven composition tests, the public acceptance workflow, build, and check:fast.

Fixes #1656
2026-10-11 08:27:46 +08:00
morluto a11fbc0df9 fix(ci): format JavaScript getter mutation regressions 2026-10-11 08:20:39 +08:00
morluto 639b73097b fix(javascript): preserve getter results on direct property writes
Replacing or deleting a class getter property does not mutate the object it would return. Apply the existing object-accessor write boundary to instance and static getters.

Validation: source mutation regressions and compiled CLI/stdio MCP acceptance preserve literal 1 for all four assignment/deletion cases; check:fast passes.
2026-10-11 08:15:47 +08:00
morluto bb074d84cb perf(javascript): reuse cycle sets during semantic evaluation
Track active bindings with per-evaluation push/pop instead of copying ancestor sets on every alias. Restore membership in finally so cycles and sibling branches keep their existing semantics.

Validation: 173 domain tests, compiled CLI/stdio MCP returned-reference acceptance, and check:fast pass. A randomized six-block Flameox experiment retained identical complete semantic output and reduced the bounded 112-alias analyzer median from 36.33 ms to 17.00 ms.
2026-10-11 08:14:45 +08:00
morluto 204e2c42a1 fix(ci): register Termux browser verifier as Knip entry point 2026-10-11 08:13:41 +08:00
tyloryangGitHubtyloryang <[邮箱]>morluto
87edf14940 fix(browser): support Termux Android scenarios (#1655)
* fix(browser): support Termux Android scenarios

* fix(build): lock Termux artifact generation

* ci(android): add manual Termux browser smoke verification

---------

Co-authored-by: tyloryang <[邮箱]>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 08:10:06 +08:00
YsopwnableandGitHub 890dfa49f6 fix(javascript): preserve cleanup failures during preparation cancellation (#1654) 2026-10-11 08:05:08 +08:00
morluto bc7846db91 fix(javascript): track references returned by getters
Follow object, instance, and static getter results when invalidating shared values. Respect descriptor replacement, known spread overrides, inheritance alternatives, and unresolved computed keys. Preserve conservative uncertainty for fresh getter results and verify shared mutations through CLI and MCP.

Fixes #1494
2026-10-11 08:03:17 +08:00
morluto 5485483f7d fix(javascript): preserve exact CommonJS export keys
Retain literal property keys as member segments, omit unresolved computed keys, and distinguish nested writes from dotted export names. Preserve empty keys and omit object-literal prototype setters. Verify bundler observations through Node, CLI, and MCP consumers.
2026-10-11 07:59:47 +08:00
morluto c916850852 fix(process): release terminals after observation failures
Keep queued observation failures available through frames while allowing the resource owner to release xterm and its owned addons. Catch write-callback failures so cleanup cannot hang, and retain actual disposal failures for retry.
2026-10-11 07:47:33 +08:00
morluto 26038912f6 fix(javascript): preserve cancellation during worker preparation 2026-10-11 07:36:39 +08:00
morluto aa7c5b7b5e fix(ghidra): reject address names owned by another symbol
Check function and label ownership in the existing namespace before renaming. Keep same-address renames idempotent and reject duplicate batch members inside their individual transactions. Extend real-provider namespace checks for single and batch collisions without conflating function and label fixture names.

Validation: real Ghidra 12.1.4 host-native verification passed on macOS ARM64 with JDK 21.0.12.1, including public CLI/stdIO MCP, qualified-name round trips, rollback, source-byte immutability, and complete cleanup. Windows mutation authority remains unchanged.

Fixes #1649
2026-10-11 07:35:48 +08:00
morluto 095fad9406 Merge remote-tracking branch 'origin/main' 2026-10-11 07:32:50 +08:00
morluto 8b2f8514c5 fix(javascript): ignore bare CommonJS exports rebinding
Assigning a new value to the exports alias does not replace module.exports. Stop inventing a default export link and definite return-shape changes for that assignment.

Validation: 22 semantic structure source tests and 49 compiled CLI/MCP acceptance tests passed. The new regression checks Node require returns an empty export object for both versions and both adapters report missing defaults without definite changes.
2026-10-11 07:30:52 +08:00
morluto c57e759ae6 fix(browser): bypass desktop keyrings in verification fixtures
Use password-store=basic for owned raw Chrome verification processes and document the same option for dedicated caller-owned Linux investigation profiles. Locked keyrings can stall navigation and leave placeholder frame URLs; keep the existing committed-URL authorization boundary.

Validation: real Chrome 154 CLI and stdio MCP browser verifier passed on macOS ARM64, plus docs freshness and check:fast. The locked Linux desktop keyring environment was not reproduced locally.
2026-10-11 07:30:25 +08:00
morluto 7b25a73ac4 fix(browser): bound CDP messages by the shared budget
Apply the 64 MiB protocol message limit when connect callers omit an override. Reject invalid explicit limits, including values that wrap through ws signed 32-bit coercion and disable enforcement. Preserve the typed payload_limit reason through disconnect and later commands.

Validation: 16 transport boundary tests passed, including UTF-8 exact/+1 limits and an oversized frame header without allocating its body. Real Chrome 154 CLI/MCP browser verification and check:fast passed. This caps message bytes, not decoded JSON heap size.
2026-10-11 07:30:14 +08:00
morluto aa257ea4b1 fix(browser): decode source-map response framing
Share the ECMA-426 JSON parse view across fetched maps and local point tracing while retaining exact response text, bytes, and digest. Handle BOM and HTTP newline variants without treating unterminated prefixes as valid JSON.

Validation: 89 source-map tests passed; source/runtime build and check:fast passed. Real Chrome 154 CLI and stdio MCP browser verification passed on macOS ARM64.
2026-10-11 07:29:51 +08:00
YsopwnableandGitHub 4999e4be2c fix(javascript): isolate source analysis and retain bounded partial evidence (#1651)
* refactor(javascript): validate transferable file projections

* test(javascript): consume decoded module and static facts

* fix(javascript): isolate source analysis and retain bounded partial evidence

* test(javascript): verify cleanup retention and repeated source paths
2026-10-11 07:22:50 +08:00
morluto 92cc0edd8e docs(setup): identify Qwen and OpenCode release gaps
Document the npm 6.3.0 boundary beside Qwen registration and OpenCode configuration override workflows. Registry metadata still identifies 6.3.0 as latest; source support requires publication and actual registry-package verification before issue closure.

Refs #1650.
2026-10-11 07:10:46 +08:00
morluto 573598c632 fix(artifacts): reject Windows stream paths before extraction
Validate Windows destination representability at the writer before registering or creating any entry. Preserve literal logical names in inventory and POSIX output. Refuse NTFS alternate stream syntax with typed path evidence and roll back prior extracted files.

Return producer path diagnostics directly, removing the presentation layer English-prefix classifier. Verify stream leaf/parent syntax, input stream cleanup, extraction rollback, inventory fidelity, POSIX materialization, and error projection: 63 passed, one platform-specific skip. Native Windows execution remains unverified here.

Fixes #1574. Adapted from and supersedes #1575.
2026-10-11 07:10:45 +08:00
morluto 8d3cfef753 fix(plist): disclose binary prototype-key omissions
Reuse the bounded binary preflight to observe dictionary keys before the pinned decoder drops them. Keyed archive results disclose the omission and Interface Builder marks archive decoding partial. XML keeps its existing exact counts; binary reporting deliberately leaves the count unknown because overwritten subtrees and decoder projection can change visibility.

Remove the superseded size-only preflight contract. Cover ASCII, UTF-16 and extended lengths, shared references, nested and shadowed dictionaries, and the value-only negative control. Synthetic plist and filesystem fixtures; no plutil or ibtool verification.

Fixes #1594. Supersedes #1593.
2026-10-11 07:02:39 +08:00
morluto 975dff2e25 fix(dotnet): bound RVA reads by section virtual size
Use the shared resolver for virtual containment, overlap checks, and file-backed extent. Raw alignment padding no longer supplies method bytes or shadows a later virtual section. Preserve malformed body coverage and suppress CIL hashes outside the admitted range.

Verify zero, exact-end, truncated-body, padding, and true-overlap cases through compiled CLI and MCP consumers. Synthetic PE fixtures only; no Windows loader or real managed-provider verification.

Fixes #1596. Supersedes #1595.
2026-10-11 07:02:16 +08:00
299 changed files with 26731 additions and 1913 deletions
@@ -0,0 +1,127 @@
---
name: rea-changelog-update
description: Prepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.
---
# REA changelog update
Use this for release-note preparation and contributor attribution. Read
[docs/releasing.md](../../../docs/releasing.md) from the repository root for
publication authority and the existing checkpoint checks. This skill prepares
notes; it does not authorize merging, tagging, or publishing.
## Inventory before prose
Fetch complete history and tags. Select the published baseline tag from
`.release-please-manifest.json` on the release PR's **base branch**, and pin that
branch's exact SHA as the editorial target. REA tags are `rea-agents-VERSION`.
Use the proposed version, including its prerelease suffix; do not change the
`always-bump-minor` strategy to imitate OpenClaw's calendar versions.
```bash
node scripts/release-notes.mjs inventory \
--repository morluto/rea --base rea-agents-PREVIOUS \
--target FULL_BASE_BRANCH_SHA --version VERSION \
--manifest /tmp/rea-VERSION.contributions.json
node scripts/release-notes.mjs record \
--manifest /tmp/rea-VERSION.contributions.json \
--output /tmp/rea-VERSION.record.md
```
Read the inventory, not just the existing bot draft. It includes side-branch
commits, GitHub-associated PRs whose merge commits are in the range, commit
authors/co-authors, and confirmed closing issues with their reporters. GitHub
connections are paginated. Network errors stop generation; rerun into a new
output file after recovery. Existing output files are never overwritten.
Unresolved commit identities are retained as names/emails with `user: null`;
never guess a handle or thank the merger as a substitute. The report preserves
unknowns. This is a reviewed metadata snapshot, not independent proof that a
human authored a patch. For unattributed identities, unusual merge strategies,
backports, reversions, or references that GitHub did not associate, inspect
the source PRs and diffs and report the limitation. The current collector uses
confirmed PR-closing relationships for issue-reporter credit; unrelated issue
mentions are not automatically credited. Do not claim that merely inventoried
work still ships after a revert.
## Write for users and retain credit
Rewrite only the newest `## [VERSION]` section in `CHANGELOG.md`, preserving its
version/date/link heading. Use single-line Markdown list items under:
- `### Highlights`: a few release-defining user outcomes; no quota for small releases.
- `### Changes`: capabilities and meaningful behavior changes.
- `### Fixes`: reliability, correctness, compatibility, and safety fixes.
- `### ⚠ BREAKING CHANGES`: retain when applicable, with migrations and original
PR/commit references required by the existing checkpoint validator.
Group related work by analyst workflow or user impact. Explain the resulting
behavior; internal cleanup, tests, CI, and routine documentation generally
belong only in the contribution record. Read ambiguous PRs/diffs before making
claims. Distinguish simulated-provider checks from real-provider verification.
Each bullet covering a PR carries its link and `Thanks @...` for its verified
human author, known co-authors, and confirmed closing-issue reporters. Grouping
must preserve **all** those credits in the same bullet. Include the issue
links. Direct-commit bullets carry the commit link and known human credit.
Exclude GitHub bots and known agent accounts. REA's observed Anthropic and
Cursor co-author trailers resolve to `@claude` and `@cursoragent` with GitHub
type `User`; the credit policy excludes them while retaining their metadata.
Do not copy OpenClaw's maintainer-specific exclusions.
Use explicit GitHub links for cross-repository references to avoid ambiguous
`#NNN` numbers. Unknown handles stay unknown; they are not fabricated credit.
Append the generated `### Complete contribution record` verbatim as the last
section of this release. It accounts for every discovered in-range PR and
direct contribution, including internal work omitted from the prose. Do not
hand-edit generated credits to make validation pass. Refresh the inventory
when product history changes; a prose-only correction can reuse the snapshot.
## Finalize with Release Please
Before changing the live release PR, have authorization for that PR update.
Local preparation and validation can proceed independently.
1. Add `rea:release-notes-finalized` to the selected open release PR **before**
editing its branch/body. Main proposal pushes respect this label. This is
our workflow guard, not Release Please's `autorelease: snooze` mechanism.
A proposal already running must finish before editing. Do not dispatch
manual preparation over finalized notes.
2. On that release branch, save the reviewed inventory as
`docs/releases/VERSION.contributions.json`. This is durable public release
provenance, not a temporary planning file. Do not add credentials or local
environment dumps. Keep the existing version artifacts aligned.
3. Run the existing checkpoint validator and the new checker:
```bash
node scripts/release-notes.mjs check \
--manifest docs/releases/VERSION.contributions.json
node scripts/release-notes.mjs render \
--manifest docs/releases/VERSION.contributions.json \
--output /tmp/rea-VERSION.notes.md
```
The checker supports inline PR bodies, including their surrounding metadata,
up to GitHub's 65,536-character limit. It fails on larger notes rather than
truncating credit; Release Please's overflow-link bodies are not integrated.
4. Update the release PR body with the rendered section while preserving its
Release Please header/footer and component/version structure. Use a body
file for `gh pr edit --body-file`; changing only `CHANGELOG.md` does not
change Release Please's release body. Push only authorized changes.
5. Freshly fetch the PR body to a file and run `check --pr-body FILE`. Review
the complete diff and wait for current CI. Publication checks the body
again before release creation. Avoid further body edits while publishing.
The finalization label freezes bot refreshes, not main. New product changes
after the inventory target require refreshing the notes before publication.
The workflow checks target ancestry and allows only release metadata paths
after that target. It also requires the inventory target to equal the reviewed
merge's first parent, so intervening main commits require a fresh inventory.
Review diffs within the allowed release metadata paths too: the path guard does
not prove that package dependency/script edits are version-only changes.
To resume automatic proposals, remove the finalization label and let the bot
regenerate; expect editorial changes to be replaced.
Existing releases without a contribution inventory retain their current
publication path. Never retag or republish a package to correct release prose.
@@ -38,3 +38,11 @@ or absence:
Process captures are opt-in behavioral evidence, not a security sandbox. V3 Process captures are opt-in behavioral evidence, not a security sandbox. V3
captures cannot be upgraded to V4; rerun the original scenario. Distinguish root captures cannot be upgraded to V4; rerun the original scenario. Distinguish root
exit from descendant settlement and require freshness when the task needs it. exit from descendant settlement and require freshness when the task needs it.
A timed-out target that must write a final report needs `finalization_ms`: REA
sends SIGTERM to the captured root, keeps observing, and sends SIGKILL once
only if the target has not exited by the end of the interval.
`exit.finalization.signals` lists each attempt with its delivery result; the
observed exit signal stays in `exit.signal`, `exit.code` is null for every
deadline exit, and the code of a finalizer that exited on its own is in
`exit.finalization.exit_code` (null on a signal exit or when no exit was
observed).
@@ -1,5 +1,13 @@
# JavaScript and Electron application artifacts # JavaScript and Electron application artifacts
For captured or bundled JavaScript that needs readable source recovery, use
`recover_javascript_sources` with an absolute `path` and absent
`output_directory`. Supply related runtime/shared/lazy files in
`additional_paths` when advertised, then analyze the returned `analysis_input`
directory together. Each input retains its own snapshot, digest and byte-range
provenance; inputs and recovered code are never executed. Recovery requires a
caller-supplied Wakaru on Linux x64; see [the recovery guide](https://github.com/morluto/rea/blob/main/docs/javascript-recovery.md).
Use `analyze_javascript_application` directly on the operator-supplied ASAR or Use `analyze_javascript_application` directly on the operator-supplied ASAR or
extracted tree. When size is unknown or large and the connected schema advertises extracted tree. When size is unknown or large and the connected schema advertises
`detail`, request `detail: "summary"`. This returns an `inspect_analysis_view` `detail`, request `detail: "summary"`. This returns an `inspect_analysis_view`
@@ -9,6 +17,22 @@ that static analysis. Read findings, coverage, limitations and unknowns, then
follow a specific unanswered question. Older servers without `detail` return follow a specific unanswered question. Older servers without `detail` return
the complete application Evidence inline when it fits. the complete application Evidence inline when it fits.
When the connected schema advertises `max_heap_mb` and `analysis_timeout_ms`,
these select the isolated source analyzer's old-space heap and per-source
deadline. Defaults are 1024 MiB and 300000 ms. They do not change the MCP
process heap. Resource-limited analyses preserve completed static facts and
report partial semantic coverage with specific unknown scopes; this does not
mean the source is malformed. Inspect those limits before retrying a source.
When expanded relationships exceed the parent's available memory, inspect the
file's modules item view for retained `static_analysis` and
`semantic_module_analysis`. Their facts remain available even when graph links
are unknown. Cumulative result exhaustion can stop analysis before every file
is examined; the inventory and remaining unknown scopes distinguish that case.
Timeout or cancellation after source analysis begins retains partial Evidence
in the same session. A returned `error.details.partial_observation` reference
can be supplied directly to `inspect_analysis_view`. If the SDK cancels locally, retrieve its recorded ID from
the Evidence bundle before inspecting the partial analysis.
## Read a module without repeating analysis ## Read a module without repeating analysis
Use the actual returned parent ID, on the same connection, as `source.evidence_id`. Use the actual returned parent ID, on the same connection, as `source.evidence_id`.
+6
View File
@@ -0,0 +1,6 @@
self-hosted-runner:
labels:
- hopper
- ghidra-12-1-2
# Hosted label introduced after the pinned actionlint release.
- macos-15-intel
+145 -50
View File
@@ -5,6 +5,10 @@ on:
branches: [main] branches: [main]
pull_request: pull_request:
branches: [main, "release/*"] branches: [main, "release/*"]
workflow_dispatch:
permissions:
contents: read
concurrency: concurrency:
group: ci-${{ github.event.pull_request.number || github.ref }} group: ci-${{ github.event.pull_request.number || github.ref }}
@@ -36,37 +40,39 @@ jobs:
name: Classify changes name: Classify changes
runs-on: ubuntu-latest runs-on: ubuntu-latest
outputs: outputs:
code: ${{ steps.package.outputs.required }} code: ${{ steps.plan.outputs.code }}
package: ${{ steps.package.outputs.required }} runtime: ${{ steps.plan.outputs.runtime }}
package: ${{ steps.plan.outputs.package }}
apple: ${{ steps.plan.outputs.apple }}
fixtures: ${{ steps.plan.outputs.fixtures }}
readiness: ${{ steps.plan.outputs.readiness }}
docs: ${{ steps.plan.outputs.docs }}
workflows: ${{ steps.plan.outputs.workflows }}
browser: ${{ steps.plan.outputs.browser }}
binary_layout: ${{ steps.plan.outputs.binary_layout }}
evm: ${{ steps.plan.outputs.evm }}
javascript_recovery: ${{ steps.plan.outputs.javascript_recovery }}
recorded_crash: ${{ steps.plan.outputs.recorded_crash }}
web_network: ${{ steps.plan.outputs.web_network }}
web_runtime: ${{ steps.plan.outputs.web_runtime }}
web_source_map: ${{ steps.plan.outputs.web_source_map }}
nativeaot: ${{ steps.plan.outputs.nativeaot }}
website: ${{ steps.plan.outputs.website }}
matrix: ${{ steps.plan.outputs.matrix }}
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Select package and Windows lanes - uses: actions/setup-node@v4
id: package with:
shell: bash node-version-file: .nvmrc
- name: Select verification lanes
id: plan
env: env:
BASE_SHA: ${{ github.event.pull_request.base.sha }} BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }} HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: | CI_FULL: ${{ contains(github.event.pull_request.labels.*.name, 'ci:full') }}
if [[ "${GITHUB_EVENT_NAME}" != "pull_request" ]]; then run: node scripts/ci/plan.mjs
echo "required=true" >> "${GITHUB_OUTPUT}"
# Compare the PR's changes, excluding unrelated updates on its base.
elif git diff --quiet "${BASE_SHA}...${HEAD_SHA}" -- \
. \
':(exclude)docs/**' \
':(top,exclude,glob)README*.md' \
':(exclude)AGENTS.md' \
':(exclude)CONTRIBUTING.md'; then
echo "required=false" >> "${GITHUB_OUTPUT}"
else
diff_status=$?
# A failed comparison must fail classification, not report a scope.
if [[ "${diff_status}" -ne 1 ]]; then
exit "${diff_status}"
fi
echo "required=true" >> "${GITHUB_OUTPUT}"
fi
static: static:
name: Static checks name: Static checks
@@ -84,17 +90,17 @@ jobs:
node-version-file: .nvmrc node-version-file: .nvmrc
cache: npm cache: npm
- run: npm ci - run: npm ci
- if: github.event_name == 'push' || needs.changes.outputs.code == 'true' - if: needs.changes.outputs.code == 'true'
run: npm run check:ci run: npm run check:ci
- if: github.event_name == 'push' || needs.changes.outputs.code == 'true' - if: needs.changes.outputs.code == 'true'
run: npm run verify:test-discovery run: npm run verify:test-discovery
- if: github.event_name == 'pull_request' && needs.changes.outputs.code != 'true' - if: needs.changes.outputs.code != 'true'
run: npm run format:check run: npm run format:check
test-shard: test-shard:
name: Test shard (${{ matrix.shard }}) name: Test shard (${{ matrix.shard }})
needs: [changes, build] needs: [changes, build]
if: github.event_name == 'push' || needs.changes.outputs.code == 'true' if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-latest runs-on: ubuntu-latest
strategy: strategy:
fail-fast: false fail-fast: false
@@ -131,7 +137,7 @@ jobs:
build: build:
name: Build package name: Build package
needs: changes needs: changes
if: github.event_name == 'push' || needs.changes.outputs.code == 'true' if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -224,7 +230,7 @@ jobs:
conformance-fixtures: conformance-fixtures:
name: Source-built conformance fixtures name: Source-built conformance fixtures
needs: changes needs: changes
if: github.event_name == 'push' || needs.changes.outputs.code == 'true' if: needs.changes.outputs.fixtures == 'true'
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 10 timeout-minutes: 10
steps: steps:
@@ -235,7 +241,7 @@ jobs:
windows-native-build: windows-native-build:
name: Build bundled Windows x64 native controls name: Build bundled Windows x64 native controls
needs: changes needs: changes
if: github.event_name == 'push' || needs.changes.outputs.package == 'true' if: needs.changes.outputs.runtime == 'true'
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 10 timeout-minutes: 10
steps: steps:
@@ -265,24 +271,11 @@ jobs:
package-e2e: package-e2e:
name: Native package baseline (${{ matrix.platform }} ${{ matrix.architecture }}) name: Native package baseline (${{ matrix.platform }} ${{ matrix.architecture }})
needs: [changes, windows-native-build] needs: [changes, windows-native-build]
if: github.event_name == 'push' || needs.changes.outputs.package == 'true' if: needs.changes.outputs.runtime == 'true'
strategy: strategy:
fail-fast: false fail-fast: false
max-parallel: 2 max-parallel: 2
matrix: matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}
include:
- os: ubuntu-latest
platform: linux
architecture: x64
- os: ubuntu-24.04-arm
platform: linux
architecture: arm64
- os: macos-15
platform: darwin
architecture: arm64
- os: macos-15-intel
platform: darwin
architecture: x64
runs-on: ${{ matrix.os }} runs-on: ${{ matrix.os }}
timeout-minutes: 20 timeout-minutes: 20
env: env:
@@ -317,7 +310,7 @@ jobs:
apple-artifacts: apple-artifacts:
name: Real Apple artifact CLI/MCP E2E name: Real Apple artifact CLI/MCP E2E
needs: changes needs: changes
if: github.event_name == 'push' || needs.changes.outputs.code == 'true' if: needs.changes.outputs.apple == 'true'
runs-on: macos-15 runs-on: macos-15
timeout-minutes: 15 timeout-minutes: 15
steps: steps:
@@ -350,7 +343,7 @@ jobs:
reconstruction-readiness: reconstruction-readiness:
name: Reconstruction readiness name: Reconstruction readiness
needs: changes needs: changes
if: github.event_name == 'push' || needs.changes.outputs.code == 'true' if: needs.changes.outputs.readiness == 'true'
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 15 timeout-minutes: 15
steps: steps:
@@ -379,7 +372,7 @@ jobs:
windows-curated: windows-curated:
name: Windows capability and package lane name: Windows capability and package lane
needs: [changes, windows-native-build] needs: [changes, windows-native-build]
if: github.event_name == 'push' || needs.changes.outputs.package == 'true' if: needs.changes.outputs.package == 'true'
runs-on: windows-latest runs-on: windows-latest
timeout-minutes: 25 timeout-minutes: 25
steps: steps:
@@ -476,7 +469,8 @@ jobs:
docs: docs:
name: Docs Generation name: Docs Generation
if: github.event_name == 'pull_request' needs: changes
if: needs.changes.outputs.docs == 'true'
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -496,3 +490,104 @@ jobs:
docs/verification/managed-conformance-*.json docs/verification/managed-conformance-*.json
skills/reverse-engineer-anything skills/reverse-engineer-anything
if-no-files-found: error if-no-files-found: error
workflows:
name: Validate workflows
needs: changes
if: needs.changes.outputs.workflows == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Validate workflow syntax and expressions
run: go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 -shellcheck= -pyflakes=
browser:
needs: changes
if: needs.changes.outputs.browser == 'true'
uses: ./.github/workflows/real-browser.yml
binary_layout:
needs: changes
if: needs.changes.outputs.binary_layout == 'true'
uses: ./.github/workflows/real-binary-layout.yml
evm:
needs: changes
if: needs.changes.outputs.evm == 'true'
uses: ./.github/workflows/real-evm-interface.yml
javascript_recovery:
needs: changes
if: needs.changes.outputs.javascript_recovery == 'true'
uses: ./.github/workflows/real-javascript-recovery.yml
recorded_crash:
needs: changes
if: needs.changes.outputs.recorded_crash == 'true'
uses: ./.github/workflows/real-recorded-crash.yml
web_network:
needs: changes
if: needs.changes.outputs.web_network == 'true'
uses: ./.github/workflows/real-web-network-captures.yml
web_runtime:
needs: changes
if: needs.changes.outputs.web_runtime == 'true'
uses: ./.github/workflows/real-web-runtime.yml
web_source_map:
needs: changes
if: needs.changes.outputs.web_source_map == 'true'
uses: ./.github/workflows/real-web-source-map.yml
nativeaot:
needs: changes
if: needs.changes.outputs.nativeaot == 'true'
uses: ./.github/workflows/nativeaot-fixtures.yml
website:
needs: changes
if: needs.changes.outputs.website == 'true'
uses: ./.github/workflows/website-check.yml
required:
name: CI required
if: ${{ always() }}
needs:
[
changes,
pr-title,
static,
build,
test-shard,
test,
conformance-fixtures,
windows-native-build,
package-e2e,
apple-artifacts,
reconstruction-readiness,
windows-curated,
docs,
workflows,
browser,
binary_layout,
evm,
javascript_recovery,
recorded_crash,
web_network,
web_runtime,
web_source_map,
nativeaot,
website,
]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version-file: .nvmrc
- name: Require every selected lane to pass
env:
CI_JOBS: ${{ toJSON(needs) }}
run: node scripts/ci/required.mjs
+1 -6
View File
@@ -2,12 +2,7 @@ name: Source-built NativeAOT fixtures
on: on:
workflow_dispatch: workflow_dispatch:
pull_request: workflow_call:
paths:
- "bridge/ghidra/extensions/nativeaot/**"
- "scripts/generate-nativeaot-fixtures.mjs"
- "tests/fixtures/nativeaot/**"
- ".github/workflows/nativeaot-fixtures.yml"
permissions: permissions:
contents: read contents: read
+1 -30
View File
@@ -7,36 +7,7 @@ on:
description: "Also verify the complete 65,281-section result over MCP (slow)" description: "Also verify the complete 65,281-section result over MCP (slow)"
type: boolean type: boolean
default: false default: false
pull_request: workflow_call:
paths:
- ".github/workflows/real-binary-layout.yml"
- "bridge/pwntools/**"
- "src/native/pwntools/**"
- "src/domain/native/binaryLayout*"
- "src/application/binaryDiagnostics/**"
- "src/contracts/native/binaryDiagnosticsToolContracts.ts"
- "src/composition/binaryDiagnostics.ts"
- "src/server/registerBinaryDiagnosticsTools.ts"
- "src/server/createServer.ts"
- "src/cli/binaryDiagnosticsCommands.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/process/OwnedCommand.ts"
- "src/process/ProviderProcess.ts"
- "src/process/ProviderDeadline.ts"
- "src/process/ProcessOwnership*.ts"
- "src/process/PrivateRuntimeRoot.ts"
- "src/artifacts/readStableArtifact*.ts"
- "src/artifacts/ArtifactReader.ts"
- "src/domain/analysisError*.ts"
- "scripts/verify-binary-layout.mjs"
- "scripts/lib/elf-section-name-fixtures.mjs"
- "scripts/lib/large-result-mcp.mjs"
- "scripts/fixtures/binary-layout*"
- "package.json"
- "package-lock.json"
permissions: permissions:
contents: read contents: read
+1 -34
View File
@@ -1,40 +1,7 @@
name: Real browser verification name: Real browser verification
on: on:
pull_request: workflow_call:
paths:
- ".github/workflows/real-browser.yml"
- "scripts/verify-real-browser.mjs"
- "scripts/verify-browser-network.mjs"
- "scripts/verify-browser-scripts.mjs"
- "scripts/verify-browser-modules.mjs"
- "scripts/verify-browser-dom-destinations.mjs"
- "scripts/lib/browser-*.mjs"
- "scripts/fixtures/browser-network-site.mjs"
- "scripts/fixtures/browser-script-site.mjs"
- "scripts/fixtures/browser-module-site.mjs"
- "scripts/fixtures/browser-verifier-site.mjs"
- "scripts/fixtures/page-cdp-proxy.mjs"
- "src/browser/**"
- "src/domain/browser*.ts"
- "src/domain/webScript*.ts"
- "src/domain/webModule*.ts"
- "src/application/WebModule*.ts"
- "src/composition/webModules.ts"
- "src/artifacts/readStableArtifact.ts"
- "src/artifacts/ArtifactPaths.ts"
- "src/application/WebScriptExportService.ts"
- "src/contracts/webScriptToolContracts.ts"
- "src/server/registerWebScriptTool.ts"
- "src/server/registerWebModuleTool.ts"
- "src/server/createServer.ts"
- "src/cli/webModuleCommands.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolEffects.ts"
- "src/cli/webScriptCommands.ts"
- "package.json"
- "package-lock.json"
workflow_dispatch: workflow_dispatch:
concurrency: concurrency:
+1 -27
View File
@@ -2,33 +2,7 @@ name: Real offline EVM interface
on: on:
workflow_dispatch: workflow_dispatch:
pull_request: workflow_call:
paths:
- ".github/workflows/real-evm-interface.yml"
- "src/evm/**"
- "src/domain/evm/**"
- "src/application/evm/**"
- "src/contracts/evm/**"
- "src/composition/evm.ts"
- "src/cli/evmCommands.ts"
- "src/server/registerEvmTools.ts"
- "src/server/createServer.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/process/OwnedCommand.ts"
- "src/process/ProviderProcess.ts"
- "src/process/ProcessOwnership*.ts"
- "src/process/ProviderDeadline.ts"
- "src/process/PrivateRuntimeRoot.ts"
- "src/artifacts/readStableArtifact*.ts"
- "src/artifacts/ArtifactReader.ts"
- "scripts/verify-evm-interface.mjs"
- "scripts/fixtures/evm-interface.sol"
- "third_party/evmole/**"
- "package.json"
- "package-lock.json"
permissions: permissions:
contents: read contents: read
+1 -26
View File
@@ -1,32 +1,7 @@
name: Real JavaScript recovery name: Real JavaScript recovery
on: on:
pull_request: workflow_call:
paths:
- ".github/workflows/real-javascript-recovery.yml"
- "src/javascript/recovery/**"
- "src/domain/javascript/**"
- "src/application/javascript/JavaScriptRecovery*.ts"
- "src/composition/javascriptRecovery.ts"
- "src/contracts/javascript/javascriptRecoveryToolContracts.ts"
- "src/cli/javascriptRecoveryCommands.ts"
- "src/cli/javascriptApplicationAnalysis.ts"
- "src/server/registerJavaScriptRecoveryTool.ts"
- "src/server/createServer.ts"
- "src/server/sessionAvailabilityPolicy.ts"
- "src/application/CapabilityInventory.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolContracts.ts"
- "src/contracts/toolEffects.ts"
- "scripts/rea.mjs"
- "src/process/**"
- "src/artifacts/SafeOutputTree.ts"
- "scripts/verify/javascript/recovery.mjs"
- "scripts/fixtures/javascript-recovery/**"
- "third_party/wakaru"
- "package.json"
- "package-lock.json"
workflow_dispatch: workflow_dispatch:
permissions: permissions:
+1 -28
View File
@@ -2,34 +2,7 @@ name: Real recorded crash evidence
on: on:
workflow_dispatch: workflow_dispatch:
pull_request: workflow_call:
paths:
- ".github/workflows/real-recorded-crash.yml"
- "bridge/pwntools/**"
- "bridge/pwndbg/**"
- "src/native/pwntools/**"
- "src/native/pwndbg/**"
- "src/domain/native/recordedCrash*"
- "src/application/binaryDiagnostics/**"
- "src/contracts/native/binaryDiagnosticsToolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/contracts/toolOutputSchemaGroups.ts"
- "src/composition/binaryDiagnostics.ts"
- "src/server/registerRecordedCrashTools.ts"
- "src/server/createServer.ts"
- "src/server/sessionAvailabilityPolicy.ts"
- "src/cli/binaryDiagnosticsCommands.ts"
- "src/cliCommandNames.ts"
- "src/process/**"
- "src/artifacts/readStableArtifact*.ts"
- "src/domain/analysisError*.ts"
- "scripts/verify-recorded-crash.mjs"
- "scripts/verify/recorded-crash/**"
- "scripts/lib/recorded-core-fixtures.py"
- "scripts/fixtures/recorded-crash.c"
- "third_party/pwndbg/**"
- "package.json"
- "package-lock.json"
permissions: permissions:
contents: read contents: read
+57
View File
@@ -0,0 +1,57 @@
name: Real Termux browser verification
on:
workflow_dispatch:
concurrency:
group: real-termux-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
env:
HUSKY: "0"
jobs:
real-termux:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- name: Download the pinned Termux app and verify its release checksum
shell: bash
run: |
mkdir -p "$RUNNER_TEMP/rea-termux"
cd "$RUNNER_TEMP/rea-termux"
release=https://github.com/termux/termux-app/releases/download/v0.118.3
apk=termux-app_v0.118.3+github-debug_x86_64.apk
sums=termux-app_v0.118.3+github-debug_sha256sums
curl --fail --location --retry 3 --output "$apk" "$release/$apk"
curl --fail --location --retry 3 --output "$sums" "$release/$sums"
awk -v apk="$apk" '$2 == apk || $2 == "*" apk' "$sums" > selected.sha256
test "$(wc -l < selected.sha256)" -eq 1
sha256sum --check selected.sha256
- name: Enable emulator acceleration
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm
- name: Build and capture a page inside Termux
uses: reactivecircus/android-emulator-runner@4c44018e59b437e86cdfc41da381398f93ed8808 # v2
with:
api-level: 30
arch: x86_64
target: google_apis
ram-size: 4096M
heap-size: 512M
disk-size: 8192M
disable-animations: true
script: bash scripts/verify/termux/emulator.sh
- name: Upload Termux and Android diagnostics
if: always()
uses: actions/upload-artifact@v4
with:
name: termux-browser-diagnostics
path: ${{ runner.temp }}/rea-termux/diagnostics
if-no-files-found: warn
@@ -2,38 +2,7 @@ name: Real historical web network captures
on: on:
workflow_dispatch: workflow_dispatch:
pull_request: workflow_call:
paths:
- ".github/workflows/real-web-network-captures.yml"
- "bridge/mitmproxy/**"
- "src/browser/history/**"
- "src/domain/webNetworkCapture*.ts"
- "src/domain/analysisError*.ts"
- "src/contracts/errorSchemas.ts"
- "src/domain/explicitSensitive*.ts"
- "src/application/WebNetworkCapture*"
- "src/composition/webNetworkCaptures.ts"
- "src/cli/webNetworkCaptureCommands.ts"
- "src/server/registerWebNetworkCaptureTool.ts"
- "src/server/createServer.ts"
- "src/server/toolRegistrationOptions.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/process/OwnedCommand.ts"
- "src/process/ProviderProcess.ts"
- "src/process/ProviderDeadline.ts"
- "src/process/ProcessOwnership*.ts"
- "src/process/PrivateRuntimeRoot.ts"
- "src/process/capture/ProcessCaptureLifecycle.ts"
- "src/artifacts/readStableArtifact*.ts"
- "src/artifacts/ArtifactReader.ts"
- "src/contracts/webNetworkCaptureToolContracts.ts"
- "scripts/verify-web-network-captures.mjs"
- "scripts/fixtures/generate-mitmproxy-capture.py"
- "package.json"
- "package-lock.json"
permissions: permissions:
contents: read contents: read
+1 -29
View File
@@ -1,34 +1,6 @@
name: Real website runtime attribution name: Real website runtime attribution
on: on:
pull_request: workflow_call:
paths:
- ".github/workflows/real-web-runtime.yml"
- "scripts/verify-browser-runtime.mjs"
- "scripts/lib/browser-runtime-*.mjs"
- "src/process/ProviderProcess.ts"
- "src/process/ProcessOwnership*.ts"
- "scripts/fixtures/browser-runtime-site.mjs"
- "src/browser/execution/**"
- "src/browser/CdpConnection.ts"
- "src/browser/CdpEndpoint.ts"
- "src/browser/CdpTarget*.ts"
- "src/browser/CdpCaptureValues.ts"
- "src/browser/CdpAuthorized*.ts"
- "src/domain/webRuntime.ts"
- "src/domain/webExecution.ts"
- "src/domain/webEventListeners.ts"
- "src/domain/browserObservationError*.ts"
- "src/application/WebRuntime*.ts"
- "src/composition/webRuntime.ts"
- "src/contracts/webRuntimeToolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/server/registerWebRuntimeTools.ts"
- "src/server/createServer.ts"
- "src/cli/webRuntimeCommands.ts"
- "src/cli/commandCancellation.ts"
- "src/cli.ts"
- "package.json"
- "package-lock.json"
workflow_dispatch: workflow_dispatch:
permissions: permissions:
contents: read contents: read
+1 -22
View File
@@ -1,27 +1,6 @@
name: Real website source-map verification name: Real website source-map verification
on: on:
pull_request: workflow_call:
paths:
- ".github/workflows/real-web-source-map.yml"
- "scripts/verify-browser-source-maps.mjs"
- "scripts/lib/browser-source-map-e2e.mjs"
- "scripts/fixtures/browser-source-map-site.mjs"
- "src/javascript/sourceMaps/**"
- "src/browser/assets/**"
- "src/domain/webSource*.ts"
- "src/domain/webScript*.ts"
- "src/application/WebSourceLocation*.ts"
- "src/application/ValidateSelectedWebScript.ts"
- "src/composition/webSourceLocations.ts"
- "src/server/registerWebSourceLocationTool.ts"
- "src/server/createServer.ts"
- "src/contracts/webScriptToolContracts.ts"
- "src/process/**"
- "src/cli/webSourceLocationCommands.ts"
- "src/cli.ts"
- "src/artifacts/readStableArtifact.ts"
- "package.json"
- "package-lock.json"
workflow_dispatch: workflow_dispatch:
permissions: permissions:
contents: read contents: read
+34
View File
@@ -35,7 +35,21 @@ jobs:
contents: write contents: write
pull-requests: write pull-requests: write
steps: steps:
- name: Respect finalized release notes
id: editorial
env:
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
run: |
finalized=$(gh api --paginate "repos/${GH_REPO}/pulls?state=open&base=main&per_page=100" \
--jq '.[] | select(.head.repo.full_name == .base.repo.full_name) | select(.head.ref | startswith("release-please--branches--main--")) | select(any(.labels[]; .name == "rea:release-notes-finalized")) | .number')
if [[ -n "${finalized}" ]]; then
echo "Finalized release notes on PR ${finalized}; automatic proposal refresh is paused." >> "$GITHUB_STEP_SUMMARY"
echo "frozen=true" >> "$GITHUB_OUTPUT"
fi
- uses: googleapis/release-please-action@v4 - uses: googleapis/release-please-action@v4
if: steps.editorial.outputs.frozen != 'true'
with: with:
token: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }} token: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
target-branch: main target-branch: main
@@ -80,6 +94,16 @@ jobs:
node scripts/verify-release-checkpoint.mjs \ node scripts/verify-release-checkpoint.mjs \
--phase publish --release-branch main --source-sha "$SOURCE_SHA" --phase publish --release-branch main --source-sha "$SOURCE_SHA"
- name: Validate finalized notes before release creation
env:
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
RELEASE_PR: ${{ github.event.pull_request.number }}
SOURCE_SHA: ${{ github.event.pull_request.merge_commit_sha }}
run: |
node scripts/release-notes.mjs publication --repository "$GH_REPO" \
--source "$SOURCE_SHA" --pr "$RELEASE_PR"
- uses: googleapis/release-please-action@v4 - uses: googleapis/release-please-action@v4
id: release id: release
with: with:
@@ -192,6 +216,16 @@ jobs:
--phase "$RELEASE_PHASE" --release-branch "$RELEASE_BRANCH" --phase "$RELEASE_PHASE" --release-branch "$RELEASE_BRANCH"
--source-sha "$SOURCE_SHA" --source-sha "$SOURCE_SHA"
- name: Validate manual finalized notes before release creation
if: inputs.phase == 'publish'
env:
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
SOURCE_SHA: ${{ steps.branch.outputs.sha }}
run: |
node scripts/release-notes.mjs publication --repository "$GH_REPO" \
--source "$SOURCE_SHA"
- uses: googleapis/release-please-action@v4 - uses: googleapis/release-please-action@v4
id: release id: release
with: with:
+1 -6
View File
@@ -1,12 +1,7 @@
name: Website checks name: Website checks
on: on:
pull_request: workflow_call:
branches: [main, "release/*"]
paths:
- "website/**"
- "scripts/*website*.py"
- ".github/workflows/**"
workflow_dispatch: workflow_dispatch:
permissions: permissions:
+5
View File
@@ -148,10 +148,15 @@ Generated fixtures and manifests remain ignored.
Use the [release guide](docs/releasing.md). Main pushes automatically refresh a Use the [release guide](docs/releasing.md). Main pushes automatically refresh a
Release Please PR with the next version and changelog. Review its migration Release Please PR with the next version and changelog. Review its migration
notes, wait for the current CI checks to pass, and merge it into main to publish. notes, wait for the current CI checks to pass, and merge it into main to publish.
The Release workflow validates and tags that merge; both publishers build its The Release workflow validates and tags that merge; both publishers build its
exact SHA. Ordinary main pushes cannot publish packages. An optional manual exact SHA. Ordinary main pushes cannot publish packages. An optional manual
`release/VERSION` checkpoint retains explicit preparation and publication. `release/VERSION` checkpoint retains explicit preparation and publication.
For grouped release notes with verified contributor thanks, use the repository's
`rea-changelog-update` skill in `.agents/skills/rea-changelog-update/`. Its inventory,
credit checks, and finalization process are described in the release guide.
Keep new implementation commits on main for the next release. The workflow Keep new implementation commits on main for the next release. The workflow
owns packaged-artifact verification, npm publication, the published CLI/MCP owns packaged-artifact verification, npm publication, the published CLI/MCP
canary, and MCP Registry publication. Sync manual checkpoint metadata back to canary, and MCP Registry publication. Sync manual checkpoint metadata back to
-1
View File
@@ -9,7 +9,6 @@
**See a feature you like. Understand how it works, down to the binary level.** **See a feature you like. Understand how it works, down to the binary level.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents) [![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog) [![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#what-you-can-analyze) [![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#what-you-can-analyze)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything) [![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
+151
View File
@@ -0,0 +1,151 @@
import ghidra.app.script.GhidraScript;
import ghidra.program.model.address.Address;
import ghidra.program.model.address.AddressSpace;
import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.Program;
import ghidra.program.model.mem.MemoryBlock;
import ghidra.program.model.symbol.SourceType;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
/**
* Apply caller-declared analysis seeds before default auto-analysis.
*
* The single argument is REA's canonical seed snapshot: one
* {@code 0x<hex>\t(function|code|label)[\tname]} line per seed. Code seeds are
* applied after functions (highest address first), then labels. The outcome is stored as a
* program property that the bridge reports during its handshake.
*/
public final class ReaGhidraApplySeeds extends GhidraScript {
static final String REPORT_OPTION = "REA analysis seeds";
private record Seed(Address address, String kind, String name) {}
@Override
public void run() throws Exception {
String[] args = getScriptArgs();
if (args.length != 1) {
throw new IllegalArgumentException("REA seed script requires exactly one seed file");
}
Path path = Path.of(args[0]);
if (!Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS)) {
throw new IllegalArgumentException("REA seed file is not a regular file");
}
byte[] bytes = Files.readAllBytes(path);
String sha256 = HexFormat.of().formatHex(
MessageDigest.getInstance("SHA-256").digest(bytes)
);
AddressSpace space = currentProgram.getAddressFactory().getDefaultAddressSpace();
List<Seed> seeds = new ArrayList<>();
for (String line : new String(bytes, StandardCharsets.UTF_8).split("\n")) {
if (line.isEmpty()) continue;
String[] fields = line.split("\t", -1);
if (fields.length < 2 || fields.length > 3 || !fields[0].startsWith("0x")) {
throw new IllegalArgumentException("REA seed snapshot is malformed");
}
Address address;
try {
address = space.getAddress(Long.parseUnsignedLong(fields[0].substring(2), 16));
} catch (ghidra.program.model.address.AddressOutOfBoundsException e) {
address = null;
}
seeds.add(new Seed(address, fields[1], fields.length == 3 ? fields[2] : null));
}
// Highest address first: a function body computed before analysis stops at
// an existing function entry, so later creations cannot swallow earlier ones.
seeds.sort((left, right) -> {
if (left.address() == null) return right.address() == null ? 0 : 1;
if (right.address() == null) return -1;
return right.address().compareTo(left.address());
});
int unmapped = 0;
int logged = 0;
int codeDecoded = 0, codeFailed = 0;
int functionCreated = 0, functionExisting = 0, functionFailed = 0;
int labelApplied = 0, labelFailed = 0;
for (String pass : new String[] { "function", "code", "label" }) {
for (Seed seed : seeds) {
if (!seed.kind().equals(pass)) continue;
monitor.checkCancelled();
MemoryBlock block = seed.address() == null
? null
: currentProgram.getMemory().getBlock(seed.address());
if (block == null || (!pass.equals("label") && !block.isInitialized())) {
unmapped++;
continue;
}
switch (pass) {
case "code" -> {
if (decoded(seed.address())) codeDecoded++;
else {
codeFailed++;
if (logged++ < 20) println("REA seed failed: code " + seed.address() + " " + why(seed.address()));
}
}
case "function" -> {
Function existing = getFunctionAt(seed.address());
if (existing != null) {
functionExisting++;
if (seed.name() != null) {
existing.setName(seed.name(), SourceType.USER_DEFINED);
}
} else if (decoded(seed.address()) &&
createFunction(seed.address(), seed.name()) != null) {
functionCreated++;
} else {
functionFailed++;
if (logged++ < 20) println("REA seed failed: function " + seed.address() + " " + why(seed.address()));
}
}
default -> {
try {
createLabel(seed.address(), seed.name(), true, SourceType.USER_DEFINED);
labelApplied++;
} catch (Exception e) {
labelFailed++;
}
}
}
}
}
String report = "{\"format\":\"rea-ghidra-seeds-v1\",\"sha256\":\"" + sha256 + "\"" +
",\"entries\":" + seeds.size() +
",\"function_created\":" + functionCreated +
",\"function_existing\":" + functionExisting +
",\"function_failed\":" + functionFailed +
",\"code_decoded\":" + codeDecoded +
",\"code_failed\":" + codeFailed +
",\"label_applied\":" + labelApplied +
",\"label_failed\":" + labelFailed +
",\"unmapped\":" + unmapped + "}";
currentProgram.getOptions(Program.PROGRAM_INFO).setString(REPORT_OPTION, report);
println("REA seeds applied: " + report);
}
private String why(Address address) {
if (getDataAt(address) != null) return "defined data " + getDataAt(address).getDataType().getName();
var containing = getInstructionContaining(address);
if (containing != null && !containing.getAddress().equals(address))
return "inside instruction at " + containing.getAddress();
Function owner = getFunctionContaining(address);
if (owner != null) return "inside function " + owner.getEntryPoint();
return getInstructionAt(address) == null ? "undecodable" : "function creation refused";
}
/** Decode one instruction flow at an address, keeping an existing decode. */
private boolean decoded(Address address) {
if (getInstructionAt(address) != null) return true;
if (getDataAt(address) != null || getInstructionContaining(address) != null) {
return false;
}
disassemble(address);
return getInstructionAt(address) != null;
}
}
+520 -202
View File
@@ -27,6 +27,7 @@ import java.util.Comparator;
import java.util.HashSet; import java.util.HashSet;
import java.util.IdentityHashMap; import java.util.IdentityHashMap;
import java.util.Iterator; import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List; import java.util.List;
import java.util.Locale; import java.util.Locale;
import java.util.Map; import java.util.Map;
@@ -80,12 +81,14 @@ import ghidra.program.database.mem.FileBytes;
import ghidra.program.model.address.SegmentedAddress; import ghidra.program.model.address.SegmentedAddress;
import ghidra.program.model.reloc.Relocation; import ghidra.program.model.reloc.Relocation;
import java.io.ByteArrayOutputStream; import java.io.ByteArrayOutputStream;
import ghidra.program.model.symbol.Namespace;
import ghidra.program.model.symbol.RefType; import ghidra.program.model.symbol.RefType;
import ghidra.program.model.symbol.Reference; import ghidra.program.model.symbol.Reference;
import ghidra.program.model.symbol.ReferenceIterator; import ghidra.program.model.symbol.ReferenceIterator;
import ghidra.program.model.symbol.SourceType; import ghidra.program.model.symbol.SourceType;
import ghidra.program.model.symbol.Symbol; import ghidra.program.model.symbol.Symbol;
import ghidra.program.model.symbol.SymbolIterator; import ghidra.program.model.symbol.SymbolIterator;
import ghidra.program.model.symbol.SymbolTable;
import ghidra.program.model.pcode.FunctionPrototype; import ghidra.program.model.pcode.FunctionPrototype;
import ghidra.program.model.pcode.HighFunction; import ghidra.program.model.pcode.HighFunction;
import ghidra.program.model.pcode.HighParam; import ghidra.program.model.pcode.HighParam;
@@ -375,7 +378,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
case "inspect_native_instruction" -> inspectNativeInstruction(request.params); case "inspect_native_instruction" -> inspectNativeInstruction(request.params);
case "resolve_native_call_targets" -> resolveNativeCallTargets(request.params); case "resolve_native_call_targets" -> resolveNativeCallTargets(request.params);
case "xrefs" -> xrefs(request.params); case "xrefs" -> xrefs(request.params);
case "analyze_function" -> analyzeFunction(request.params); case "analyze_function" -> {
FunctionAnalysisObservation observation = analyzeFunctionObservation(request.params);
yield functionObservation(observation.dossier(), observation.limitations());
}
case "set_address_name", "set_addresses_names" -> { case "set_address_name", "set_addresses_names" -> {
if (!descriptor.transport.equals("unix-socket")) if (!descriptor.transport.equals("unix-socket"))
throw new RequestFailure("method_unavailable", "Windows P0 does not admit database mutation"); throw new RequestFailure("method_unavailable", "Windows P0 does not admit database mutation");
@@ -432,6 +438,11 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.add("target", target); result.add("target", target);
JsonArray extensions = analysisExtensions.reports(); JsonArray extensions = analysisExtensions.reports();
if (extensions.size() > 0) result.add("analysis_extensions", extensions); if (extensions.size() > 0) result.add("analysis_extensions", extensions);
// Written by ReaGhidraApplySeeds before auto-analysis; REA checks it
// against the committed profile.
String seeds = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
.getString("REA analysis seeds", null);
if (seeds != null) result.add("analysis_seeds", JsonParser.parseString(seeds));
return result; return result;
} }
@@ -650,10 +661,14 @@ public final class ReaGhidraBridge extends HeadlessScript {
private JsonElement procedurePseudocode(JsonObject params) throws Exception { private JsonElement procedurePseudocode(JsonObject params) throws Exception {
requireKeys(params, Set.of("document", "procedure")); requireKeys(params, Set.of("document", "procedure"));
requireDocument(params); requireDocument(params);
String value = decompiledText( Function function = resolveProcedure(requireString(params, "procedure"));
decompile(resolveProcedure(requireString(params, "procedure"))) String value = decompiledText(decompile(function));
return functionObservation(
value == null ? JsonNull.INSTANCE : GSON.toJsonTree(value),
noReturnCallLimitations(
currentProgram.getListing().getInstructions(function.getBody(), true)
)
); );
return value == null ? JsonNull.INSTANCE : GSON.toJsonTree(value);
} }
private JsonElement procedureAssembly(JsonObject params) throws Exception { private JsonElement procedureAssembly(JsonObject params) throws Exception {
@@ -661,7 +676,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
requireDocument(params); requireDocument(params);
Function function = resolveProcedure(requireString(params, "procedure")); Function function = resolveProcedure(requireString(params, "procedure"));
InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE); InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE);
return GSON.toJsonTree(renderAssembly(scan.instructions)); return functionObservation(
GSON.toJsonTree(renderAssembly(scan.instructions)),
noReturnCallLimitations(scan.instructions)
);
} }
private JsonObject readFunctionInstructions(JsonObject params) throws Exception { private JsonObject readFunctionInstructions(JsonObject params) throws Exception {
@@ -679,11 +697,13 @@ public final class ReaGhidraBridge extends HeadlessScript {
limitations.add( limitations.add(
"The fast path does not invoke the decompiler or scan whole-program names and strings." "The fast path does not invoke the decompiler or scan whole-program names and strings."
); );
List<String> boundaryLimitations = noReturnCallLimitations(scan.instructions);
for (String limitation : boundaryLimitations) limitations.add(limitation);
JsonObject result = new JsonObject(); JsonObject result = new JsonObject();
result.add("procedure", procedureIdentity(function)); result.add("procedure", procedureIdentity(function));
result.add("instructions", items); result.add("instructions", items);
result.add("limitations", limitations); result.add("limitations", limitations);
return result; return functionObservation(result, boundaryLimitations);
} }
private JsonArray procedureCalls(JsonObject params, boolean callers) throws Exception { private JsonArray procedureCalls(JsonObject params, boolean callers) throws Exception {
@@ -716,7 +736,12 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.addProperty("signature", function.getPrototypeString(false, true)); result.addProperty("signature", function.getPrototypeString(false, true));
result.add("locals", functionLocals(function)); result.add("locals", functionLocals(function));
result.add("classification", functionClassification(function)); result.add("classification", functionClassification(function));
return result; return functionObservation(
result,
noReturnCallLimitations(
currentProgram.getListing().getInstructions(function.getBody(), true)
)
);
} }
private JsonObject procedureReferences(JsonObject params) throws Exception { private JsonObject procedureReferences(JsonObject params) throws Exception {
@@ -1128,6 +1153,10 @@ public final class ReaGhidraBridge extends HeadlessScript {
} }
private JsonObject analyzeFunction(JsonObject params) throws Exception { private JsonObject analyzeFunction(JsonObject params) throws Exception {
return analyzeFunctionObservation(params).dossier();
}
private FunctionAnalysisObservation analyzeFunctionObservation(JsonObject params) throws Exception {
requireKeys(params, Set.of("procedure")); requireKeys(params, Set.of("procedure"));
Function function = resolveProcedure(requireString(params, "procedure")); Function function = resolveProcedure(requireString(params, "procedure"));
InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE); InstructionScan scan = scanInstructions(function, Integer.MAX_VALUE);
@@ -1185,12 +1214,16 @@ public final class ReaGhidraBridge extends HeadlessScript {
limitations.add( limitations.add(
"Synthetic Ghidra entry-point references without actionable memory sources are omitted." "Synthetic Ghidra entry-point references without actionable memory sources are omitted."
); );
List<String> boundaryLimitations = noReturnCallLimitations(scan.instructions);
for (String limitation : boundaryLimitations) {
limitations.add(limitation);
}
if (decompilation != null && decompilation.getErrorMessage() != null && if (decompilation != null && decompilation.getErrorMessage() != null &&
!decompilation.getErrorMessage().isBlank()) { !decompilation.getErrorMessage().isBlank()) {
limitations.add("Ghidra decompiler diagnostic: " + decompilation.getErrorMessage()); limitations.add("Ghidra decompiler diagnostic: " + decompilation.getErrorMessage());
} }
result.add("limitations", limitations); result.add("limitations", limitations);
return result; return new FunctionAnalysisObservation(result, boundaryLimitations);
} }
private JsonObject nativeValueFlow(DecompileResults decompilation) throws Exception { private JsonObject nativeValueFlow(DecompileResults decompilation) throws Exception {
@@ -1434,18 +1467,22 @@ public final class ReaGhidraBridge extends HeadlessScript {
boolean commit = false; boolean commit = false;
try { try {
Function function = currentProgram.getFunctionManager().getFunctionAt(address); Function function = currentProgram.getFunctionManager().getFunctionAt(address);
String expected = name; SymbolTable table = currentProgram.getSymbolTable();
Symbol primary = function == null ? table.getPrimarySymbol(address) : function.getSymbol();
Namespace namespace = primary == null
? currentProgram.getGlobalNamespace()
: primary.getParentNamespace();
String expected = primary == null ? name : symbolLeafName(primary, name);
for (Symbol symbol : table.getLabelOrFunctionSymbols(expected, namespace)) {
if (!address.equals(symbol.getAddress())) return false;
}
if (function != null) { if (function != null) {
expected = symbolLeafName(function.getSymbol(), name);
function.setName(expected, SourceType.USER_DEFINED); function.setName(expected, SourceType.USER_DEFINED);
} else { } else {
var table = currentProgram.getSymbolTable();
Symbol primary = table.getPrimarySymbol(address);
if (primary != null && !primary.isDynamic()) { if (primary != null && !primary.isDynamic()) {
expected = symbolLeafName(primary, name);
primary.setName(expected, SourceType.USER_DEFINED); primary.setName(expected, SourceType.USER_DEFINED);
} else { } else {
Symbol created = table.createLabel(address, name, SourceType.USER_DEFINED); Symbol created = table.createLabel(address, expected, SourceType.USER_DEFINED);
created.setPrimary(); created.setPrimary();
} }
} }
@@ -1473,16 +1510,33 @@ public final class ReaGhidraBridge extends HeadlessScript {
requireDocument(params); requireDocument(params);
if (!params.has("names") || !params.get("names").isJsonObject()) if (!params.has("names") || !params.get("names").isJsonObject())
throw new RequestFailure("invalid_request", "names must be an object of address -> name"); throw new RequestFailure("invalid_request", "names must be an object of address -> name");
Map<Address, List<AddressNameRequest>> requestsByAddress = new LinkedHashMap<>();
JsonObject result = new JsonObject(); JsonObject result = new JsonObject();
for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet()) { for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet()) {
boolean ok;
try { try {
ok = applyAddressName(parseReaAddress(entry.getKey()), entry.getValue().getAsString()); Address address = parseReaAddress(entry.getKey());
requestsByAddress.computeIfAbsent(address, ignored -> new ArrayList<>())
.add(new AddressNameRequest(entry.getKey(), address, entry.getValue().getAsString()));
} }
catch (RequestFailure failure) { catch (RequestFailure failure) {
ok = false; result.addProperty(entry.getKey(), false);
} }
result.addProperty(entry.getKey(), ok); }
for (List<AddressNameRequest> requests : requestsByAddress.values()) {
String name = requests.get(0).name;
boolean conflictingAliasNames = requests.stream()
.anyMatch(request -> !request.name.equals(name));
boolean success = false;
if (!conflictingAliasNames) {
try {
success = applyAddressName(requests.get(0).address, name);
}
catch (RequestFailure failure) {
// Keep per-address failures in the existing batch result shape.
}
}
for (AddressNameRequest request : requests)
result.addProperty(request.inputAddress, success);
} }
return result; return result;
} }
@@ -1950,9 +2004,20 @@ public final class ReaGhidraBridge extends HeadlessScript {
); );
JumpTable.LoadTable[] loadTables = jumpTable.getLoadTables(); JumpTable.LoadTable[] loadTables = jumpTable.getLoadTables();
RecoveredAArch64RelativeTable recoveredAArch64 = RecoveredAArch64RelativeTable recoveredAArch64 =
loadTables.length == 0 recoverAArch64RelativeTable(jumpTable);
? recoverAArch64RelativeTable(jumpTable) if (recoveredAArch64 != null && loadTables.length > 0) {
: null; // A decoded table only adds evidence when Ghidra's own load table
// describes the same bytes; any disagreement keeps Ghidra's view.
boolean agrees = false;
for (JumpTable.LoadTable loadTable : loadTables) {
if (loadTable.getAddress().equals(recoveredAArch64.tableAddress) &&
loadTable.getSize() == recoveredAArch64.entrySize &&
loadTable.getNum() == recoveredAArch64.mappings.size()) {
agrees = true;
}
}
if (!agrees) recoveredAArch64 = null;
}
List<Address> referencedDataAddresses = loadTables.length == 0 List<Address> referencedDataAddresses = loadTables.length == 0
? dispatchPathDataReferences(jumpTable) ? dispatchPathDataReferences(jumpTable)
: List.of(); : List.of();
@@ -1986,7 +2051,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
source.add("evidence", evidence); source.add("evidence", evidence);
dataSources.add(source); dataSources.add(source);
} }
if (recoveredAArch64 != null) { if (recoveredAArch64 != null && loadTables.length == 0) {
JsonObject source = new JsonObject(); JsonObject source = new JsonObject();
source.addProperty( source.addProperty(
"address", "address",
@@ -2001,9 +2066,16 @@ public final class ReaGhidraBridge extends HeadlessScript {
inferenceEvidence( inferenceEvidence(
"jump-table", "jump-table",
"ghidra-instruction-and-memory", "ghidra-instruction-and-memory",
"Decoded " + recoveredAArch64.mappings.size() + "Decoded " + recoveredAArch64.mappings.size() + " " +
(recoveredAArch64.signedEntries ? "signed" : "unsigned") +
" entries of size " + recoveredAArch64.entrySize + " bytes from " + canonicalAddress(recoveredAArch64.tableAddress) + " entries of size " + recoveredAArch64.entrySize + " bytes from " + canonicalAddress(recoveredAArch64.tableAddress) +
"; the AArch64 LDRB-or-LDRH/ADR/ADD/BR chain uses a selector guarded by an unsigned upper-bound branch." "; the AArch64 ADRP/ADD table, indexed load, ADR base, ADD" +
(recoveredAArch64.offsetShift == 0 ? "" : " LSL #" + recoveredAArch64.offsetShift) +
"/BR chain uses a selector bounded by CMP #" + recoveredAArch64.compareImmediate +
" and " + recoveredAArch64.guardMnemonic + " on one straight-line path" +
(recoveredAArch64.nullTarget == null
? "."
: "; zero entries are routed by CBZ to " + canonicalAddress(recoveredAArch64.nullTarget) + ".")
) )
); );
source.add("evidence", evidence); source.add("evidence", evidence);
@@ -2042,6 +2114,33 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.add("data_sources", dataSources); result.add("data_sources", dataSources);
Address[] targets = jumpTable.getCases(); Address[] targets = jumpTable.getCases();
TypedJumpTableCases typedCases = typedJumpTableCases(jumpTable, caseTokens); TypedJumpTableCases typedCases = typedJumpTableCases(jumpTable, caseTokens);
// Table indices identify the register bounded by CMP. Typed labels may
// instead identify a selector before normalization (for example, SUB
// index, selector, #10). Do not merge those distinct value identities.
if (recoveredAArch64 != null && recoveredAArch64.priorSelectorDefinition != null) {
typedCases.limitations.add(
"REA decoded the relative table, but the compared selector has an earlier register definition at " +
canonicalAddress(recoveredAArch64.priorSelectorDefinition) + "; its table indices are not proven to use the typed switch selector's case values. Ghidra's typed labels are retained and additional case labels remain unknown."
);
recoveredAArch64 = null;
}
if (recoveredAArch64 != null) {
for (Map.Entry<Long, Address> typed : typedCases.cases.entrySet()) {
long value = typed.getKey();
if (value < 0 || value >= recoveredAArch64.mappings.size()) {
typedCases.limitations.add("Typed case labels and decoded table indices do not use a verified common selector; Ghidra's typed labels are retained.");
recoveredAArch64 = null;
break;
}
RecoveredAArch64Mapping decoded = recoveredAArch64.mappings.get((int) value);
boolean divertedZero = decoded.nullEntry && typed.getValue().equals(recoveredAArch64.unreachableZeroTarget);
if (!typed.getValue().equals(decoded.targetAddress) && !divertedZero) {
typedCases.limitations.add("Typed case evidence and the decoded relative table disagree on a destination; Ghidra's typed labels are retained.");
recoveredAArch64 = null;
break;
}
}
}
JsonArray mappings = new JsonArray(); JsonArray mappings = new JsonArray();
JsonArray defaultTargets = new JsonArray(); JsonArray defaultTargets = new JsonArray();
for (Address target : typedCases.defaults) { for (Address target : typedCases.defaults) {
@@ -2049,27 +2148,30 @@ public final class ReaGhidraBridge extends HeadlessScript {
} }
Set<Address> coveredTargets = new HashSet<>(typedCases.cases.values()); Set<Address> coveredTargets = new HashSet<>(typedCases.cases.values());
coveredTargets.addAll(typedCases.defaults); coveredTargets.addAll(typedCases.defaults);
if (recoveredAArch64 != null && recoveredAArch64.unreachableZeroTarget != null) {
Address unreachable = recoveredAArch64.unreachableZeroTarget;
coveredTargets.add(unreachable);
for (RecoveredAArch64Mapping recovered : recoveredAArch64.mappings) {
if (recovered.nullEntry && unreachable.equals(typedCases.cases.get((long) recovered.tableIndex))) {
typedCases.cases.remove((long) recovered.tableIndex);
}
}
typedCases.limitations.add(
"Ghidra recovered destination " + canonicalAddress(unreachable) +
" as the zero-offset decode of the relative table, but the CBZ before the dispatch diverts zero entries to " +
canonicalAddress(recoveredAArch64.nullTarget) + ", so the indirect branch never reaches it for an in-range case."
);
}
if (recoveredAArch64 != null) { if (recoveredAArch64 != null) {
for (int index = 0; index < recoveredAArch64.mappings.size(); index += 1) { for (int index = 0; index < recoveredAArch64.mappings.size(); index += 1) {
monitor.checkCancelled(); monitor.checkCancelled();
RecoveredAArch64Mapping recovered = recoveredAArch64.mappings.get(index); RecoveredAArch64Mapping recovered = recoveredAArch64.mappings.get(index);
if ( if (
typedCases.unknownTargets.contains(recovered.targetAddress) || typedCases.unknownTargets.contains(recovered.targetAddress) ||
typedCases.ambiguousValues.contains((long) recovered.caseValue) typedCases.ambiguousValues.contains((long) recovered.tableIndex)
) continue; ) continue;
Address typedTarget = typedCases.cases.get((long) recovered.caseValue);
if (typedTarget != null && !typedTarget.equals(recovered.targetAddress)) {
typedCases.cases.remove((long) recovered.caseValue);
typedCases.ambiguousValues.add((long) recovered.caseValue);
typedCases.unknownTargets.add(typedTarget);
typedCases.unknownTargets.add(recovered.targetAddress);
typedCases.limitations.add(
"Typed case evidence and the verified relative table disagree on a destination; both destinations remain unknown for that case."
);
continue;
}
JsonObject mapping = new JsonObject(); JsonObject mapping = new JsonObject();
mapping.addProperty("case_value", recovered.caseValue); mapping.addProperty("case_value", recovered.tableIndex);
String targetAddress = canonicalAddress(recovered.targetAddress); String targetAddress = canonicalAddress(recovered.targetAddress);
mapping.addProperty("target_address", targetAddress); mapping.addProperty("target_address", targetAddress);
mapping.addProperty("confidence", "high"); mapping.addProperty("confidence", "high");
@@ -2078,14 +2180,18 @@ public final class ReaGhidraBridge extends HeadlessScript {
inferenceEvidence( inferenceEvidence(
"jump-table", "jump-table",
"ghidra-instruction-and-memory", "ghidra-instruction-and-memory",
"Case value " + recovered.caseValue + (recoveredAArch64.entrySize == 1 ? " indexes byte " : " indexes halfword at byte offset ") + "Case value " + recovered.tableIndex + (recoveredAArch64.entrySize == 1 ? " indexes byte " : " indexes halfword at byte offset ") +
(recovered.caseValue * recoveredAArch64.entrySize) + " at " + canonicalAddress(recoveredAArch64.tableAddress) + (recovered.tableIndex * recoveredAArch64.entrySize) + " at " + canonicalAddress(recoveredAArch64.tableAddress) +
"; the loaded unsigned entry selects a four-byte instruction offset from " + (recovered.nullEntry
canonicalAddress(recoveredAArch64.branchBase) + ", reaching " + targetAddress + "." ? "; the entry is zero, so the CBZ before the dispatch branches to " + targetAddress + "."
: "; the loaded " + (recoveredAArch64.signedEntries ? "signed" : "unsigned") +
" entry " + recovered.encodedEntry + " selects a " +
(recoveredAArch64.offsetShift == 2 ? "four-byte instruction" : "byte") +
" offset from " + canonicalAddress(recoveredAArch64.branchBase) + ", reaching " + targetAddress + ".")
) )
); );
mapping.add("evidence", evidence); mapping.add("evidence", evidence);
if (!typedCases.cases.containsKey((long) recovered.caseValue)) { if (!typedCases.cases.containsKey((long) recovered.tableIndex)) {
mappings.add(mapping); mappings.add(mapping);
} }
coveredTargets.add(recovered.targetAddress); coveredTargets.add(recovered.targetAddress);
@@ -2159,7 +2265,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
); );
} }
if (recoveredAArch64 == null && loadTables.length == 0 && currentProgram.getLanguageID().getIdAsString().startsWith("AARCH64:")) { if (recoveredAArch64 == null && loadTables.length == 0 && currentProgram.getLanguageID().getIdAsString().startsWith("AARCH64:")) {
limitations.add("REA did not verify a supported byte/halfword relative dispatch chain with exact unsigned bounds and register definitions; unsupported encodings and ambiguous bounds remain unknown."); limitations.add("REA did not verify a supported signed or unsigned byte/halfword relative dispatch chain with exact unsigned bounds and register definitions; unsupported encodings and ambiguous bounds remain unknown.");
} }
if (targets.length == 0) { if (targets.length == 0) {
limitations.add( limitations.add(
@@ -2289,12 +2395,21 @@ public final class ReaGhidraBridge extends HeadlessScript {
} }
private static final class RecoveredAArch64Mapping { private static final class RecoveredAArch64Mapping {
private final int caseValue; private final int tableIndex;
private final Address targetAddress; private final Address targetAddress;
private final long encodedEntry;
private final boolean nullEntry;
private RecoveredAArch64Mapping(int caseValue, Address targetAddress) { private RecoveredAArch64Mapping(
this.caseValue = caseValue; int tableIndex,
Address targetAddress,
long encodedEntry,
boolean nullEntry
) {
this.tableIndex = tableIndex;
this.targetAddress = targetAddress; this.targetAddress = targetAddress;
this.encodedEntry = encodedEntry;
this.nullEntry = nullEntry;
} }
} }
@@ -2302,24 +2417,70 @@ public final class ReaGhidraBridge extends HeadlessScript {
private final Address tableAddress; private final Address tableAddress;
private final Address branchBase; private final Address branchBase;
private final int entrySize; private final int entrySize;
private final Address priorSelectorDefinition;
private final boolean signedEntries;
private final int offsetShift;
private final String guardMnemonic;
private final long compareImmediate;
private final Address nullTarget;
private final Address unreachableZeroTarget;
private final List<RecoveredAArch64Mapping> mappings; private final List<RecoveredAArch64Mapping> mappings;
private RecoveredAArch64RelativeTable( private RecoveredAArch64RelativeTable(
Address tableAddress, Address tableAddress,
Address branchBase, Address branchBase,
int entrySize, int entrySize,
Address priorSelectorDefinition,
boolean signedEntries,
int offsetShift,
String guardMnemonic,
long compareImmediate,
Address nullTarget,
Address unreachableZeroTarget,
List<RecoveredAArch64Mapping> mappings List<RecoveredAArch64Mapping> mappings
) { ) {
this.tableAddress = tableAddress; this.tableAddress = tableAddress;
this.branchBase = branchBase; this.branchBase = branchBase;
this.entrySize = entrySize; this.entrySize = entrySize;
this.priorSelectorDefinition = priorSelectorDefinition;
this.signedEntries = signedEntries;
this.offsetShift = offsetShift;
this.guardMnemonic = guardMnemonic;
this.compareImmediate = compareImmediate;
this.nullTarget = nullTarget;
this.unreachableZeroTarget = unreachableZeroTarget;
this.mappings = mappings; this.mappings = mappings;
} }
} }
private static final Pattern AARCH64_DISPATCH_ADD = Pattern.compile(
"(?i)^([wx]\\d+),\\s*([wx]\\d+),\\s*([wx]\\d+)(?:,\\s*LSL\\s*#(?:0x)?([0-9a-f]+))?$"
);
private static final Pattern AARCH64_TABLE_LOAD = Pattern.compile(
"(?i)^([wx]\\d+),\\s*\\[([wx]\\d+),\\s*([wx]\\d+)(?:,\\s*(LSL|UXTW)\\s*(?:#(?:0x)?([0-9a-f]+))?)?\\s*\\]$"
);
private static final Pattern AARCH64_REGISTER_AND_ADDRESS = Pattern.compile(
"(?i)^([wx]\\d+),\\s*(0x[0-9a-f]+)$"
);
private static final Pattern AARCH64_ADD_IMMEDIATE = Pattern.compile(
"(?i)^([wx]\\d+),\\s*([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$"
);
private static final Pattern AARCH64_COMPARE_IMMEDIATE = Pattern.compile(
"(?i)^([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$"
);
private static final Pattern AARCH64_MOVE_REGISTER = Pattern.compile(
"(?i)^([wx]\\d+),\\s*([wx]\\d+)$"
);
private static final Set<String> AARCH64_FLAG_PRESERVING =
Set.of("MOV", "ADRP", "ADR", "ADD");
/** /**
* Recover only the directly verified AArch64 byte/halfword-indexed relative branch * Recover only directly verified AArch64 relative branch tables:
* table form. Other instruction sequences keep their case labels unknown. * ADRP/ADD table base, LDRB/LDRH/LDRSB/LDRSH entry load with an exact
* index scale, ADR branch base, optional CBZ on the loaded entry, and
* ADD (byte or LSL #2 scale)/BR, with a selector bounded by CMP #imm and an
* unsigned B.HI or B.CS/B.HS guard on one straight-line path. Any other
* instruction sequence keeps its case labels unknown.
*/ */
private RecoveredAArch64RelativeTable recoverAArch64RelativeTable( private RecoveredAArch64RelativeTable recoverAArch64RelativeTable(
JumpTable jumpTable JumpTable jumpTable
@@ -2341,167 +2502,265 @@ public final class ReaGhidraBridge extends HeadlessScript {
if (instruction.getAddress().equals(jumpTable.getSwitchAddress())) break; if (instruction.getAddress().equals(jumpTable.getSwitchAddress())) break;
} }
int branchIndex = instructions.size() - 1; int branchIndex = instructions.size() - 1;
if (branchIndex < 1 || if (branchIndex < 1 || !mnemonic(instructions.get(branchIndex)).equals("BR")) {
!instructions.get(branchIndex).getMnemonicString().equalsIgnoreCase("BR")) { return null;
}
Set<String> recoveredTargetSet = new HashSet<>();
for (Address target : jumpTable.getCases()) recoveredTargetSet.add(canonicalAddress(target));
if (recoveredTargetSet.isEmpty()) return null;
return matchAArch64RelativeDispatch(
instructions,
branchIndex,
recoveredTargetSet,
dispatchPathDataReferences(jumpTable)
);
}
private RecoveredAArch64RelativeTable matchAArch64RelativeDispatch(
List<Instruction> instructions,
int branchIndex,
Set<String> recoveredTargetSet,
List<Address> dataAddresses
) throws Exception {
String branchRegister = normalizedRegister(operands(instructions.get(branchIndex)));
// ADD Xd, Xn, Xm{, LSL #s}; BR Xd. LSL applies to Xm only.
Instruction dispatchAdd = instructions.get(branchIndex - 1);
if (!mnemonic(dispatchAdd).equals("ADD")) return null;
java.util.regex.Matcher addMatcher = AARCH64_DISPATCH_ADD.matcher(operands(dispatchAdd));
if (!addMatcher.matches() || !addMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) return null;
if (!normalizedRegister(addMatcher.group(1)).equals(branchRegister)) return null;
int offsetShift = addMatcher.group(4) == null ? 0 : Integer.parseInt(addMatcher.group(4), 16);
if (offsetShift != 0 && offsetShift != 2) return null;
String firstSource = normalizedRegister(addMatcher.group(2));
String secondSource = normalizedRegister(addMatcher.group(3));
if (firstSource.equals(secondSource)) return null;
// The entry register is the source whose latest definition is the
// table load; the other source must be defined by ADR.
int loadIndex = -1;
String entryRegister = null;
String baseRegister = null;
for (String candidate : offsetShift == 0 ? List.of(secondSource, firstSource) : List.of(secondSource)) {
int writer = latestWriter(instructions, branchIndex - 1, candidate);
if (writer >= 0 && Set.of("LDRB", "LDRH", "LDRSB", "LDRSH").contains(mnemonic(instructions.get(writer)))) {
loadIndex = writer;
entryRegister = candidate;
baseRegister = candidate.equals(firstSource) ? secondSource : firstSource;
break;
}
}
if (loadIndex < 0) return null;
Instruction load = instructions.get(loadIndex);
String loadMnemonic = mnemonic(load);
int entrySize = loadMnemonic.endsWith("B") ? 1 : 2;
boolean signedEntries = loadMnemonic.startsWith("LDRS");
java.util.regex.Matcher loadMatcher = AARCH64_TABLE_LOAD.matcher(operands(load));
if (!loadMatcher.matches()) return null;
// A signed load into a W register sign-extends only to 32 bits; the
// 64-bit dispatch ADD would then read a zero-extended value.
if (signedEntries && !loadMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) return null;
String tableBaseRegister = normalizedRegister(loadMatcher.group(2));
String indexRegister = normalizedRegister(loadMatcher.group(3));
boolean wordIndex = loadMatcher.group(3).toLowerCase(Locale.ROOT).startsWith("w");
String extend = loadMatcher.group(4) == null ? null : loadMatcher.group(4).toUpperCase(Locale.ROOT);
int indexShift = loadMatcher.group(5) == null ? 0 : Integer.parseInt(loadMatcher.group(5), 16);
if (wordIndex ? !"UXTW".equals(extend) : "UXTW".equals(extend)) return null;
if (indexShift != (entrySize == 1 ? 0 : 1)) return null;
// Only ADR and one CBZ on the entry may sit between load and ADD.
Address nullTarget = null;
for (int index = loadIndex + 1; index < branchIndex - 1; index += 1) {
Instruction instruction = instructions.get(index);
String name = mnemonic(instruction);
if (name.equals("ADR")) continue;
if (name.equals("CBZ") && nullTarget == null) {
String[] parts = operands(instruction).split(",");
Address[] flows = instruction.getFlows();
if (parts.length != 2 || flows.length != 1 ||
!normalizedRegister(parts[0]).equals(entryRegister)) return null;
nullTarget = flows[0];
continue;
}
return null; return null;
} }
Address[] recoveredTargets = jumpTable.getCases(); int baseIndex = latestWriter(instructions, branchIndex - 1, baseRegister);
Set<String> recoveredTargetSet = new HashSet<>(); if (baseIndex < 0 || !mnemonic(instructions.get(baseIndex)).equals("ADR")) return null;
for (Address target : recoveredTargets) recoveredTargetSet.add(canonicalAddress(target)); java.util.regex.Matcher baseMatcher = AARCH64_REGISTER_AND_ADDRESS.matcher(operands(instructions.get(baseIndex)));
List<Address> dataAddresses = dispatchPathDataReferences(jumpTable); if (!baseMatcher.matches()) return null;
Address branchBase = addressFromOperand(baseMatcher.group(2));
for (int loadIndex = 0; loadIndex < branchIndex; loadIndex += 1) { int tableAddIndex = latestWriter(instructions, loadIndex, tableBaseRegister);
Instruction load = instructions.get(loadIndex); if (tableAddIndex < 0 || !mnemonic(instructions.get(tableAddIndex)).equals("ADD")) return null;
int entrySize = load.getMnemonicString().equalsIgnoreCase("LDRB") ? 1 : load.getMnemonicString().equalsIgnoreCase("LDRH") ? 2 : 0; java.util.regex.Matcher tableAddMatcher = AARCH64_ADD_IMMEDIATE.matcher(operands(instructions.get(tableAddIndex)));
if (entrySize == 0 || branchIndex != loadIndex + 2 || loadIndex < 3 || if (!tableAddMatcher.matches() ||
!instructions.get(loadIndex - 1).getMnemonicString().equalsIgnoreCase("ADR") || !normalizedRegister(tableAddMatcher.group(1)).equals(tableBaseRegister) ||
!instructions.get(loadIndex - 2).getMnemonicString().equalsIgnoreCase("ADD") || !normalizedRegister(tableAddMatcher.group(2)).equals(tableBaseRegister)) return null;
!instructions.get(loadIndex - 3).getMnemonicString().equalsIgnoreCase("ADRP")) continue; int tablePageIndex = latestWriter(instructions, tableAddIndex, tableBaseRegister);
String loadOperands = operands(load); if (tablePageIndex < 0 || !mnemonic(instructions.get(tablePageIndex)).equals("ADRP")) return null;
java.util.regex.Matcher loadMatcher = java.util.regex.Pattern java.util.regex.Matcher pageMatcher = AARCH64_REGISTER_AND_ADDRESS.matcher(operands(instructions.get(tablePageIndex)));
.compile(entrySize == 1 ? "(?i)^[wx]\\d+,\\s*\\[([wx]\\d+),\\s*([wx]\\d+)\\]$" : "(?i)^[wx]\\d+,\\s*\\[([wx]\\d+),\\s*([wx]\\d+),\\s*LSL\\s*#(?:0x)?1\\]$") if (!pageMatcher.matches() || !normalizedRegister(pageMatcher.group(1)).equals(tableBaseRegister)) return null;
.matcher(loadOperands); Address tableAddress = addressFromOperand(pageMatcher.group(2)).add(parseInteger(tableAddMatcher.group(3)));
if (!loadMatcher.matches()) continue; if (!currentProgram.getMemory().contains(tableAddress) ||
String loadedRegister = normalizedRegister(loadOperands.substring(0, loadOperands.indexOf(','))); (!dataAddresses.isEmpty() && !dataAddresses.contains(tableAddress))) return null;
String tableBaseRegister = normalizedRegister(loadMatcher.group(1));
String indexRegister = normalizedRegister(loadMatcher.group(2));
Address resolvedTableAddress = null; // Unsigned guard: CMP Rs, #imm, then B.HI (imm + 1 entries) or
for (int index = loadIndex - 3; index < loadIndex - 1; index += 1) { // B.CS/B.HS (imm entries), with only flag-preserving moves and
Instruction instruction = instructions.get(index); // address arithmetic between the compare and the load.
String mnemonic = instruction.getMnemonicString().toUpperCase(Locale.ROOT); int guardIndex = -1;
String text = operands(instruction); for (int index = loadIndex - 1; index >= 0; index -= 1) {
if (mnemonic.equals("ADRP") && String name = mnemonic(instructions.get(index));
text.matches("(?i)^[wx]\\d+,\\s*0x[0-9a-f]+$")) { if (name.startsWith("B.")) {
String[] parts = text.split(","); guardIndex = index;
if (normalizedRegister(parts[0]).equals(tableBaseRegister)) { break;
resolvedTableAddress = addressFromOperand(parts[1]);
}
}
else if (mnemonic.equals("ADD") && resolvedTableAddress != null &&
text.matches("(?i)^[wx]\\d+,\\s*[wx]\\d+,\\s*#?(?:0x[0-9a-f]+|[0-9]+)$")) {
String[] parts = text.split(",");
if (normalizedRegister(parts[0]).equals(tableBaseRegister) &&
normalizedRegister(parts[1]).equals(tableBaseRegister)) {
resolvedTableAddress = resolvedTableAddress.add(parseInteger(parts[2].trim().replaceFirst("^#", "")));
}
}
} }
if (resolvedTableAddress == null || !currentProgram.getMemory().contains(resolvedTableAddress) || if (!AARCH64_FLAG_PRESERVING.contains(name)) return null;
(!dataAddresses.isEmpty() && !dataAddresses.contains(resolvedTableAddress))) continue;
Address tableAddress = resolvedTableAddress;
Address branchBase = null;
boolean addressChain = false;
for (int index = loadIndex - 1; index < branchIndex; index += 1) {
Instruction instruction = instructions.get(index);
String mnemonic = instruction.getMnemonicString().toUpperCase(Locale.ROOT);
String text = operands(instruction);
if (index < loadIndex && mnemonic.equals("ADR") && text.matches("(?i)^[wx]\\d+,\\s*0x[0-9a-f]+$")) {
int comma = text.indexOf(',');
String destination = normalizedRegister(text.substring(0, comma));
if (!destination.equals(normalizedRegister(operands(instructions.get(branchIndex))))) continue;
branchBase = addressFromOperand(text.substring(comma + 1));
}
if (index > loadIndex && mnemonic.equals("ADD") && branchBase != null &&
text.matches("(?i)^[wx]\\d+,\\s*[wx]\\d+,\\s*[wx]\\d+,\\s*LSL\\s*#(?:0x)?2$")) {
String[] parts = text.split(",");
String destination = normalizedRegister(parts[0]);
String base = normalizedRegister(parts[1]);
String offset = normalizedRegister(parts[2]);
if (destination.equals(base) &&
base.equals(normalizedRegister(operands(instructions.get(branchIndex)))) &&
offset.equals(loadedRegister)) {
addressChain = true;
}
}
}
if (!addressChain || branchBase == null) continue;
int entryCount = -1;
Address defaultTarget = null;
for (int compareIndex = 0; compareIndex < loadIndex; compareIndex += 1) {
Instruction compare = instructions.get(compareIndex);
if (!compare.getMnemonicString().equalsIgnoreCase("CMP")) continue;
java.util.regex.Matcher compareMatcher = java.util.regex.Pattern
.compile("(?i)^([wx]\\d+),\\s*#?(0x[0-9a-f]+|[0-9]+)$")
.matcher(operands(compare));
if (!compareMatcher.matches()) continue;
String comparedRegister = normalizedRegister(compareMatcher.group(1));
long upperBound = parseInteger(compareMatcher.group(2));
if (upperBound < 0 || upperBound >= 4096) continue;
for (int guardIndex = compareIndex + 1; guardIndex < loadIndex; guardIndex += 1) {
Instruction guard = instructions.get(guardIndex);
String mnemonic = guard.getMnemonicString().toUpperCase(Locale.ROOT);
if (!mnemonic.equals("B.HI")) continue;
Address[] flows = guard.getFlows();
if (flows.length != 1 || !recoveredTargetSet.contains(canonicalAddress(flows[0]))) continue;
boolean supportedSequence = true;
for (int sequenceIndex = compareIndex + 1; sequenceIndex < loadIndex; sequenceIndex += 1) {
String sequenceMnemonic = instructions.get(sequenceIndex)
.getMnemonicString().toUpperCase(Locale.ROOT);
if (!Set.of("B.HI", "MOV", "ADRP", "ADD", "ADR").contains(sequenceMnemonic)) {
supportedSequence = false;
break;
}
}
Set<String> selectorAliases = new HashSet<>();
selectorAliases.add(comparedRegister);
Set<String> boundedIndexes = new HashSet<>();
if (compareMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x")) boundedIndexes.add(comparedRegister);
for (int sequenceIndex = compareIndex + 1; sequenceIndex < loadIndex; sequenceIndex++) {
Instruction instruction = instructions.get(sequenceIndex);
String[] parts = operands(instruction).split(",");
if (parts.length == 0) continue;
if (instruction.getMnemonicString().equalsIgnoreCase("MOV")) {
String destination = normalizedRegister(parts[0]);
String source = parts.length == 2 ? normalizedRegister(parts[1]) : "";
boolean aliasesSelector = selectorAliases.contains(source);
boolean fullWidthBounded = boundedIndexes.contains(source) ||
(parts[0].trim().toLowerCase(Locale.ROOT).startsWith("w") && aliasesSelector);
if (aliasesSelector) selectorAliases.add(destination);
else selectorAliases.remove(destination);
if (fullWidthBounded) boundedIndexes.add(destination);
else boundedIndexes.remove(destination);
} else if (Set.of("ADRP", "ADD", "ADR").contains(instruction.getMnemonicString().toUpperCase(Locale.ROOT))) {
selectorAliases.remove(normalizedRegister(parts[0]));
boundedIndexes.remove(normalizedRegister(parts[0]));
}
}
boolean selectorFeedsTable = boundedIndexes.contains(indexRegister);
if (compareIndex < loadIndex - 3 && selectorFeedsTable && supportedSequence) {
entryCount = (int) upperBound + 1;
defaultTarget = flows[0];
break;
}
}
if (entryCount > 0) break;
}
if (entryCount <= 0 || entryCount > 4096 || defaultTarget == null) continue;
List<RecoveredAArch64Mapping> mappings = new ArrayList<>();
Set<String> mappedTargets = new HashSet<>();
boolean valid = true;
for (int caseValue = 0; caseValue < entryCount; caseValue += 1) {
Address entry = tableAddress.add((long) caseValue * entrySize);
int encodedOffset = entrySize == 1 ? currentProgram.getMemory().getByte(entry) & 0xff : currentProgram.getMemory().getShort(entry) & 0xffff;
Address target = branchBase.add((long) encodedOffset * 4L);
String canonicalTarget = canonicalAddress(target);
if (!recoveredTargetSet.contains(canonicalTarget)) {
valid = false;
break;
}
mappedTargets.add(canonicalTarget);
mappings.add(new RecoveredAArch64Mapping(caseValue, target));
}
if (!valid || mappedTargets.contains(canonicalAddress(defaultTarget))) continue;
Set<String> accountedTargets = new HashSet<>(mappedTargets);
accountedTargets.add(canonicalAddress(defaultTarget));
if (!accountedTargets.equals(recoveredTargetSet)) continue;
return new RecoveredAArch64RelativeTable(tableAddress, branchBase, entrySize, List.copyOf(mappings));
} }
return null; if (guardIndex < 1) return null;
Instruction guard = instructions.get(guardIndex);
String guardMnemonic = mnemonic(guard);
if (!Set.of("B.HI", "B.CS", "B.HS").contains(guardMnemonic)) return null;
Address[] guardFlows = guard.getFlows();
if (guardFlows.length != 1) return null;
Address defaultTarget = guardFlows[0];
int compareIndex = -1;
for (int index = guardIndex - 1; index >= 0; index -= 1) {
String name = mnemonic(instructions.get(index));
if (name.equals("CMP")) {
compareIndex = index;
break;
}
if (!AARCH64_FLAG_PRESERVING.contains(name)) return null;
}
if (compareIndex < 0) return null;
java.util.regex.Matcher compareMatcher = AARCH64_COMPARE_IMMEDIATE.matcher(operands(instructions.get(compareIndex)));
if (!compareMatcher.matches()) return null;
String comparedRegister = normalizedRegister(compareMatcher.group(1));
boolean comparedFullWidth = compareMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("x");
long compareImmediate = parseInteger(compareMatcher.group(2));
long entryCountLong = guardMnemonic.equals("B.HI") ? compareImmediate + 1 : compareImmediate;
if (entryCountLong <= 0 || entryCountLong > 4096) return null;
int entryCount = (int) entryCountLong;
// The loaded index must carry the compared value: either the compared
// register itself, unchanged since CMP, or a MOV copy made after CMP.
// A W compare bounds only the low 32 bits, so an X index needs a
// zero-extending W copy; a W index is zero-extended by UXTW.
int indexWriter = latestWriter(instructions, loadIndex, indexRegister);
boolean bounded = false;
if (indexWriter < compareIndex) {
bounded = indexRegister.equals(comparedRegister) && (wordIndex || comparedFullWidth);
}
else if (mnemonic(instructions.get(indexWriter)).equals("MOV")) {
java.util.regex.Matcher moveMatcher = AARCH64_MOVE_REGISTER.matcher(operands(instructions.get(indexWriter)));
if (moveMatcher.matches() &&
normalizedRegister(moveMatcher.group(2)).equals(comparedRegister) &&
latestWriter(instructions, indexWriter, comparedRegister) < compareIndex) {
boolean wordCopy = moveMatcher.group(1).toLowerCase(Locale.ROOT).startsWith("w");
bounded = wordCopy || comparedFullWidth;
}
}
if (!bounded) return null;
// Every fact above holds only on one straight-line fall-through path
// with no other entry into the window.
int windowStart = Math.min(Math.min(tablePageIndex, baseIndex), compareIndex);
for (int index = windowStart + 1; index <= branchIndex; index += 1) {
monitor.checkCancelled();
Instruction previous = instructions.get(index - 1);
Instruction current = instructions.get(index);
Address fallThrough = previous.getFallThrough();
if (fallThrough == null || !fallThrough.equals(current.getAddress())) return null;
ReferenceIterator incoming = currentProgram.getReferenceManager().getReferencesTo(current.getAddress());
while (incoming.hasNext()) {
if (incoming.next().getReferenceType().isFlow()) return null;
}
}
List<RecoveredAArch64Mapping> mappings = new ArrayList<>();
Set<String> mappedTargets = new HashSet<>();
for (int tableIndex = 0; tableIndex < entryCount; tableIndex += 1) {
monitor.checkCancelled();
Address entry = tableAddress.add((long) tableIndex * entrySize);
if (!currentProgram.getMemory().contains(entry)) return null;
long encoded;
if (entrySize == 1) {
byte value = currentProgram.getMemory().getByte(entry);
encoded = signedEntries ? value : value & 0xffL;
}
else {
short value = currentProgram.getMemory().getShort(entry);
encoded = signedEntries ? value : value & 0xffffL;
}
if (encoded == 0 && nullTarget != null) {
mappings.add(new RecoveredAArch64Mapping(tableIndex, nullTarget, encoded, true));
continue;
}
Address target;
try {
target = branchBase.addNoWrap(encoded << offsetShift);
}
catch (ghidra.program.model.address.AddressOverflowException overflow) {
return null;
}
String canonicalTarget = canonicalAddress(target);
if (!recoveredTargetSet.contains(canonicalTarget)) return null;
mappedTargets.add(canonicalTarget);
mappings.add(new RecoveredAArch64Mapping(tableIndex, target, encoded, false));
}
if (mappedTargets.contains(canonicalAddress(defaultTarget))) return null;
// Ghidra's recovered destinations must be fully accounted for by the
// decoded entries, the guard default, and the CBZ null path.
Set<String> accountedTargets = new HashSet<>(mappedTargets);
accountedTargets.add(canonicalAddress(defaultTarget));
if (nullTarget != null) accountedTargets.add(canonicalAddress(nullTarget));
// Ghidra does not model the CBZ, so it may decode a zero entry as a
// branch to the base itself. Only a zero entry encodes offset 0, and
// the CBZ diverts every zero entry, so the BR never reaches the base.
Address unreachableZeroTarget = null;
boolean divertedZeroEntry = mappings.stream().anyMatch(mapping -> mapping.nullEntry);
if (divertedZeroEntry &&
recoveredTargetSet.contains(canonicalAddress(branchBase)) &&
!mappedTargets.contains(canonicalAddress(branchBase)) &&
!branchBase.equals(defaultTarget) &&
!branchBase.equals(nullTarget)) {
unreachableZeroTarget = branchBase;
accountedTargets.add(canonicalAddress(branchBase));
}
if (!accountedTargets.containsAll(recoveredTargetSet)) return null;
int selectorWriter = latestWriter(instructions, compareIndex, comparedRegister);
return new RecoveredAArch64RelativeTable(
tableAddress,
branchBase,
entrySize,
selectorWriter < 0 ? null : instructions.get(selectorWriter).getAddress(),
signedEntries,
offsetShift,
guardMnemonic,
compareImmediate,
nullTarget,
unreachableZeroTarget,
List.copyOf(mappings)
);
}
private static String mnemonic(Instruction instruction) {
return instruction.getMnemonicString().toUpperCase(Locale.ROOT);
}
/** Index of the last instruction before {@code before} that writes {@code register}, or -1. */
private static int latestWriter(List<Instruction> instructions, int before, String register) {
for (int index = before - 1; index >= 0; index -= 1) {
for (Object result : instructions.get(index).getResultObjects()) {
if (result instanceof Register written &&
normalizedRegister(written.getBaseRegister().getName()).equals(register)) {
return index;
}
}
}
return -1;
} }
private static String operands(Instruction instruction) { private static String operands(Instruction instruction) {
@@ -2648,6 +2907,63 @@ public final class ReaGhidraBridge extends HeadlessScript {
return new InstructionScan(List.copyOf(instructions), iterator.hasNext()); return new InstructionScan(List.copyOf(instructions), iterator.hasNext());
} }
private JsonObject functionObservation(
JsonElement value,
List<String> limitations) {
JsonObject result = new JsonObject();
result.add("value", value);
result.add("limitations", GSON.toJsonTree(limitations));
return result;
}
private List<String> noReturnCallLimitations(Iterable<Instruction> instructions)
throws Exception {
List<String> limitations = new ArrayList<>();
for (Instruction instruction : instructions) {
monitor.checkCancelled();
if (!instruction.getFlowType().isCall() || !instruction.getFlowType().isTerminal()) {
continue;
}
Address[] destinations = instruction.getFlows();
StringBuilder limitation = new StringBuilder();
limitation.append("Ghidra Listing reports a terminal call at ")
.append(canonicalAddress(instruction.getAddress()));
if (destinations.length == 0) {
limitation.append(" with no resolved callee; its no-return basis is unknown.");
}
else {
limitation.append(" to ");
for (int index = 0; index < destinations.length; index++) {
if (index > 0) limitation.append(", ");
Address destination = destinations[index];
limitation.append(canonicalAddress(destination));
Function target = currentProgram.getFunctionManager().getFunctionAt(destination);
if (target == null) {
limitation.append(" (callee identity unknown)");
}
else {
limitation.append(" ").append(procedureName(target));
limitation.append(" (external=").append(target.isExternal());
limitation.append(", thunk=").append(target.isThunk());
limitation.append(", hasNoReturn=").append(target.hasNoReturn()).append(")");
}
}
limitation.append("; fallthrough is excluded by Ghidra's flow model.");
Function target = destinations.length == 1
? currentProgram.getFunctionManager().getFunctionAt(destinations[0])
: null;
if (target != null && target.hasNoReturn()) {
limitation.append(" This records Ghidra's FunctionManager flag, not an independent verification that the callee cannot return.");
}
else {
limitation.append(" The reason for the terminal flow is not established by this observation.");
}
}
limitations.add(limitation.toString());
}
return limitations;
}
private static String renderAssembly(List<Instruction> instructions) { private static String renderAssembly(List<Instruction> instructions) {
return String.join("\n", renderAssemblyLines(instructions)); return String.join("\n", renderAssemblyLines(instructions));
} }
@@ -3476,7 +3792,9 @@ public final class ReaGhidraBridge extends HeadlessScript {
.thenComparing(Reference::isPrimary); .thenComparing(Reference::isPrimary);
private record InventoryItem(Address address, String value, JsonObject facts) {} private record InventoryItem(Address address, String value, JsonObject facts) {}
private record AddressNameRequest(String inputAddress, Address address, String name) {}
private record FunctionEntry(Function function, InventoryItem item) {} private record FunctionEntry(Function function, InventoryItem item) {}
private record FunctionAnalysisObservation(JsonObject dossier, List<String> limitations) {}
private record InstructionScan(List<Instruction> instructions, boolean truncated) {} private record InstructionScan(List<Instruction> instructions, boolean truncated) {}
private record SessionDescriptor( private record SessionDescriptor(
String transport, String transport,
+74 -40
View File
@@ -3,20 +3,25 @@ import ghidra.app.script.GhidraScript;
import ghidra.program.model.address.Address; import ghidra.program.model.address.Address;
import ghidra.program.model.listing.*; import ghidra.program.model.listing.*;
import ghidra.program.model.symbol.FlowType; import ghidra.program.model.symbol.FlowType;
import ghidra.program.model.symbol.Reference;
import ghidra.program.model.pcode.PcodeOp; import ghidra.program.model.pcode.PcodeOp;
import java.util.ArrayList; import java.util.ArrayDeque;
import java.util.Deque;
import java.util.HashSet; import java.util.HashSet;
import java.util.List;
import java.util.Set; import java.util.Set;
import java.util.TreeSet; import java.util.TreeSet;
/** /**
* Undo wrong no-return flags from auto-analysis. A function flagged no-return that contains a * Undo wrong no-return flags from auto-analysis. A function flagged no-return that contains a
* real RET is an ordinary function (stream readers, cxa_guard_acquire, ...). Callers treated the * real RET is an ordinary function (stream readers, cxa_guard_acquire, ...). Known returning
* call as a terminator and the decompiler truncated them. Clear the flag, then re-disassemble only * libc/TLS imports have no body to inspect, so clear their flags by exact external ABI name.
* Callers treated the call as a terminator and the decompiler truncated them. Clear the flag, then re-disassemble only
* the call sites of the functions that were cleared and refit their owning functions in place. * the call sites of the functions that were cleared and refit their owning functions in place.
*/ */
public final class ReaGhidraNoReturnFix extends GhidraScript { public final class ReaGhidraNoReturnFix extends GhidraScript {
private static final Set<String> RETURNING_IMPORTS = Set.of(
"malloc", "__tls_get_addr", "memcpy", "memset", "strlen", "__errno_location");
@Override @Override
public void run() throws Exception { public void run() throws Exception {
// Windows P0 does not admit analysis-database mutation. // Windows P0 does not admit analysis-database mutation.
@@ -24,66 +29,95 @@ public final class ReaGhidraNoReturnFix extends GhidraScript {
FunctionManager fm = currentProgram.getFunctionManager(); FunctionManager fm = currentProgram.getFunctionManager();
Listing li = currentProgram.getListing(); Listing li = currentProgram.getListing();
Set<Address> cleared = new HashSet<>(); Set<Address> cleared = new HashSet<>();
for (Function f : fm.getFunctions(true)) { Deque<Address> pending = new ArrayDeque<>();
for (Function f : fm.getExternalFunctions()) {
monitor.checkCancelled(); monitor.checkCancelled();
if (!f.hasNoReturn() || f.isThunk()) continue; // Do not apply import-name assumptions to local functions or to arbitrary externals.
for (Instruction ins : li.getInstructions(f.getBody(), true)) { if (!f.hasNoReturn() || !RETURNING_IMPORTS.contains(externalAbiName(f))) continue;
FlowType ft = ins.getFlowType(); f.setNoReturn(false);
boolean decodedReturn = false; enqueueReturning(f, cleared, pending);
for (PcodeOp operation : ins.getPcode()) {
if (operation.getOpcode() == PcodeOp.RETURN) decodedReturn = true;
}
if (decodedReturn && !ft.isJump() && !ft.isCall()) {
f.setNoReturn(false);
cleared.add(f.getEntryPoint());
break;
}
}
} }
// A thunk inherits the flag of the function it forwards to.
for (Function f : fm.getFunctions(true)) { for (Function f : fm.getFunctions(true)) {
monitor.checkCancelled(); monitor.checkCancelled();
if (!f.isThunk()) continue; if (clearDecodedReturn(f, li)) {
Function target = f.getThunkedFunction(true); enqueueReturning(f, cleared, pending);
if (target != null && cleared.contains(target.getEntryPoint())) {
cleared.add(f.getEntryPoint());
} }
} }
int redone = 0; int redone = 0;
TreeSet<Address> owners = new TreeSet<>(); TreeSet<Address> owners = new TreeSet<>();
if (!cleared.isEmpty()) { while (!pending.isEmpty()) {
List<Address> sites = new ArrayList<>(); Address target = pending.removeFirst();
for (Instruction ins : li.getInstructions(true)) { TreeSet<Address> sites = new TreeSet<>();
// References include computed import calls after Ghidra resolves their destination.
// Collect before redisassembly mutates the reference table.
for (Reference reference : currentProgram.getReferenceManager().getReferencesTo(target)) {
monitor.checkCancelled(); monitor.checkCancelled();
FlowType ft = ins.getFlowType(); sites.add(reference.getFromAddress());
if (!(ft.isCall() && ft.isTerminal())) continue;
Address[] flows = ins.getFlows();
if (flows.length == 0 || !cleared.contains(flows[0])) continue;
// Do not redisassemble into an independently identified function.
Address next = ins.getMaxAddress().add(1);
if (fm.getFunctionAt(next) != null || !currentProgram.getMemory().contains(next)) continue;
sites.add(ins.getAddress());
} }
for (Address s : sites) { for (Address s : sites) {
monitor.checkCancelled(); monitor.checkCancelled();
Instruction ins = li.getInstructionAt(s); Instruction ins = li.getInstructionAt(s);
if (ins == null) continue; if (ins == null) continue;
FlowType ft = ins.getFlowType();
if (!(ft.isCall() && ft.isTerminal())) continue;
Address[] flows = ins.getFlows();
boolean callsTarget = false;
for (Address flow : flows) {
if (flow.equals(target)) callsTarget = true;
}
if (!callsTarget) continue;
// Do not redisassemble into an independently identified function.
Address next = ins.getMaxAddress().add(1);
if (fm.getFunctionAt(next) != null || !currentProgram.getMemory().contains(next)) continue;
Function owner = fm.getFunctionContaining(s); Function owner = fm.getFunctionContaining(s);
if (owner != null && !owner.isThunk()) owners.add(owner.getEntryPoint()); if (owner != null && !owner.isThunk()) owners.add(owner.getEntryPoint());
clearListing(ins.getMinAddress(), ins.getMaxAddress()); clearListing(ins.getMinAddress(), ins.getMaxAddress());
disassemble(s); disassemble(s);
redone++; redone++;
}
for (Address e : owners) {
monitor.checkCancelled();
Function f = fm.getFunctionAt(e);
// Refit the body in place so the name, signature and calling convention survive. // Refit the body in place so the name, signature and calling convention survive.
if (f != null) CreateFunctionCmd.fixupFunctionBody(currentProgram, f, monitor); if (owner != null && !owner.isThunk()) {
CreateFunctionCmd.fixupFunctionBody(currentProgram, owner, monitor);
// Auto-analysis may have propagated the false flag to this truncated caller.
// Its RET becomes visible only now; repair its callers in the same pass.
if (clearDecodedReturn(owner, li)) enqueueReturning(owner, cleared, pending);
}
} }
// No analyzeChanges here: on libpl2.so it adds about 70 s, which pushes startup past the 330 s deadline.
} }
// No analyzeChanges here: on libpl2.so it adds about 70 s, which pushes startup past the 330 s deadline.
println( println(
"REA no-return fix: cleared=" + cleared.size() + " sites=" + redone + " refit=" + owners.size()); "REA no-return fix: cleared=" + cleared.size() + " sites=" + redone + " refit=" + owners.size());
} }
private void enqueueReturning(Function function, Set<Address> cleared, Deque<Address> pending) {
if (cleared.add(function.getEntryPoint())) pending.addLast(function.getEntryPoint());
// A thunk inherits the flag of the function it forwards to, including through other thunks.
Address[] thunks = function.getFunctionThunkAddresses(true);
if (thunks == null) return;
for (Address thunk : thunks) {
if (cleared.add(thunk)) pending.addLast(thunk);
}
}
private String externalAbiName(Function function) {
String name = function.getName();
// Mach-O stores C-linkage symbols with one object-format underscore prefix.
if (currentProgram.getExecutableFormat().contains("Mach-O") && name.startsWith("_"))
return name.substring(1);
return name;
}
private boolean clearDecodedReturn(Function function, Listing listing) throws Exception {
if (!function.hasNoReturn() || function.isThunk()) return false;
for (Instruction instruction : listing.getInstructions(function.getBody(), true)) {
monitor.checkCancelled();
FlowType flow = instruction.getFlowType();
if (flow.isJump() || flow.isCall()) continue;
for (PcodeOp operation : instruction.getPcode()) {
if (operation.getOpcode() != PcodeOp.RETURN) continue;
function.setNoReturn(false);
return true;
}
}
return false;
}
} }
+11
View File
@@ -53,8 +53,19 @@ google-chrome \
REA does not launch, own, or terminate this browser. Use a dedicated profile and stop it yourself when the investigation is complete. Do not expose the debugging port on a non-loopback interface. REA does not launch, own, or terminate this browser. Use a dedicated profile and stop it yourself when the investigation is complete. Do not expose the debugging port on a non-loopback interface.
On Linux, a reachable but locked desktop keyring can stall Chrome before its
first navigation. For a dedicated investigation profile, add
`--password-store=basic` to bypass that keyring. Empty or placeholder frame URLs
then describe an unloaded page; REA continues to require an allowed, committed
URL. See [the startup investigation](https://github.com/morluto/rea/issues/1384#issuecomment-6093781686).
## Request boundary ## Request boundary
CDP connections enforce the shared 64 MiB limit per WebSocket message. A larger
reply, including a valid reply from Chrome, fails with `payload_limit`. The limit
counts message bytes before JSON parsing; it does not bound the memory used by
decoded strings or JSON objects.
Discovery supplies a literal loopback CDP endpoint and returns all eligible Discovery supplies a literal loopback CDP endpoint and returns all eligible
page targets unless an exact HTTP(S) origin filter narrows the result. Follow-up page targets unless an exact HTTP(S) origin filter narrows the result. Follow-up
inspection requests select one target; their default scope is its current origin. `localhost`, private-LAN inspection requests select one target; their default scope is its current origin. `localhost`, private-LAN
+12
View File
@@ -156,6 +156,18 @@ Storage cookies are scoped to the page URL at each capture. Unsupported action
tags, unknown fields, duplicate step IDs, and undeclared secret references fail tags, unknown fields, duplicate step IDs, and undeclared secret references fail
validation. validation.
Cookies are installed before the initial navigation. Local storage entries are
seeded once per browser storage partition; session storage entries are seeded
once per origin in each top-level browsing context. Seeding occurs before the
first document scripts for that area. An origin first reached by a later action
receives its seed on that first visit. Application updates, deletions, and
empty values then persist across later navigations in the same area; a new
scenario starts with its own seeds. Partitioned frames can receive separate
local storage seeds even when they share an origin. The pre-script mechanism
uses CDP debugger instrumentation. The observer resumes only pauses whose top
frame belongs to its private seed script; page breakpoints and `debugger`
statements remain under the attached debugger client's control.
The result is Evidence with an initial state followed by one record per The result is Evidence with an initial state followed by one record per
declared action. Each step reports action status, elapsed time, sanitized URLs, declared action. Each step reports action status, elapsed time, sanitized URLs,
event bounds, and independently typed capture states for screenshot, DOM, event bounds, and independently typed capture states for screenshot, DOM,
+64 -35
View File
@@ -1,16 +1,25 @@
# Optional NativeAOT metadata recovery # Optional NativeAOT metadata recovery
REA integrates [Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot) REA has a built-in read-only snapshot parser for x64 PE targets with .NET 8 RTR
through an optional headless adapter. The original source is an opt-in submodule, 9.1 metadata. It parses the exact admitted Ghidra session snapshot and uses an
pinned to `effeb734fc570c32650f88b159608979dc7b423e`; its MIT license and existing in-memory overlay for rehydrated metadata; it does not define Ghidra DataTypes,
source notices are retained. No upstream source or algorithms are rewritten. labels, functions, or memory bytes. The hydration semantics are adapted from
[Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot) at
`effeb734fc570c32650f88b159608979dc7b423e` (MIT), cross-checked against
.NET's `MethodTable.h` at runtime commit
`a3b2d40328be0be3f8dd43f0e5cc925b8827b044` (MIT). The required notice ships in
[`licenses/nativeaot-parser-MIT.txt`](../licenses/nativeaot-parser-MIT.txt).
The initial tested layout is .NET runtime **8.0.22**, NativeAOT RTR **9.1**, The optional headless Ghidra adapter remains available for database annotations
x86-64 ELF and native Windows PE **targets analyzed on Linux x64**, using and broader recovery. It uses the same pinned upstream commit as an opt-in
Ghidra **12.1.4** and JDK **21**. The build accepts any Ghidra 12.1.x release and submodule and is a separate, more capable workflow.
javac 21 or newer, and it compiles with `--release 21`. Real verification of
this layout remains the Ghidra 12.1.4 and JDK 21 pair. Other NativeAOT layouts, The parser currently handles x64 PE32+ and RTR 9.1, including bounded
architectures, and hosts are unsupported. DEHYDRATED_DATA processing, MethodTable relationships, virtual slots, and
frozen strings. Other RTR versions and PE architectures are reported as
unsupported or partial. The matching .NET 8.0.22 fixture is source-built and
the parser's Windows-host Ghidra workflow has not yet had a real Windows-host
run. This does not establish Windows Ghidra import or decompiler behavior.
Windows Ghidra P0 has no database mutation authority. A PE/CLI or ReadyToRun Windows Ghidra P0 has no database mutation authority. A PE/CLI or ReadyToRun
assembly should use `inspect_managed_artifact`; it is not NativeAOT. assembly should use `inspect_managed_artifact`; it is not NativeAOT.
@@ -43,42 +52,62 @@ arbitrary caller-supplied bytes. The bounded loader accepts regular JARs up to
## Use existing tools ## Use existing tools
Open the executable with provider `ghidra`. `inspect_native_load_image` returns Open the executable with provider `ghidra`. For an x64 PE, `inspect_native_load_image`
optional `observations.metadata_recovery` inline, including the detected header, returns `observations.metadata_recovery` inline from the captured source snapshot,
layout, recovery status, type category paths/addresses, diagnostics, annotation including the RTR header, MethodTable addresses, frozen strings, derived overlay
coverage and derived-memory SHA. The enclosing `unsupported` load-image status identity, and examined/recovered counts. Its enclosing `unsupported` load-image
for PE/ELF concerns independent DOS loader verification; metadata recovery has status means independent PE load-image verification was not performed; it does
its own status. It is not a failed import. not mean the metadata parser failed or the executable failed to load.
Pass one returned type path or its typed address to `inspect_native_data_type`. Pass a returned MethodTable address to `inspect_native_data_type`. A direct PE
Its optional `metadata_recovery` includes related/base type, implemented query also derives the snapshot metadata from the authenticated Ghidra session
interfaces and virtual slots with targets and known procedure names. Follow handshake, so it does not require a preceding load-image call. In a persistent
these with `analyze_function`, string searches and ordinary cross-references. MCP session, load-image followed by type inspection reuses that same derivation.
For CLI use: The database result remains `unavailable` when Ghidra has no DataType at that
address; its `metadata_recovery.source` is `read-only-derived-overlay` and
carries related type, interface, slot, and target observations. Generated
address identities do not claim original managed names. The parser does not
merge frozen literals into Ghidra's database string inventory. Follow slot
targets with `analyze_function` and ordinary cross-references.
The built-in CLI path uses a MethodTable address returned by the parser or
another observation:
```sh ```sh
rea inspect-native-load-image ./NativeAotFixture --provider ghidra --json rea inspect-native-load-image ./NativeAotFixture --provider ghidra --json
rea inspect-native-data-type ./NativeAotFixture --type /NativeAOT/MethodTables/Class_ADDRESS_MT --provider ghidra --json rea inspect-native-data-type ./NativeAotFixture --address 0xMETHODTABLE_ADDRESS --provider ghidra --json
rea function ./NativeAotFixture 0xADDRESS --provider ghidra --json rea function ./NativeAotFixture 0xADDRESS --provider ghidra --json
``` ```
No new MCP workflow or approval flag is required. Configuration opts into the The generated `/NativeAOT/...` category path belongs to the optional headless
adapter; without it ordinary Ghidra profiles/results remain unchanged. adapter. Use `--type /NativeAOT/MethodTables/Class_ADDRESS_MT` only when that
adapter is configured and has created the corresponding Ghidra DataType.
Built-in PE analysis includes its parser contract and resource policy in the
Ghidra analysis profile; non-PE profiles do not include that contract.
## Evidence boundaries ## Evidence boundaries
Discovery prefers an imported RTR symbol, otherwise the upstream initialized, The built-in parser uses an initialized non-executable PE signature scan and
non-executable-memory signature heuristic. REA rejects ambiguous candidates, requires a unique RTR 9.1 header and exactly one DEHYDRATED_DATA row. Rehydrated
unknown layout versions, malformed directory rows, unsupported flags and bytes live only in a parser-owned overlay, identified by virtual address, size,
unmapped ranges before recovery. It requires the verified dehydrated layout. and SHA-256; the caller's source Buffer remains unchanged. Limits derive from
captured section extents and format field bounds. Separate resource policies
admit up to 128 MiB of captured PE bytes, 64 MiB-equivalent parser work, 64 MiB
of exact parser-owned overlay/index Buffers, and 4 MiB of serialized NativeAOT
report JSON. These are local parser limits, independent of MCP delivery; the
buffer counter does not claim to cap V8 heap or total process memory. Rows that
do not fit the report budget are omitted with explicit partial status and
coverage. Coverage counts describe the pointer scan, recovered MethodTables,
and frozen-string candidates examined, not every possible runtime type or
object. Cancellation is honored during snapshot I/O and checked immediately
before and after parsing; the synchronous parser call itself is bounded but
cannot be interrupted mid-call. Parser output never claims that a Ghidra
database type was defined.
The adapter performs rehydration, method-table recovery and frozen-object The optional adapter performs its annotations in an ephemeral Program
annotation in one ephemeral Program transaction. Fatal failure/cancellation transaction. That workflow may change analysis-memory bytes, never the original
rolls it back. Recovery may change analysis-memory bytes, never the original executable file. It is unavailable on Windows P0, which has no database
executable file. Derived ranges have a digest and **no original file offset**; mutation authority.
they are not captured runtime state. Candidate/committed instance annotation
counts distinguish partial frozen-object coverage. Coverage concerns the
recovered candidate set, not every possible object or every runtime type.
REA preserves pre-recovery calling conventions and uses the loaded compiler REA preserves pre-recovery calling conventions and uses the loaded compiler
specification default for newly created methods. The upstream universal specification default for newly created methods. The upstream universal
+34 -4
View File
@@ -8,9 +8,12 @@ Executable bytes remain unchanged.
Use [installation](installation.md#ghidra) for supported hosts, prerequisites, Use [installation](installation.md#ghidra) for supported hosts, prerequisites,
configuration, startup deadlines, resource controls, and cleanup recovery. configuration, startup deadlines, resource controls, and cleanup recovery.
[Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary; [Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary;
[NativeAOT recovery](ghidra-nativeaot.md) is an optional, separately verified [NativeAOT observations](ghidra-nativeaot.md) include a built-in read-only PE
extension. Check the [release boundary](installation.md#released-package-and-main) snapshot parser and a separate optional annotation extension. Check the
when using npm. [release boundary](installation.md#released-package-and-main) when using npm.
See [import language and analysis seeds](installation.md#ghidra-import-language-and-analysis-seeds)
to override the header-detected language or seed analysis.
Provider selection and profile identity follow Provider selection and profile identity follow
[MCP contracts](mcp-contracts.md#identity-and-discovery). One deep [MCP contracts](mcp-contracts.md#identity-and-discovery). One deep
@@ -29,9 +32,15 @@ not equivalence with another engine's pseudocode or analysis database.
| Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. | | Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. |
| Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. | | Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. |
| Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. | | Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. |
| Search | Literal and Java-regex searches scan the complete immutable inventory and return all matching entries inline. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. | | Search | Literal and Java-regex searches return every match in the immutable provider-defined inventory; string search does not cover every text sequence in artifact bytes. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. |
| Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. | | Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. |
String-search completeness applies only to Ghidra-defined string `Data`. Packed
or undefined literals, including Rust strings, may be absent; an empty result
does not prove text is absent. Both list and search Evidence carry this
limitation. When an address is known, use `read_bytes` and `xrefs` to inspect
bytes and references.
## Admitted function-analysis semantics ## Admitted function-analysis semantics
| Concern | Ghidra contract | | Concern | Ghidra contract |
@@ -46,6 +55,27 @@ not equivalence with another engine's pseudocode or analysis database.
| P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. | | P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. |
| Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. | | Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. |
Function extents and instruction completeness refer to Ghidra's analyzed body.
An inferred no-return call can terminate that body before subsequent bytes.
Pseudocode, assembly, procedure info, instruction windows, and dossiers report
observed terminal call sites in Evidence limitations, including resolved callee
names, addresses, and FunctionManager flags when available. Instruction windows
and dossiers also retain these warnings in their result limitations. An
unresolved callee or unestablished terminal-flow basis remains explicit. These
observations qualify the analysis; they do not prove that a no-return flag is
incorrect or recover excluded code. Analysis profiles version this observation
contract so older snapshots cannot replay unqualified results.
On non-Windows hosts, the ephemeral import repair clears no-return flags on the
known returning external ABI names `malloc`, `__tls_get_addr`, `memcpy`, `memset`,
`strlen`, and `__errno_location`. This relies on their expected ABI behavior;
external implementations are not inspected. The repair re-disassembles affected
terminal calls and refits their owning functions, preserving independent function
entries. If a recovered caller reveals a decoded return, its propagated no-return
flag and affected callers are repaired in the same pass. Unknown imports and
genuine no-return imports keep their flags. Analysis profiles version this repair
so snapshots produced before it are not reused. Windows P0 does not run it.
`npm run verify:ghidra` compiles the versioned C oracle into debug and stripped `npm run verify:ghidra` compiles the versioned C oracle into debug and stripped
host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native
DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name
+38 -9
View File
@@ -68,13 +68,15 @@ selects the npm release, and persistent MCP registrations are pinned to the
version that performed setup. Installing newer instructions does not update a version that performed setup. Installing newer instructions does not update a
running server or its registration. running server or its registration.
The npm release checked on **2026-10-10** was **6.3.0**, published from The npm release checked on **2026-10-11** was **6.3.0**, published from
[`74cf9e14`](https://github.com/morluto/rea/commit/74cf9e1401fcd89482fe39c27cc898f5858e9b72), [`74cf9e14`](https://github.com/morluto/rea/commit/74cf9e1401fcd89482fe39c27cc898f5858e9b72),
with bundled skill version 34. It does not include automatic Pi or Hermes with bundled skill version 34. It does not include automatic Qwen Code, Pi, or
registration: `setup --client pi` and `setup --client hermes` reject those IDs Hermes registration: `setup --client qwen_code`, `setup --client pi`, and
before writing configuration. Their setup workflows below require a source build `setup --client hermes` reject those IDs before writing configuration. It also
or a later release containing those integrations; `@latest` alone cannot select does not honor `OPENCODE_CONFIG_DIR` when selecting OpenCode's configuration
unpublished changes. directory. These workflows below require a source build or a later release
containing the relevant changes; `@latest` alone cannot select unpublished
changes.
The earlier release checked on **2026-10-07** was **5.0.0** (133 MCP tools), published The earlier release checked on **2026-10-07** was **5.0.0** (133 MCP tools), published
from the fixed checkpoint from the fixed checkpoint
@@ -199,7 +201,10 @@ but unavailable in a project. An extension that registers `/mcp` can replace
Pi's built-in MCP support and use its own configuration instead. Run `/reload` Pi's built-in MCP support and use its own configuration instead. Run `/reload`
in an existing Pi session after changing servers outside the session. in an existing Pi session after changing servers outside the session.
For Qwen Code, setup registers `rea` in the `mcpServers` table of the For Qwen Code, automatic setup requires a build containing the integration;
npm 6.3.0 does not include it. See
[released package and main](#released-package-and-main). In a supported build,
setup registers `rea` in the `mcpServers` table of the
user-level `~/.qwen/settings.json`, or `$QWEN_HOME/settings.json` when configured. user-level `~/.qwen/settings.json`, or `$QWEN_HOME/settings.json` when configured.
Setup and doctor follow Qwen Code's tilde and working-directory-relative home Setup and doctor follow Qwen Code's tilde and working-directory-relative home
overrides. Qwen Code also discovers personal skills in `~/.agents/skills`, overrides. Qwen Code also discovers personal skills in `~/.agents/skills`,
@@ -288,7 +293,10 @@ for its native configuration contract.
### Other client settings ### Other client settings
For OpenCode, setup writes the V1 `mcp.rea` entry, which OpenCode V1 and V2 For OpenCode, npm 6.3.0 does not honor `OPENCODE_CONFIG_DIR`; use a source build
or a later release containing the override support when selecting a custom
configuration directory. See [released package and main](#released-package-and-main).
In a supported build, setup writes the V1 `mcp.rea` entry, which OpenCode V1 and V2
both load. If the configuration already uses OpenCode V2's native both load. If the configuration already uses OpenCode V2's native
`mcp.servers` table, setup registers REA there instead and replaces any earlier `mcp.servers` table, setup registers REA there instead and replaces any earlier
`mcp.rea` entry from REA. Setup honors `OPENCODE_CONFIG_DIR` and `mcp.rea` entry from REA. Setup honors `OPENCODE_CONFIG_DIR` and
@@ -455,7 +463,7 @@ vendor-defined limits, and a paid license is optional. REA reuses any detected
installation and preserves Hopper during uninstall. installation and preserves Hopper during uninstall.
The supported native host baseline is macOS 12+, Ubuntu 24.04+, Fedora 41+, The supported native host baseline is macOS 12+, Ubuntu 24.04+, Fedora 41+,
Nobara 44+, 64-bit Arch Linux, or CachyOS. Ghidra and IDA have their own provider-specific Nobara 44+, 64-bit Arch Linux, CachyOS, or Omarchy. Ghidra and IDA have their own provider-specific
host requirements; Windows Ghidra uses the [experimental P0 boundary](windows-ghidra-p0.md). host requirements; Windows Ghidra uses the [experimental P0 boundary](windows-ghidra-p0.md).
On macOS, approved setup downloads the official DMG, checks its published size On macOS, approved setup downloads the official DMG, checks its published size
@@ -725,6 +733,27 @@ is no fixed per-operation or response-size ceiling. Unresolved computed calls
remain unknown, reference-kind provenance is preserved, and provider-specific remain unknown, reference-kind provenance is preserved, and provider-specific
pseudocode is never treated as original source or Hopper-equivalent text. pseudocode is never treated as original source or Hopper-equivalent text.
### Ghidra import language and analysis seeds
Ghidra detects the language from the executable header. To set it (for example, Borland C++ PE images):
```bash
export REA_GHIDRA_LANGUAGE_ID="x86:LE:32:default"
export REA_GHIDRA_COMPILER_SPEC_ID="borlandcpp" # optional; requires a language
```
REA passes `-processor`/`-cspec`, checks both against the installation's `.ldefs`, and records them in the analysis profile. DOS imports reject an override.
To seed code that auto-analysis cannot reach (computed jumps, pointers in data):
```bash
export REA_GHIDRA_SEED_FILE=/absolute/path/to/seeds.tsv
```
Each line is `address<TAB>kind[<TAB>name]`, where kind is `function`, `code` or `label` (a label needs a name). Addresses are `0x` hex in the default space; `#` lines are comments. A pre-script applies seeds before auto-analysis: functions from the highest address down, then code, then labels. The seed digest is part of the analysis profile, and the session fails if the applied seeds differ. Result limitations report the outcome counts; a seed fails when it overlaps an instruction Ghidra already decoded.
Label seeds can name mapped uninitialized storage such as `.bss`; function and code seeds require initialized bytes.
### Ghidra heap and CPU controls ### Ghidra heap and CPU controls
Set resource controls in the environment that launches the CLI or MCP server, Set resource controls in the environment that launches the CLI or MCP server,
+97 -3
View File
@@ -60,6 +60,12 @@ manifest and graph commitments, JavaScript Application Graph, static
Electron summary, reconstruction statistics, and explicit limitations. It does Electron summary, reconstruction statistics, and explicit limitations. It does
not require a live Hopper, Ghidra, browser, or Electron process. not require a live Hopper, Ghidra, browser, or Electron process.
The application graph selects the shallowest `package.json` outside
`node_modules` as its package root, breaking equal-depth ties by Unicode
code-point order. Dependency manifests remain package observations but do not
declare Electron main or renderer entries. If only dependency manifests are
present, the inventory artifact remains the graph root.
## ASAR integrity ## ASAR integrity
ASAR inventory checks Electron integrity metadata for embedded archive entries ASAR inventory checks Electron integrity metadata for embedded archive entries
@@ -89,9 +95,9 @@ queries use per-file range indexes, while call sites use exact-range lookups.
Existing node budgets and coverage reporting remain in effect. Existing node budgets and coverage reporting remain in effect.
Progress identifies the source file being parsed and projected. Cancellation is Progress identifies the source file being parsed and projected. Cancellation is
checked between files, with event-loop yields that allow notifications and checked between files and throughout result transfer. The owned analyzer can
completed file-local allocations to be released. A single file's synchronous be stopped while its synchronous parser or semantic extractor is active;
parser and semantic extraction still need to finish before that boundary. the CLI/MCP process remains available for control requests.
Graph and Evidence identifiers hash canonical JSON incrementally, without Graph and Evidence identifiers hash canonical JSON incrementally, without
assembling a single string for the whole graph. The canonical bytes and existing assembling a single string for the whole graph. The canonical bytes and existing
@@ -113,6 +119,46 @@ smaller view, for example `--format json --filter-output
evidence_id,normalized_result.statistics`. Streaming output does not bound the evidence_id,normalized_result.statistics`. Streaming output does not bound the
memory needed to construct the analysis graph itself. memory needed to construct the analysis graph itself.
JavaScript source parsing, static analysis and semantic projection run in one
owned analysis process, reused sequentially across files. Target code is never
executed. The analyzer defaults to 1024 MiB of V8 old-space and a 300000 ms
deadline per source, including startup and fact transfer. Select
`max_heap_mb` and `analysis_timeout_ms` through MCP, or `--max-heap-mb` and
`--analysis-timeout-ms` on the dedicated CLI command. These choices control the
analyzer; the caller's `NODE_OPTIONS` still controls the CLI/MCP process heap.
V8 old-space limits do not impose a process RSS limit.
A source that exhausts the analyzer heap does not terminate the CLI/MCP
process. Completed static facts are retained, other files can still be analyzed,
and unexamined semantic relationships remain explicit unknowns. The parent
also checks transfer expansion and accumulated results against its remaining
heap before accepting semantic facts. A resource-limited result has partial
coverage, the observed limits and an `analyze-javascript-source` unknown scope
with the specific failure, including captured diagnostics when available.
Resource failures do not increment syntax parse failures. There is no fixed
source-length cutoff.
Static observations also need memory when expanded into nodes, relationships
and Evidence. If that expansion does not fit, the file node retains
`static_analysis` and any admitted `semantic_module_analysis` observations,
including their exact values and source locations. Use the public modules item
view to read them. The graph marks those unexpanded relationships as unknown.
If cumulative result capacity is exhausted, analysis stops; inventoried but
unexamined files remain present with explicit unknown scopes.
A timeout stops the remaining analysis and returns typed failure details with
completed partial Evidence. MCP failure details return
`partial_observation: { kind: "retained-evidence", evidence_id: "..." }`
after recording succeeds, so `inspect_analysis_view` can open that analysis
directly without serializing the entire partial graph into the error response.
CLI failure output retains the complete partial Evidence.
Cancellation after source analysis begins likewise
retains completed facts; an SDK client may reject its cancelled request before
receiving that reply. On the same MCP connection, inspect the Evidence bundle
and use the retained analysis ID with `inspect_analysis_view`. Owned analyzer
cleanup is verified before its private files are released. Cleanup uncertainty
is reported separately and retains ownership for retry.
CLI workflows parse larger JSON input files incrementally from one verified CLI workflows parse larger JSON input files incrementally from one verified
regular-file handle, without constructing a document-sized string. Smaller regular-file handle, without constructing a document-sized string. Smaller
files retain native JSON parsing. Both paths require strict JSON and valid files retain native JSON parsing. Both paths require strict JSON and valid
@@ -207,6 +253,46 @@ while locally shadowed receivers can still contribute network endpoint
candidates. A template recovered with a parser error and no cooked value stays candidates. A template recovered with a parser error and no cooked value stays
dynamic; its raw spelling is not treated as a valid JavaScript string. dynamic; its raw spelling is not treated as a valid JavaScript string.
RPC client syntax also contributes network endpoint candidates, in both
`analyze_javascript_application` and `analyze_web_bundle`, distinguished by
`mechanism`:
- `trpc:procedure:<method>` records the static member path between the root
expression and the method, such as `invoice.list` in
`root.invoice.list.useQuery(...)`. tRPC proxies keep these property names
through identifier minification. React hooks and TanStack option factories
(`useQuery`, `useMutation`, `useInfiniteQuery`, `useSubscription`, suspense
and prefetch variants, `queryOptions`, `infiniteQueryOptions`,
`mutationOptions`, `subscriptionOptions`) are recognized directly. The
vanilla `query` and `mutate` methods are recognized on a root bound to
`createTRPCClient` or `createTRPCProxyClient` in the same lexical binding, or on a
path of at least two segments in a source containing `TRPCClientError`,
`trpc-accept`, or `/trpc`. Factory bindings must have one unconditional
initializer and no binding assignments; shadowed locals do not inherit client
identity. `subscribe` additionally requires a tRPC observer
argument (`onData`, `onError`, `onStarted`, `onStopped`, `onComplete`, or
`onConnectionStateChange`). `this` roots, browser globals, and any path
containing an RTK Query `endpoints` segment are never procedures; cache-key
helpers such as `queryKey` issue no request and are excluded.
- `trpc:link:<link>` records the literal `url` option of `httpLink`,
`httpBatchLink`, `httpBatchStreamLink`, and `httpSubscriptionLink` when the
link's callee name survives bundling, including `(0, module.httpBatchLink)`
import-call wrappers.
- `graphql:query`, `graphql:mutation`, and `graphql:subscription` record a
named operation found at the start of a string or cooked template literal —
the name must be followed by variable definitions (`($`), a directive, or a
selection set starting with a field name — or in a precompiled
`OperationDefinition` document object. A graphql-tag `loc.source.body` copy is
not reported twice. Anonymous operations are not reported.
A procedure path is the server-side procedure name, not a resolved URL; the
transport URL depends on the link configuration and batching. When the proxy is
reached through a member expression, such as a webpack module binding
(`l.S.invoice.list`) or a property holding the proxy (`o.api.post.byId`), the
recorded path keeps those leading carrier segments; the procedure path is its
suffix. Utility calls such as `useUtils().invoice.list.fetch()`, `useQueries`,
and links whose callee was renamed by a minifier are not recognized.
Each recovered bundle module retains the exact factory-source digest. A complete Each recovered bundle module retains the exact factory-source digest. A complete
bounded AST also receives a `babel-ast-v1` structural fingerprint that ignores bounded AST also receives a `babel-ast-v1` structural fingerprint that ignores
ordinary identifier names while retaining syntax, literals, operators, object ordinary identifier names while retaining syntax, literals, operators, object
@@ -230,6 +316,14 @@ substitute version-dependent Electron defaults for omitted `webPreferences`.
Dynamic option objects, bridge keys, API objects, and IPC channel expressions Dynamic option objects, bridge keys, API objects, and IPC channel expressions
remain unknown and make coverage partial. remain unknown and make coverage partial.
Electron API identity follows proven lexical module aliases. Reassigned bindings,
dynamic selections, and visibly overwritten namespace members are unresolved.
Direct and destructuring assignment, loop, update, and delete targets are checked
through namespace aliases; unrelated member writes do not discard observed API
identity. This static check does
not evaluate arbitrary call side effects or runtime registration. Missing findings
do not establish that an application has no Electron boundary usage.
IPC pairing is an inference, not an observation. A renderer `invoke` pairs only IPC pairing is an inference, not an observation. A renderer `invoke` pairs only
with one unique `ipcMain.handle`/`handleOnce` candidate on the same exact literal with one unique `ipcMain.handle`/`handleOnce` candidate on the same exact literal
channel; a renderer send pairs only with one unique `ipcMain.on`/`once` channel; a renderer send pairs only with one unique `ipcMain.on`/`once`
+31 -7
View File
@@ -1,7 +1,7 @@
# JavaScript source recovery # JavaScript source recovery
`recover_javascript_sources` / `recover-javascript-sources` derives readable `recover_javascript_sources` / `recover-javascript-sources` derives readable
modules from one explicitly selected UTF-8 JavaScript file or bundle. Use it on modules from explicitly selected UTF-8 JavaScript files or bundles. Use it on
files returned by `export_web_scripts` or other local scripts. The operation files returned by `export_web_scripts` or other local scripts. The operation
never executes the selected or recovered application code. never executes the selected or recovered application code.
@@ -52,35 +52,58 @@ Both give the same binary (SHA-256
```sh ```sh
rea recover-javascript-sources /tmp/web-scripts/app.js /tmp/recovered-app --json rea recover-javascript-sources /tmp/web-scripts/app.js /tmp/recovered-app --json
rea analyze-javascript-application /tmp/recovered-app/modules --json rea analyze-javascript-application /tmp/recovered-app/modules --json
# Recover related code-split chunks together; repeat --additional-path.
rea recover-javascript-sources /tmp/web-scripts/main.js /tmp/recovered-chunks \
--additional-path /tmp/web-scripts/runtime.js \
--additional-path /tmp/web-scripts/shared.js \
--additional-path /tmp/web-scripts/checkout.chunk.js --json
``` ```
MCP input: MCP input:
```json ```json
{ {
"path": "/tmp/web-scripts/app.js", "path": "/tmp/web-scripts/main.js",
"output_directory": "/tmp/recovered-app" "additional_paths": [
"/tmp/web-scripts/runtime.js",
"/tmp/web-scripts/shared.js",
"/tmp/web-scripts/checkout.chunk.js"
],
"output_directory": "/tmp/recovered-chunks"
} }
``` ```
The output directory must be absent, absolute, and have an existing parent. The output directory must be absent, absolute, and have an existing parent.
The source is snapshotted in a private workspace, and the engine writes only to Each source is snapshotted separately in a private workspace. Multiple inputs
retain their relative directory layout and are passed together to one Wakaru
invocation, allowing cross-chunk module IDs to resolve. Inputs must be distinct;
`additional_paths` accepts up to 16 additional files. The engine writes only to
its own staging directory. Reported module paths, input identities, UTF-8 byte its own staging directory. Reported module paths, input identities, UTF-8 byte
ranges, regular files and emitted map representations are validated before ranges, regular files and emitted map representations are validated before
publication. REA writes verified bytes to an exclusively owned output tree and publication. REA writes verified bytes to an exclusively owned output tree and
rolls it back on failure. It also checks that the original source and engine rolls it back on failure. It also checks that the original source and engine
bytes remain unchanged before committing. The existing output is never overwritten. bytes remain unchanged before committing. The existing output is never overwritten.
Multi-input relative filenames must be representable unchanged as artifact paths:
NFC Unicode spelling and no literal backslashes. Unsupported layouts fail before
invoking the engine rather than silently renaming inputs referenced by source maps.
Published contents: Published contents:
- `inputs/bundle.js`: exact original snapshot; its digest binds the Evidence. - `inputs/bundle.js`: exact original snapshot for a single input. Multiple inputs
retain paths relative to their common parent under `inputs/`.
- `modules/`: derived modules and any upstream-emitted source maps. - `modules/`: derived modules and any upstream-emitted source maps.
- `reports/stdout.json` and `reports/provenance.json`: exact producer reports. - `reports/stdout.json` and `reports/provenance.json`: exact producer reports.
- `manifest.json`: identities, modules, transform choices, warnings and limitations. - `manifest.json`: identities, modules, transform choices, warnings and limitations.
The inline result includes complete file paths/digests/sizes, extraction ranges, The inline result includes complete file paths/digests/sizes, extraction ranges,
warnings and `analysis_input`, directly usable as the arguments to warnings and `analysis_input`, directly usable as the arguments to
`analyze_javascript_application`. Raw Evidence retains selected command arguments, `analyze_javascript_application`. `source` identifies the primary input and binds
the Evidence subject; `additional_sources` records the other snapshots and their
published copies when selected. Each module names its owning snapshot, original
path and original digest; all extraction and context ranges refer to that input.
Raw Evidence retains selected command arguments,
reported input paths, exit status, stderr and reports; temporary paths are recorded reported input paths, exit status, stderr and reports; temporary paths are recorded
as historical execution coordinates after their workspace is removed. as historical execution coordinates after their workspace is removed.
@@ -107,7 +130,8 @@ equivalence remain unknown. No network fetching occurs.
## Resource policy and verification ## Resource policy and verification
Each operation uses one worker, a 1 GiB address-space ceiling and a 120-second Each operation uses one worker, a 1 GiB address-space ceiling and a 120-second
deadline; it accepts up to 64 MiB of input and 128 MiB/10000 entries of staged deadline; it accepts up to 64 MiB of total input across all selected files and
128 MiB/10000 entries of staged
output, with 8 MiB per diagnostic/report stream. Exceeding a limit is an explicit output, with 8 MiB per diagnostic/report stream. Exceeding a limit is an explicit
failure with rollback. These limits apply per operation and do not claim aggregate failure with rollback. These limits apply per operation and do not claim aggregate
host containment across concurrent callers. host containment across concurrent callers.
+3
View File
@@ -167,6 +167,9 @@ Native call tracing and process capture retain available observations in
`details.partial_observation` on failure, including when cleanup succeeds. `details.partial_observation` on failure, including when cleanup succeeds.
The observation reports its partial coverage; cleanup details describe host The observation reports its partial coverage; cleanup details describe host
state separately from the execution failure. state separately from the execution failure.
A process capture that fails during a finalization interval keeps every signal
attempt and its delivery result in `details.partial_observation`, with
`elapsed_ms` and `exit_code` null when no exit was observed.
Derived comparisons and reconstruction verification yield before computation Derived comparisons and reconstruction verification yield before computation
and before publication, so cancellation cannot race with successful Evidence. and before publication, so cancellation cannot race with successful Evidence.
+94
View File
@@ -181,6 +181,100 @@ run the same owned-process cleanup path. Settlement reports whether the
sampled process group quiesced or whether cleanup was needed or unverifiable; sampled process group quiesced or whether cleanup was needed or unverifiable;
sampling cannot prove that every short-lived or detached descendant was seen. sampling cannot prove that every short-lived or detached descendant was seen.
### Finalization interval
By default a deadline sends `SIGKILL` at once. Set `finalization_ms` to give the
target time to finish: when `timeout_ms` or `idle_timeout_ms` fires, REA sends
`SIGTERM`, keeps capturing output and selected files, and sends `SIGKILL` once
if the interval ends first. Cancellation is not delayed: it sends `SIGKILL`
immediately, also during finalization, and ends the run as cancelled.
```json
{
"executable": "node",
"arguments": ["./job.mjs"],
"filesystem_observation_paths": ["./reports"],
"timeout_ms": 30000,
"finalization_ms": 1500
}
```
REA binds the captured root's first start identity to its ownership token,
checking the identity before and after reading that token. A missing or
mismatched token, or an identity change during validation, leaves the identity
unverified rather than adopting a replacement process. Both finalization
signals recheck that retained identity immediately before the call; the check
and signal remain separate operating-system operations. REA does not fall back
to a bare PID kill for either finalization signal; failed identity or ownership
validation is recorded as `unverified` with its reason. Cancellation before any
deadline keeps the existing terminal kill. Descendants are not signalled during
finalization; owned-process cleanup handles them afterwards.
`exit.reason` keeps the initiating deadline (`timeout` or `idle_timeout`).
`exit.code` is `null` for every deadline exit; the code of an exit observed
during finalization is `exit.finalization.exit_code`, and the observed signal
stays in `exit.signal`. `exit.finalization` records what was observed and is
absent when no finalization was attempted:
```json
{
"code": null,
"signal": 0,
"reason": "timeout",
"finalization": {
"requested_ms": 1500,
"elapsed_ms": 12,
"exit_code": 0,
"signals": [
{ "signal": "SIGTERM", "sent_at_ms": 0, "delivery": "signaled" }
]
}
}
```
- `signals` lists each attempt in order: `SIGTERM` first, at most one `SIGKILL`.
`sent_at_ms` is the attempt time on a monotonic clock, relative to the start of
finalization, also for an attempt that could not be delivered.
- `delivery` is `signaled` (the identity matched and the signal call returned),
`gone` (no live process matched), `identity-changed` (the PID now belongs to a
different process, which was not signalled) or `unverified` (the identity could
not be established or the call failed). None of them proves the process
exited.
An `unverified` attempt may carry a `reason` string naming why: an unavailable
identity and its reason, an inspection or signal error, or delivery that did
not settle within the bound (1000 ms). An attempt recorded `unverified` after
that bound can still complete later; it stays identity-checked, and its late
result is not added to the record.
- `elapsed_ms` is when REA observed the exit, relative to the start of
finalization, or `null` when no exit was observed. The PTY layer can deliver the
exit slightly after the operating system ended the process.
- `exit_code` is the code the PTY reported when the exit was observed during
finalization and was not caused by a signal (`exit.signal` is `0`); a
finalizer that exits with `3` records `3`, one that exits cleanly records `0`.
It is `null` when no exit was observed or the target ended on a signal, and it
is never inferred. A target that exited on its own near the deadline is
recorded the same way.
- The observed signal stays in `exit.signal`; `exit.code` is `null` for every
deadline exit. The record does not attribute an exit to a particular signal,
and a `SIGKILL` from a scenario event next to the escalation cannot be told
apart from it.
- If the `SIGKILL` cannot be delivered, REA stops waiting for an exit, keeps every
observation collected so far with the delivery results, and releases the run
through owned-process cleanup.
- The `SIGKILL` is not sequenced after the `SIGTERM` delivery result: the two
deliveries are independent, because cancellation must not wait for a slow
identity check.
`timeout_ms + finalization_ms + settle_ms` must stay at or below
`Number.MAX_SAFE_INTEGER` (9007199254740991), the largest total exactly
representable as a JavaScript number; the scenario is rejected otherwise. This is
a representability limit, not a timer limit. The nominal lifecycle budget is that
sum; polling, snapshotting and cleanup lie outside it. Scenario `events` keep
being dispatched during finalization and are recorded as usual. `finalization_ms`
of `0`, the default, keeps today's behaviour and the committed identity of every
existing scenario. `elapsed_ms` and `sent_at_ms` are timing data and take part in
exit comparison like `settlement.elapsed_ms`.
Every capture requires `truncation_details`, with separate accounting for: Every capture requires `truncation_details`, with separate accounting for:
- `raw_terminal`: original UTF-8 PTY chunk bytes and observed/retained chunk - `raw_terminal`: original UTF-8 PTY chunk bytes and observed/retained chunk
+42
View File
@@ -15,6 +15,48 @@ the original run after a partial publication.
For a release that needs an independently frozen application checkpoint, use For a release that needs an independently frozen application checkpoint, use
the optional manual path below. It retains explicit prepare and publish phases. the optional manual path below. It retains explicit prepare and publish phases.
## Contributor-aware release notes
Use [the rea-changelog-update skill](https://github.com/morluto/rea/blob/main/.agents/skills/rea-changelog-update/SKILL.md)
to turn the bot draft into grouped Highlights, Changes, and Fixes with inline
`Thanks @...` credit. `npm run release:notes -- inventory` collects the full
selected Git range and paginated GitHub PR/author/closing-issue metadata;
`record` produces the complete contribution record, `check` validates credits
and optional PR-body equality, and `render` writes the verified release section.
Run a subcommand without its required options to see its usage. GitHub reads
require an authenticated `gh`; record/check/render work offline from the saved
inventory and complete local Git history.
Save a finalized inventory on the release branch at
`docs/releases/VERSION.contributions.json`. Review unknown identities and
reverted/backported work explicitly; GitHub metadata does not establish the
quality of a user-facing claim. Preserve the existing breaking-change section
and checkpoint checks. The contribution inventory is a reviewed snapshot, not
a signed attestation; the offline checker verifies local range coverage and
credit consistency, not GitHub's authorship independently.
Before editing the live PR, add `rea:release-notes-finalized` to prevent ordinary
main-push proposals from overwriting the editorial notes. Wait for any already
running proposal to finish. The guard applies to main-push automation; do not
run manual checkpoint preparation over finalized notes. Remove the label to
resume bot regeneration. The label does not freeze the base branch: new base
commits require re-inventorying and re-reviewing the notes.
Update both the release branch's `CHANGELOG.md` and the PR body, preserving
Release Please's surrounding metadata. Before creating a release, the workflow
checks an inventory when present, requires its target to equal the reviewed
merge's first parent, and compares the freshly fetched PR body to the changelog.
It rejects changes outside the release metadata paths and stale credit records
before any tag or publication. Review the diffs within allowed metadata paths
as well, especially package dependencies and scripts. Existing candidates without inventories retain their current
checks. Merging still starts publication; a notes edit does not authorize a
merge or release by itself.
The checker supports inline Release Please PR bodies up to GitHub's 65,536
character limit, including the surrounding header/footer. Overflow-link PR
bodies need a separate integration and fail explicitly; credits are never
silently truncated.
## 1. Select a source for a manual checkpoint ## 1. Select a source for a manual checkpoint
Use the next minor version proposed by Release Please, including releases with Use the next minor version proposed by Release Please, including releases with
+86 -15
View File
@@ -391,6 +391,12 @@ regressions until a real fixture establishes equivalent coverage.
`verify:browser` exercises source-map failure isolation and expanded-output limits through the compiled CLI and stdio MCP with real Chrome. It also submits five 2 MiB source-map annotations, checks retained script identities and explicit map omissions, and closes its owned fixture target. Replacement and document-reset behavior has separate producer-boundary coverage. `verify:browser` exercises source-map failure isolation and expanded-output limits through the compiled CLI and stdio MCP with real Chrome. It also submits five 2 MiB source-map annotations, checks retained script identities and explicit map omissions, and closes its owned fixture target. Replacement and document-reset behavior has separate producer-boundary coverage.
The raw Chrome fixture launchers use `--password-store=basic` so Linux startup
does not wait for an unlock prompt from a reachable but locked desktop keyring.
Such a stall leaves the initial page unloaded with empty or placeholder CDP URLs;
it does not establish permission to capture that page. See
[the keyring diagnosis](https://github.com/morluto/rea/issues/1384#issuecomment-6093781686).
`verify:browser` also captures a source-owned noise canvas as a real PNG above `verify:browser` also captures a source-owned noise canvas as a real PNG above
8 MiB through the CLI and stdio MCP, with complete byte/digest parity and real PNG 8 MiB through the CLI and stdio MCP, with complete byte/digest parity and real PNG
decoding. Its SDK client explicitly permits the larger inline JSON response; decoding. Its SDK client explicitly permits the larger inline JSON response;
@@ -444,14 +450,15 @@ installation (`application.java.min` through `application.java.max`). Current
12.1 releases require JDK 21 or newer and set no maximum. The lanes below still 12.1 releases require JDK 21 or newer and set no maximum. The lanes below still
prove behavior on the verified Ghidra 12.1.4 and JDK 21 build. prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
| Ghidra lane | Supported runner/target | Additional local tools | | Ghidra lane | Supported runner/target | Additional local tools |
| ------------------------------------------ | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------- | | ------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 | | `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 | | `npm run verify:ghidra:seeds` | POSIX host; source-owned 32-bit x86 raw fixture; function order and BSS labels | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 | | `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 | | `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites | | `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE; `-- --x86` selects native x86 PE | Ghidra 12.1.4, full JDK 21, and the matching native artifact | | `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE; `-- --x86` selects native x86 PE | Ghidra 12.1.4, full JDK 21, and the matching native artifact |
The optional Swift lane compiles a source-owned fixture and checks modern and The optional Swift lane compiles a source-owned fixture and checks modern and
provider-demangled procedure names through real stdio MCP, with and without the provider-demangled procedure names through real stdio MCP, with and without the
@@ -665,6 +672,13 @@ boundaries and budgets.
## Optional NativeAOT Ghidra analysis ## Optional NativeAOT Ghidra analysis
The source parser lane is independent of the optional annotation JAR. Its
synthetic RTR 9.1 PE golden checks captured-byte digest admission, rehydration
into an immutable parser overlay, MethodTable/slot relationships, and frozen
literal extraction; provider tests exercise `inspect_native_load_image` and
address-based `inspect_native_data_type` with the database result still
unavailable. These tests do not establish Windows Ghidra host behavior.
This lane is separate from the default native lane. `build:fixtures:nativeaot` This lane is separate from the default native lane. `build:fixtures:nativeaot`
requires an existing .NET SDK 8.0.416, the platform NativeAOT compiler/linker and requires an existing .NET SDK 8.0.416, the platform NativeAOT compiler/linker and
runtime pack 8.0.22. It builds benign sources into ignored `_reference/` and runtime pack 8.0.22. It builds benign sources into ignored `_reference/` and
@@ -765,12 +779,37 @@ Pull requests that change only root `README*.md` files, `docs/`, `AGENTS.md`,
or `CONTRIBUTING.md` run formatting and generated-document validation without or `CONTRIBUTING.md` run formatting and generated-document validation without
the source-test shards or native package lanes. Classification compares the PR the source-test shards or native package lanes. Classification compares the PR
head with its merge base, so later base-branch changes do not expand that scope. head with its merge base, so later base-branch changes do not expand that scope.
The required static and coverage aggregate jobs remain present and fail if Static and coverage aggregate jobs remain present and fail if classification
classification fails. Mixed changes, unknown paths and CI configuration changes fails. Every code change retains the complete four-shard Linux suite; individual
retain full checks. Main pushes retain the full baseline, including after test files are not selected by imports or filenames.
documentation merges; newer commits cancel superseded CI for the same PR or
main branch. Release publication has its own concurrency group and retains its `scripts/ci/plan.mjs` selects lanes using the ownership table in
package verification and public-registry canary. `scripts/ci/scopes.mjs`. Provider implementation changes run their real-provider
verifiers and the Linux x64 installed-package/Inspector baseline. Test-only
changes run the Linux suite without native package verification. Website changes
run website checks. Provider workflow changes validate the workflow and run that provider's verifier.
Manual-provider/Android workflow changes run workflow validation only.
Shared process/contract/CLI boundaries, dependencies, packaging inputs, unknown
paths, and changes to the planner select the full baseline. Package script-only
edits select their owning verifier when its entrypoint is identifiable; unknown
or shared script changes select everything. Android build-script edits retain
the Linux suite and workflow validation; real Android verification stays manual.
Renames account for both paths.
Main pushes and manual dispatch retain the full baseline, including all portable
provider lanes and the native matrix below. A `ci:full` PR label selects the full
baseline on the next PR run; manual dispatch can request it immediately. Each
classification job records its selection and broad-fallback reasons in the run
summary. Representative Git-diff cases live in
`tests/boundary/filesystem/ciChangeScope.test.ts`.
`CI required` checks classification and every selected lane, rejecting failed,
cancelled, or unexpectedly skipped work. Existing required check names are
preserved; changing branch protection to require the new aggregate is a separate
repository-settings operation. Newer commits cancel superseded CI for the same
PR or main branch. Release publication has its own concurrency group and retains
its package verification and public-registry canary. Licensed/self-hosted
provider workflows and the Android smoke test retain their explicit triggers.
CI exercises the pinned Node.js runtime on native hosted runners: CI exercises the pinned Node.js runtime on native hosted runners:
@@ -795,6 +834,35 @@ separate Apple-artifact and real-provider lanes. Actual Hopper, Ghidra, IDA,
browser and managed-tool claims require their corresponding verification lanes. browser and managed-tool claims require their corresponding verification lanes.
Runner labels follow the [GitHub hosted-runner reference](https://docs.github.com/en/actions/reference/runners/github-hosted-runners). Runner labels follow the [GitHub hosted-runner reference](https://docs.github.com/en/actions/reference/runners/github-hosted-runners).
### Termux Android browser smoke test
The **Real Termux browser verification** workflow is manual-only while its
emulator setup is being validated. After the workflow reaches the default
branch, select a revision under **Actions → Real Termux browser verification →
Run workflow**. It boots an Android 11 (API 30) x86_64 emulator, installs the
checksum-verified Termux 0.118.3 APK, and lets the app initialize its bootstrap.
The host stages the selected revision using `git archive HEAD`; no workstation
build outputs or dependencies are copied into Android.
`scripts/verify/termux/emulator.sh` sends a `RUN_COMMAND` intent to Termux's app
service. Root access is limited to staging files, sending the intent, and reading
diagnostics; installation, compilation, Node, and Chromium run as Termux's app
UID. The in-app script installs Termux's Android-linked Node distribution,
checks its version against `.nvmrc`, pins npm from `packageManager`, runs
`npm ci`, and builds with `build:termux`. Termux repository packages, including
Chromium, are resolved at run time; their installed versions are logged.
The verifier leaves Playwright's cache environment overrides unset, requires
a healthy public MCP doctor result, and captures URL plus DOM from a loopback
HTTP fixture through `capture-browser-scenario`. It checks completed steps,
the exact URL, a DOM marker, and reported browser cleanup. This lane covers one
Android emulator and Termux combination, not physical ARM devices, Electron,
or the complete desktop browser suite. A workflow definition alone does not
establish passing Android coverage; inspect its execution receipt.
The `termux-browser-diagnostics` artifact retains the Termux command log, exit
status, and Android logcat on success or failure. The emulator is disposable.
## Developer commands ## Developer commands
Use source feedback while editing, explicit boundary checks for the changed Use source feedback while editing, explicit boundary checks for the changed
@@ -968,7 +1036,10 @@ and `REA_JAVASCRIPT_FIXTURE_TOOLS` pointing to an isolated npm prefix containing
esbuild 0.25.10 and webpack 5.101.3. No global installation is required. esbuild 0.25.10 and webpack 5.101.3. No global installation is required.
The lane compiles source-owned fixtures, exercises CLI and stdio MCP, verifies The lane compiles source-owned fixtures, exercises CLI and stdio MCP, verifies
published bytes and provenance, feeds recovered modules into existing analysis, published bytes and provenance, feeds recovered modules into existing analysis,
and compares a finite set of known fixture results. The provider also accepts and compares a finite set of known fixture results. A webpack fixture with separate
runtime, shared, entry and lazy chunks verifies multi-input recovery through both
surfaces, per-file provenance and resolved cross-chunk imports in the downstream
application graph. The provider also accepts
other ^1.13.0 releases and reports them as unverified; this lane proves the other ^1.13.0 releases and reports them as unverified; this lane proves the
audited release. It does not establish audited release. It does not establish
arbitrary recovered-application equivalence. CI installs these prerequisites only arbitrary recovered-application equivalence. CI installs these prerequisites only
+6 -3
View File
@@ -21,9 +21,12 @@ additional permission flag or degraded mode is required.
- the 25 read-only Ghidra inventory, memory, and function-analysis operations. - the 25 read-only Ghidra inventory, memory, and function-analysis operations.
Loaded memory reads and file offsets preserve Ghidra's observed source mappings. Loaded memory reads and file offsets preserve Ghidra's observed source mappings.
PE load-image inspection returns measurements with an explicit unsupported PE load-image inspection returns an explicit unsupported attestation status;
attestation status; independent format-specific verification currently supports independent load-image verification currently supports DOS MZ only. For x64 PE
DOS MZ only. NativeAOT RTR 9.1, the read-only snapshot parser can report metadata through
`inspect_native_load_image` and `inspect_native_data_type` without a JAR or
database mutation. A real Windows-host NativeAOT Ghidra run is still required
to establish the complete host workflow.
This boundary does not establish general Windows feature parity. Ghidra GUI This boundary does not establish general Windows feature parity. Ghidra GUI
state, annotations, other target architectures and formats, and Hopper-only state, annotations, other target architectures and formats, and Hopper-only
+3
View File
@@ -3,9 +3,12 @@
"entry": [ "entry": [
"src/main.ts", "src/main.ts",
"src/cli.ts", "src/cli.ts",
"src/composition/javascriptAnalysisWorkerProcess.ts",
"src/contracts/errorSchemas.ts", "src/contracts/errorSchemas.ts",
"src/evaluation/AgentEvaluationFixtures.ts", "src/evaluation/AgentEvaluationFixtures.ts",
"src/application/RealHopperAssertions.ts", "src/application/RealHopperAssertions.ts",
"scripts/verify/termux/browser.mjs",
"scripts/verify-real-ghidra-noreturn.mjs",
"tests/processOwnershipRunner.ts" "tests/processOwnershipRunner.ts"
], ],
"project": ["src/**/*.ts", "tests/**/*.ts", "scripts/**/*.mjs"], "project": ["src/**/*.ts", "tests/**/*.ts", "scripts/**/*.mjs"],
+28
View File
@@ -0,0 +1,28 @@
The MIT License (MIT)
=====================
Copyright © 2023-2025 Washi
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
The read-only NativeAOT parser adapts format and hydration semantics from
Washi1337/ghidra-nativeaot commit
effeb734fc570c32650f88b159608979dc7b423e (MIT). MethodTable field layout was
cross-checked against .NET runtime src/coreclr/nativeaot/Runtime/inc/MethodTable.h
at dotnet/runtime commit a3b2d40328be0be3f8dd43f0e5cc925b8827b044 (MIT).
+17
View File
@@ -17,6 +17,7 @@
"@lydell/node-pty": "1.1.0", "@lydell/node-pty": "1.1.0",
"@modelcontextprotocol/client": "2.3.1", "@modelcontextprotocol/client": "2.3.1",
"@modelcontextprotocol/server": "2.3.1", "@modelcontextprotocol/server": "2.3.1",
"@xmldom/xmldom": "0.9.12",
"@xterm/addon-serialize": "0.14.0", "@xterm/addon-serialize": "0.14.0",
"@xterm/headless": "6.0.0", "@xterm/headless": "6.0.0",
"@zip.js/zip.js": "2.15.0", "@zip.js/zip.js": "2.15.0",
@@ -64,6 +65,7 @@
"oxlint": "1.87.0", "oxlint": "1.87.0",
"turbo": "2.11.7", "turbo": "2.11.7",
"typescript": "7.0.2", "typescript": "7.0.2",
"typescript-termux": "npm:typescript@5.9.3",
"vitepress": "1.6.4", "vitepress": "1.6.4",
"vitest": "5.0.3" "vitest": "5.0.3"
}, },
@@ -7367,6 +7369,21 @@
"@typescript/typescript-win32-x64": "7.0.2" "@typescript/typescript-win32-x64": "7.0.2"
} }
}, },
"node_modules/typescript-termux": {
"name": "typescript",
"version": "5.9.3",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
},
"engines": {
"node": ">=14.17"
}
},
"node_modules/unbash": { "node_modules/unbash": {
"version": "4.0.12", "version": "4.0.12",
"resolved": "https://registry.npmjs.org/unbash/-/unbash-4.0.12.tgz", "resolved": "https://registry.npmjs.org/unbash/-/unbash-4.0.12.tgz",
+13 -1
View File
@@ -30,6 +30,9 @@
"access": "public" "access": "public"
}, },
"type": "module", "type": "module",
"imports": {
"#javascript-analysis-worker": "./dist/composition/javascriptAnalysisWorkerProcess.js"
},
"bin": { "bin": {
"rea": "scripts/rea.mjs", "rea": "scripts/rea.mjs",
"rea-agents": "scripts/rea.mjs" "rea-agents": "scripts/rea.mjs"
@@ -50,6 +53,7 @@
"bridge/ghidra/ReaGhidraBridge.java", "bridge/ghidra/ReaGhidraBridge.java",
"bridge/ghidra/ReaGhidraExtensions.java", "bridge/ghidra/ReaGhidraExtensions.java",
"bridge/ghidra/ReaGhidraPrepareCom.java", "bridge/ghidra/ReaGhidraPrepareCom.java",
"bridge/ghidra/ReaGhidraApplySeeds.java",
"bridge/ghidra/ReaGhidraNoReturnFix.java", "bridge/ghidra/ReaGhidraNoReturnFix.java",
"bridge/android/ReaJadxBridge.java", "bridge/android/ReaJadxBridge.java",
"bridge/mitmproxy/capture.py", "bridge/mitmproxy/capture.py",
@@ -77,9 +81,11 @@
"bridge/pwndbg/launch_impl.py", "bridge/pwndbg/launch_impl.py",
"bridge/pwndbg/core_context.py", "bridge/pwndbg/core_context.py",
"third_party/pwndbg/README.md", "third_party/pwndbg/README.md",
"third_party/pwndbg/LICENSE.md" "third_party/pwndbg/LICENSE.md",
"licenses/nativeaot-parser-MIT.txt"
], ],
"scripts": { "scripts": {
"release:notes": "node scripts/release-notes.mjs",
"deps:check": "node scripts/check-dependency-install.mjs", "deps:check": "node scripts/check-dependency-install.mjs",
"metadata:generate": "node scripts/run-exclusive.mjs artifacts node scripts/generate-package-metadata.mjs", "metadata:generate": "node scripts/run-exclusive.mjs artifacts node scripts/generate-package-metadata.mjs",
"metadata:check": "node scripts/generate-package-metadata.mjs --check", "metadata:check": "node scripts/generate-package-metadata.mjs --check",
@@ -101,6 +107,8 @@
"precheck:ci": "npm run deps:check", "precheck:ci": "npm run deps:check",
"check:ci": "node scripts/run-exclusive.mjs artifacts turbo run typecheck lint format:check knip metadata:check jscpd scan:todos && npm run verify:architecture-guards", "check:ci": "node scripts/run-exclusive.mjs artifacts turbo run typecheck lint format:check knip metadata:check jscpd scan:todos && npm run verify:architecture-guards",
"build:cached": "node scripts/check-dependency-install.mjs && node scripts/run-exclusive.mjs artifacts turbo run artifacts:skills", "build:cached": "node scripts/check-dependency-install.mjs && node scripts/run-exclusive.mjs artifacts turbo run artifacts:skills",
"build:termux": "node scripts/run-exclusive.mjs artifacts npm run build:termux:run",
"build:termux:run": "npm run compile:termux && npm run artifacts:skills",
"format": "oxfmt .", "format": "oxfmt .",
"format:check": "oxfmt --check .", "format:check": "oxfmt --check .",
"lint": "oxlint && npm run verify:module-boundaries", "lint": "oxlint && npm run verify:module-boundaries",
@@ -150,6 +158,7 @@
"verify:apktool": "npm run build:cached && node scripts/verify-apktool.mjs", "verify:apktool": "npm run build:cached && node scripts/verify-apktool.mjs",
"verify:adb": "npm run build:cached && node scripts/verify-adb.mjs", "verify:adb": "npm run build:cached && node scripts/verify-adb.mjs",
"verify:ghidra": "npm run build:cached && node scripts/verify-real-ghidra.mjs", "verify:ghidra": "npm run build:cached && node scripts/verify-real-ghidra.mjs",
"verify:ghidra:seeds": "npm run build:cached && node scripts/verify/ghidra/seeds.mjs",
"verify:ghidra:mips": "npm run build:cached && node scripts/verify-real-ghidra-mips.mjs", "verify:ghidra:mips": "npm run build:cached && node scripts/verify-real-ghidra-mips.mjs",
"verify:ghidra:swift": "npm run build:cached && node scripts/verify-real-ghidra-swift.mjs", "verify:ghidra:swift": "npm run build:cached && node scripts/verify-real-ghidra-swift.mjs",
"verify:ida": "npm run build:cached && node scripts/verify-real-ida.mjs", "verify:ida": "npm run build:cached && node scripts/verify-real-ida.mjs",
@@ -236,6 +245,7 @@
"verify:recorded:crash": "npm run build:cached && node scripts/verify-recorded-crash.mjs", "verify:recorded:crash": "npm run build:cached && node scripts/verify-recorded-crash.mjs",
"mcp-catalog:check:built": "node scripts/generate-mcp-tool-catalog.mjs --check", "mcp-catalog:check:built": "node scripts/generate-mcp-tool-catalog.mjs --check",
"compile": "node scripts/clean-build-output.mjs && tsc -p tsconfig.build.json", "compile": "node scripts/clean-build-output.mjs && tsc -p tsconfig.build.json",
"compile:termux": "node scripts/clean-build-output.mjs && node --max-old-space-size=3072 node_modules/typescript-termux/lib/tsc.js -p tsconfig.build.json",
"artifacts:mcp-catalog": "node scripts/generate-mcp-tool-catalog.mjs", "artifacts:mcp-catalog": "node scripts/generate-mcp-tool-catalog.mjs",
"artifacts:skills": "node scripts/generate-skill-metadata.mjs", "artifacts:skills": "node scripts/generate-skill-metadata.mjs",
"artifacts:product-catalog": "node scripts/generate-product-catalog.mjs", "artifacts:product-catalog": "node scripts/generate-product-catalog.mjs",
@@ -261,6 +271,7 @@
"@lydell/node-pty": "1.1.0", "@lydell/node-pty": "1.1.0",
"@modelcontextprotocol/client": "2.3.1", "@modelcontextprotocol/client": "2.3.1",
"@modelcontextprotocol/server": "2.3.1", "@modelcontextprotocol/server": "2.3.1",
"@xmldom/xmldom": "0.9.12",
"@xterm/addon-serialize": "0.14.0", "@xterm/addon-serialize": "0.14.0",
"@xterm/headless": "6.0.0", "@xterm/headless": "6.0.0",
"@zip.js/zip.js": "2.15.0", "@zip.js/zip.js": "2.15.0",
@@ -309,6 +320,7 @@
"oxlint": "1.87.0", "oxlint": "1.87.0",
"turbo": "2.11.7", "turbo": "2.11.7",
"typescript": "7.0.2", "typescript": "7.0.2",
"typescript-termux": "npm:typescript@5.9.3",
"vitepress": "1.6.4", "vitepress": "1.6.4",
"vitest": "5.0.3" "vitest": "5.0.3"
} }
+1 -1
View File
@@ -78,7 +78,7 @@ const classes = await mkdtemp(join(output, "classes-"));
try { try {
const sources = [ const sources = [
...(await files(join(source, "src/main/java"), ".java")), ...(await files(join(source, "src/main/java"), ".java")),
...(await files(join(root, "bridge/ghidra/extensions/nativeaot"), ".java")), ...(await files(join(root, "bridge/nativeaot"), ".java")),
]; ];
const jars = await files(join(install, "Ghidra"), ".jar"); const jars = await files(join(install, "Ghidra"), ".jar");
const args = [ const args = [
+198
View File
@@ -0,0 +1,198 @@
import { execFileSync } from "node:child_process";
import { appendFileSync } from "node:fs";
import { matchesGlob } from "node:path";
import { SCOPES } from "./scopes.mjs";
const flags = [
"code",
"runtime",
"package",
"apple",
"fixtures",
"readiness",
"docs",
"workflows",
...Object.keys(SCOPES),
];
// Actionlint validates workflow syntax only. The semantic suite asserts these
// workflows' release, publication and self-hosted runner safeguards.
const GUARDED_WORKFLOWS = new Set([
".github/workflows/release.yml",
".github/workflows/real-ghidra-windows.yml",
]);
// Sources compiled by build-conformance-fixtures.mjs. The readiness journey
// analyzes the built C fixture as well as its own applications.
const FIXTURE_SOURCES = [
"tests/conformance/c/**",
"tests/conformance/napi/**",
"tests/conformance/objc/**",
"tests/conformance/swift/**",
"tests/conformance/versions/**",
];
const READINESS_SOURCES = ["tests/conformance/readiness/**"];
const matchesAny = (file, patterns) =>
patterns.some((pattern) => matchesGlob(file, pattern));
const selected = Object.fromEntries(flags.map((flag) => [flag, false]));
const reasons = [];
const full = (reason) => {
for (const flag of flags) selected[flag] = true;
reasons.push(reason);
};
const git = (...args) =>
execFileSync("git", args, { encoding: "utf8" }).trimEnd();
function classifyPackage(base, head) {
const before = JSON.parse(git("show", `${base}:package.json`));
const after = JSON.parse(git("show", `${head}:package.json`));
const { scripts: beforeScripts, ...beforeRest } = before;
const { scripts: afterScripts, ...afterRest } = after;
if (JSON.stringify(beforeRest) !== JSON.stringify(afterRest)) {
full("Package metadata or dependencies changed");
return;
}
selected.code = true;
const changed = [
...new Set([
...Object.keys(beforeScripts ?? {}),
...Object.keys(afterScripts ?? {}),
]),
].filter((key) => beforeScripts?.[key] !== afterScripts?.[key]);
for (const script of changed) {
if (
/^(test(?::|$)|check(?::|$)|lint(?::|$)|format(?::|$)|knip$|jscpd$)/.test(
script,
)
)
continue;
if (
["compile:termux", "build:termux", "build:termux:run"].includes(script)
) {
selected.workflows = true;
reasons.push(
`Android build script changed; real Android verification remains manual: ${script}`,
);
continue;
}
// Verify scripts own provider setup as well as execution. Reuse their
// existing source-file ownership rather than maintaining a second map.
const entrypoint = (command) =>
/^(?:npm run build:cached && )?node (scripts\/[\w/-]+\.mjs)(?: [\w./=-]+)*$/.exec(
command ?? "",
)?.[1];
const beforeEntry = entrypoint(beforeScripts?.[script]);
const afterEntry = entrypoint(afterScripts?.[script]);
const entries = [beforeEntry, afterEntry].filter(Boolean);
const simple =
(!beforeScripts?.[script] || beforeEntry) &&
(!afterScripts?.[script] || afterEntry);
const scriptOwners =
simple &&
entries.every((entry) =>
Object.values(SCOPES).some((patterns) => patterns.includes(entry)),
)
? Object.entries(SCOPES)
.filter(([, patterns]) =>
entries.some((entry) => patterns.includes(entry)),
)
.map(([owner]) => owner)
: [];
if (scriptOwners.length)
for (const owner of scriptOwners) selected[owner] = true;
else full(`Shared or unknown package script: ${script}`);
}
}
const event = process.env.GITHUB_EVENT_NAME;
let files = [];
if (
event === "push" ||
event === "schedule" ||
event === "workflow_dispatch" ||
process.env.CI_FULL === "true"
) {
full("Full baseline requested by event or ci:full label");
} else if (event === "pull_request") {
const base = git("merge-base", process.env.BASE_SHA, process.env.HEAD_SHA);
const head = process.env.HEAD_SHA;
// Disable rename detection so both the removed and added paths participate.
files = git("diff", "--name-only", "--no-renames", "-z", base, head)
.split("\0")
.filter(Boolean);
for (const file of files) {
if (
/^(README(?:_[^/]+)?\.md|AGENTS\.md|CONTRIBUTING\.md)$/.test(file) ||
file.startsWith("docs/")
) {
selected.docs = true;
continue;
}
if (/^(src|tests)\/.*\.(?:test|spec)\.[cm]?[jt]sx?$/.test(file)) {
selected.code = true;
continue;
}
const owners = Object.entries(SCOPES)
.filter(([, patterns]) =>
patterns.some((pattern) => matchesGlob(file, pattern)),
)
.map(([owner]) => owner);
for (const owner of owners) selected[owner] = true;
if (
file.startsWith(".github/workflows/") ||
file === ".github/actionlint.yaml"
) {
selected.workflows = true;
if (GUARDED_WORKFLOWS.has(file)) selected.code = true;
if (file !== ".github/workflows/ci.yml") continue;
}
if (owners.length === 1 && owners[0] === "website") continue;
if (/^src\/(contracts|domain)\//.test(file)) selected.docs = true;
if (file === "package.json") {
classifyPackage(base, head);
continue;
}
selected.code = true;
if (matchesAny(file, FIXTURE_SOURCES)) {
selected.fixtures = true;
selected.readiness = true;
}
if (matchesAny(file, READINESS_SOURCES)) selected.readiness = true;
if (file.startsWith("tests/")) continue;
if (
/^(src\/process\/|src\/filesystem\/|src\/contracts\/tool|src\/server\/createServer|src\/cli\.ts|scripts\/ci\/)/.test(
file,
)
) {
full(`Shared boundary: ${file}`);
} else if (owners.length) {
if (file.startsWith("src/") || file.startsWith("bridge/"))
selected.runtime = true;
} else {
full(`Unclassified input: ${file}`);
}
}
} else throw new Error(`Unsupported CI event: ${event}`);
selected.runtime ||= selected.package;
const linux = { os: "ubuntu-latest", platform: "linux", architecture: "x64" };
const matrix = {
include: selected.package
? [
linux,
{ os: "ubuntu-24.04-arm", platform: "linux", architecture: "arm64" },
{ os: "macos-15", platform: "darwin", architecture: "arm64" },
{ os: "macos-15-intel", platform: "darwin", architecture: "x64" },
]
: [linux],
};
const plan = { ...selected, matrix, files, reasons };
if (process.env.GITHUB_OUTPUT)
appendFileSync(
process.env.GITHUB_OUTPUT,
Object.entries({ ...selected, matrix: JSON.stringify(matrix) })
.map(([key, value]) => `${key}=${value}\n`)
.join(""),
);
if (process.env.GITHUB_STEP_SUMMARY)
appendFileSync(
process.env.GITHUB_STEP_SUMMARY,
`### Selected CI lanes\n\n${flags.filter((flag) => selected[flag]).join(", ") || "Formatting only"}\n\n${reasons.join("\n\n")}\n`,
);
console.log(JSON.stringify(plan));
+25
View File
@@ -0,0 +1,25 @@
const jobs = JSON.parse(process.env.CI_JOBS);
const outputs = jobs.changes.outputs;
const required = ["changes", "static", "test"];
const lanes = {
code: ["build", "test-shard"],
runtime: ["windows-native-build", "package-e2e"],
package: ["windows-curated"],
apple: ["apple-artifacts"],
fixtures: ["conformance-fixtures"],
readiness: ["reconstruction-readiness"],
docs: ["docs"],
};
for (const [scope, value] of Object.entries(outputs)) {
if (value === "true") required.push(...(lanes[scope] ?? [scope]));
}
const failed = [...new Set(required)].filter(
(name) => jobs[name]?.result !== "success",
);
for (const [name, job] of Object.entries(jobs)) {
if (["failure", "cancelled"].includes(job.result) && !failed.includes(name))
failed.push(name);
}
if (failed.length)
throw new Error(`CI lanes did not pass: ${failed.join(", ")}`);
console.log("Every selected CI lane passed");
+227
View File
@@ -0,0 +1,227 @@
// Ownership of the real-provider and website verification lanes. Shared build
// inputs are classified once in plan.mjs rather than repeated in every workflow.
export const SCOPES = {
browser: [
".github/workflows/real-browser.yml",
"scripts/verify-real-browser.mjs",
"scripts/verify-browser-network.mjs",
"scripts/verify-browser-scripts.mjs",
"scripts/verify-browser-modules.mjs",
"scripts/verify-browser-dom-destinations.mjs",
"scripts/lib/browser-*.mjs",
"scripts/fixtures/browser-network-site.mjs",
"scripts/fixtures/browser-script-site.mjs",
"scripts/fixtures/browser-module-site.mjs",
"scripts/fixtures/browser-verifier-site.mjs",
"scripts/fixtures/page-cdp-proxy.mjs",
"src/browser/**",
"src/domain/browser*.ts",
"src/domain/webScript*.ts",
"src/domain/webModule*.ts",
"src/application/WebModule*.ts",
"src/composition/webModules.ts",
"src/artifacts/readStableArtifact.ts",
"src/artifacts/ArtifactPaths.ts",
"src/application/WebScriptExportService.ts",
"src/contracts/webScriptToolContracts.ts",
"src/server/registerWebScriptTool.ts",
"src/server/registerWebModuleTool.ts",
"src/server/createServer.ts",
"src/cli/webModuleCommands.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolEffects.ts",
"src/cli/webScriptCommands.ts",
],
binary_layout: [
".github/workflows/real-binary-layout.yml",
"bridge/pwntools/**",
"src/native/pwntools/**",
"src/domain/native/binaryLayout*",
"src/application/binaryDiagnostics/**",
"src/contracts/native/binaryDiagnosticsToolContracts.ts",
"src/composition/binaryDiagnostics.ts",
"src/server/registerBinaryDiagnosticsTools.ts",
"src/server/createServer.ts",
"src/cli/binaryDiagnosticsCommands.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolContracts.ts",
"src/contracts/toolEffects.ts",
"src/process/OwnedCommand.ts",
"src/process/ProviderProcess.ts",
"src/process/ProviderDeadline.ts",
"src/process/ProcessOwnership*.ts",
"src/process/PrivateRuntimeRoot.ts",
"src/artifacts/readStableArtifact*.ts",
"src/artifacts/ArtifactReader.ts",
"src/domain/analysisError*.ts",
"scripts/verify-binary-layout.mjs",
"scripts/lib/elf-section-name-fixtures.mjs",
"scripts/lib/large-result-mcp.mjs",
"scripts/fixtures/binary-layout*",
],
evm: [
".github/workflows/real-evm-interface.yml",
"src/evm/**",
"src/domain/evm/**",
"src/application/evm/**",
"src/contracts/evm/**",
"src/composition/evm.ts",
"src/cli/evmCommands.ts",
"src/server/registerEvmTools.ts",
"src/server/createServer.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolContracts.ts",
"src/contracts/toolEffects.ts",
"src/process/OwnedCommand.ts",
"src/process/ProviderProcess.ts",
"src/process/ProcessOwnership*.ts",
"src/process/ProviderDeadline.ts",
"src/process/PrivateRuntimeRoot.ts",
"src/artifacts/readStableArtifact*.ts",
"src/artifacts/ArtifactReader.ts",
"scripts/verify-evm-interface.mjs",
"scripts/fixtures/evm-interface.sol",
"third_party/evmole/**",
],
javascript_recovery: [
".github/workflows/real-javascript-recovery.yml",
"src/javascript/recovery/**",
"src/domain/javascript/**",
"src/application/javascript/JavaScriptRecovery*.ts",
"src/composition/javascriptRecovery.ts",
"src/contracts/javascript/javascriptRecoveryToolContracts.ts",
"src/cli/javascriptRecoveryCommands.ts",
"src/cli/javascriptApplicationAnalysis.ts",
"src/server/registerJavaScriptRecoveryTool.ts",
"src/server/createServer.ts",
"src/server/sessionAvailabilityPolicy.ts",
"src/application/CapabilityInventory.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolContracts.ts",
"src/contracts/toolEffects.ts",
"scripts/rea.mjs",
"src/process/**",
"src/artifacts/SafeOutputTree.ts",
"scripts/verify/javascript/recovery.mjs",
"scripts/fixtures/javascript-recovery/**",
"third_party/wakaru",
],
recorded_crash: [
".github/workflows/real-recorded-crash.yml",
"bridge/pwntools/**",
"bridge/pwndbg/**",
"src/native/pwntools/**",
"src/native/pwndbg/**",
"src/domain/native/recordedCrash*",
"src/application/binaryDiagnostics/**",
"src/contracts/native/binaryDiagnosticsToolContracts.ts",
"src/contracts/toolEffects.ts",
"src/contracts/toolOutputSchemaGroups.ts",
"src/composition/binaryDiagnostics.ts",
"src/server/registerRecordedCrashTools.ts",
"src/server/createServer.ts",
"src/server/sessionAvailabilityPolicy.ts",
"src/cli/binaryDiagnosticsCommands.ts",
"src/cliCommandNames.ts",
"src/process/**",
"src/artifacts/readStableArtifact*.ts",
"src/domain/analysisError*.ts",
"scripts/verify-recorded-crash.mjs",
"scripts/verify/recorded-crash/**",
"scripts/lib/recorded-core-fixtures.py",
"scripts/fixtures/recorded-crash.c",
"third_party/pwndbg/**",
],
web_network: [
".github/workflows/real-web-network-captures.yml",
"bridge/mitmproxy/**",
"src/browser/history/**",
"src/domain/webNetworkCapture*.ts",
"src/domain/analysisError*.ts",
"src/contracts/errorSchemas.ts",
"src/domain/explicitSensitive*.ts",
"src/application/WebNetworkCapture*",
"src/composition/webNetworkCaptures.ts",
"src/cli/webNetworkCaptureCommands.ts",
"src/server/registerWebNetworkCaptureTool.ts",
"src/server/createServer.ts",
"src/server/toolRegistrationOptions.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolContracts.ts",
"src/contracts/toolEffects.ts",
"src/process/OwnedCommand.ts",
"src/process/ProviderProcess.ts",
"src/process/ProviderDeadline.ts",
"src/process/ProcessOwnership*.ts",
"src/process/PrivateRuntimeRoot.ts",
"src/process/capture/ProcessCaptureLifecycle.ts",
"src/artifacts/readStableArtifact*.ts",
"src/artifacts/ArtifactReader.ts",
"src/contracts/webNetworkCaptureToolContracts.ts",
"scripts/verify-web-network-captures.mjs",
"scripts/fixtures/generate-mitmproxy-capture.py",
],
web_runtime: [
".github/workflows/real-web-runtime.yml",
"scripts/verify-browser-runtime.mjs",
"scripts/lib/browser-runtime-*.mjs",
"src/process/ProviderProcess.ts",
"src/process/ProcessOwnership*.ts",
"scripts/fixtures/browser-runtime-site.mjs",
"src/browser/execution/**",
"src/browser/CdpConnection.ts",
"src/browser/CdpEndpoint.ts",
"src/browser/CdpTarget*.ts",
"src/browser/CdpCaptureValues.ts",
"src/browser/CdpAuthorized*.ts",
"src/domain/webRuntime.ts",
"src/domain/webExecution.ts",
"src/domain/webEventListeners.ts",
"src/domain/browserObservationError*.ts",
"src/application/WebRuntime*.ts",
"src/composition/webRuntime.ts",
"src/contracts/webRuntimeToolContracts.ts",
"src/contracts/toolEffects.ts",
"src/server/registerWebRuntimeTools.ts",
"src/server/createServer.ts",
"src/cli/webRuntimeCommands.ts",
"src/cli/commandCancellation.ts",
"src/cli.ts",
],
web_source_map: [
".github/workflows/real-web-source-map.yml",
"scripts/verify-browser-source-maps.mjs",
"scripts/lib/browser-source-map-e2e.mjs",
"scripts/fixtures/browser-source-map-site.mjs",
"src/javascript/sourceMaps/**",
"src/browser/assets/**",
"src/domain/webSource*.ts",
"src/domain/webScript*.ts",
"src/application/WebSourceLocation*.ts",
"src/application/ValidateSelectedWebScript.ts",
"src/composition/webSourceLocations.ts",
"src/server/registerWebSourceLocationTool.ts",
"src/server/createServer.ts",
"src/contracts/webScriptToolContracts.ts",
"src/process/**",
"src/cli/webSourceLocationCommands.ts",
"src/cli.ts",
"src/artifacts/readStableArtifact.ts",
],
nativeaot: [
"bridge/nativeaot/**",
"scripts/generate-nativeaot-fixtures.mjs",
"tests/fixtures/nativeaot/**",
".github/workflows/nativeaot-fixtures.yml",
],
website: [
"website/**",
"scripts/*website*.py",
".github/workflows/website-check.yml",
],
};
+55 -15
View File
@@ -1,5 +1,5 @@
import { createRequire } from "node:module"; import { createRequire } from "node:module";
import { mkdir, readFile, writeFile } from "node:fs/promises"; import { mkdir, readFile, readdir, writeFile } from "node:fs/promises";
import { join } from "node:path"; import { join } from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
@@ -20,23 +20,56 @@ export async function buildRecoveryFixtures(root, prefix) {
const source = fileURLToPath(new URL("./entry.mjs", import.meta.url)); const source = fileURLToPath(new URL("./entry.mjs", import.meta.url));
const webpackOutput = join(root, "webpack"); const webpackOutput = join(root, "webpack");
await mkdir(webpackOutput); await mkdir(webpackOutput);
await new Promise((resolve, reject) => { const compileWebpack = async (options) =>
const compiler = webpack({ new Promise((resolve, reject) => {
mode: "production", const compiler = webpack(options);
entry: source, compiler.run((error, stats) => {
parallelism: 1, compiler.close((closeError) => {
optimization: { concatenateModules: false }, if (error || closeError) return reject(error ?? closeError);
output: { path: webpackOutput, filename: "bundle.js" }, if (!stats || stats.hasErrors())
}); return reject(new Error(stats?.toString() ?? "No webpack stats"));
compiler.run((error, stats) => { resolve();
compiler.close((closeError) => { });
if (error || closeError) return reject(error ?? closeError);
if (!stats || stats.hasErrors())
return reject(new Error(stats?.toString() ?? "No webpack stats"));
resolve();
}); });
}); });
await compileWebpack({
mode: "production",
entry: source,
parallelism: 1,
optimization: { concatenateModules: false },
output: { path: webpackOutput, filename: "bundle.js" },
}); });
const splitOutput = join(root, "webpack-code-split");
await mkdir(splitOutput);
const splitSource = fileURLToPath(
new URL("./code-split/entry.mjs", import.meta.url),
);
await compileWebpack({
mode: "production",
entry: splitSource,
parallelism: 1,
optimization: {
concatenateModules: false,
runtimeChunk: "single",
splitChunks: {
chunks: "all",
minSize: 0,
cacheGroups: {
shared: { test: /shared\.mjs$/u, name: "shared", enforce: true },
},
},
},
output: {
path: splitOutput,
filename: "[name].js",
chunkFilename: "[id].chunk.js",
},
});
const splitFiles = (await readdir(splitOutput))
.filter((name) => name !== "main.js")
.sort();
if (splitFiles.length !== 3)
throw new Error("Expected runtime, shared and lazy chunks");
const esbuildPath = join(root, "esbuild.js"); const esbuildPath = join(root, "esbuild.js");
await esbuild.build({ await esbuild.build({
entryPoints: [source], entryPoints: [source],
@@ -95,6 +128,13 @@ export async function buildRecoveryFixtures(root, prefix) {
options: { extraction_mode: "inspection", rewrite_level: "minimal" }, options: { extraction_mode: "inspection", rewrite_level: "minimal" },
}, },
], ],
codeSplit: {
name: "webpack-code-split",
path: join(splitOutput, "main.js"),
additional_paths: splitFiles.map((name) => join(splitOutput, name)),
expectedFormat: "webpack5",
expectedModules: 4,
},
toolchains: { esbuild: esbuild.version, webpack: webpackVersion }, toolchains: { esbuild: esbuild.version, webpack: webpackVersion },
}; };
} }
@@ -0,0 +1,6 @@
import { Cart, formatPrice, TAX_RATE } from "./shared.mjs";
export function checkout(prices) {
const cart = new Cart();
prices.forEach((price) => cart.add(price));
return formatPrice(cart.total() * (1 + TAX_RATE));
}
@@ -0,0 +1,3 @@
import { formatPrice } from "./shared.mjs";
console.log(formatPrice(1));
import("./checkout.mjs").then(({ checkout }) => console.log(checkout([1, 2])));
@@ -0,0 +1,14 @@
export class Cart {
constructor() {
this.items = [];
}
add(price) {
this.items.push(price);
return this;
}
total() {
return this.items.reduce((sum, price) => sum + price, 0);
}
}
export const formatPrice = (value) => `$${value.toFixed(2)}`;
export const TAX_RATE = 0.08;
@@ -18,6 +18,7 @@ export async function startRuntimeFixtureBrowser(executable, origin) {
command: executable, command: executable,
arguments: [ arguments: [
"--headless=new", "--headless=new",
"--password-store=basic",
...(process.env.REA_BROWSER_NO_SANDBOX === "true" ...(process.env.REA_BROWSER_NO_SANDBOX === "true"
? ["--no-sandbox"] ? ["--no-sandbox"]
: []), : []),
@@ -0,0 +1,683 @@
import assert from "node:assert/strict";
import { createServer } from "node:http";
import { once } from "node:events";
import { setTimeout as delay } from "node:timers/promises";
import { CdpConnection } from "../../dist/browser/CdpConnection.js";
import { createBrowserScenarioProvider } from "../../dist/composition/browserScenario.js";
import { browserScenarioSchema } from "../../dist/domain/browserScenario.js";
import {
runScenarioCli,
scenarioProfiles,
} from "./browser-scenario-verifier.mjs";
const scenarioFor = (browser, origin) =>
browserScenarioSchema.parse({
browser,
start_url: { url: `${origin}/storage-main` },
actions: [
{
step_id: "frame-ready",
action: "wait_for",
locator: { kind: "css", selector: "#frame-report[data-ready='true']" },
state: "attached",
timeout_ms: 10_000,
},
{
step_id: "top-level-partition",
action: "goto",
destination: {
url: `${origin.replace("127.0.0.1", "localhost")}/storage-top`,
},
wait_until: "load",
},
{
step_id: "return-to-origin",
action: "goto",
destination: { url: `${origin}/storage-main` },
wait_until: "load",
},
],
storage: {
local_storage: [
{
origin,
entries: [
{
name: "rea_storage_probe",
value: { source: "literal", value: "main-initial" },
},
{
name: "rea_storage_remove",
value: { source: "literal", value: "main-remove-seed" },
},
],
},
{
origin: origin.replace("127.0.0.1", "localhost"),
entries: [
{
name: "rea_storage_probe",
value: { source: "literal", value: "frame-initial" },
},
{
name: "rea_storage_remove",
value: { source: "literal", value: "frame-remove-seed" },
},
],
},
],
session_storage: [
{
origin,
entries: [
{
name: "rea_storage_probe",
value: { source: "literal", value: "main-session-initial" },
},
{
name: "rea_storage_empty",
value: { source: "literal", value: "main-empty-seed" },
},
],
},
{
origin: origin.replace("127.0.0.1", "localhost"),
entries: [
{
name: "rea_storage_probe",
value: { source: "literal", value: "frame-session-initial" },
},
{
name: "rea_storage_empty",
value: { source: "literal", value: "frame-empty-seed" },
},
],
},
],
},
capture: { after_each_step: ["dom"], at_end: ["dom"] },
});
const raceWithTimeout = (operation, timeoutMs, onTimeout = () => undefined) => {
const controller = new AbortController();
return Promise.race([
operation,
delay(timeoutMs, undefined, { signal: controller.signal }).then(onTimeout),
]).finally(() => controller.abort());
};
const storagePage = (kind, port) => {
if (kind === "main")
return `<!doctype html><html><body><pre id="main-report"></pre><pre id="frame-report"></pre><iframe src="http://localhost:${port}/storage-frame"></iframe><script>
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
document.querySelector("#main-report").textContent = JSON.stringify(state);
localStorage.setItem("rea_storage_probe", "main-updated");
sessionStorage.setItem("rea_storage_probe", "main-session-updated");
localStorage.removeItem("rea_storage_remove");
sessionStorage.setItem("rea_storage_empty", "");
addEventListener("message", (event) => {
if (event.data?.kind !== "rea-storage-frame") return;
const report = document.querySelector("#frame-report");
report.textContent = JSON.stringify(event.data.state);
report.dataset.ready = "true";
});
</script></body></html>`;
if (kind === "frame")
return `<!doctype html><html><body><pre id="frame-state"></pre><script>
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
document.querySelector("#frame-state").textContent = JSON.stringify(state);
localStorage.setItem("rea_storage_probe", "frame-updated");
sessionStorage.setItem("rea_storage_probe", "frame-session-updated");
localStorage.removeItem("rea_storage_remove");
sessionStorage.setItem("rea_storage_empty", "");
parent.postMessage({ kind: "rea-storage-frame", state }, "*");
</script></body></html>`;
if (kind === "debugger")
return `<!doctype html><html><body><pre id="debugger-state"></pre><script>
window.reaDebuggerBefore = true;
debugger;
window.reaDebuggerAfter = true;
document.querySelector("#debugger-state").textContent = String(window.reaDebuggerAfter);
</script></body></html>`;
return `<!doctype html><html><body><pre id="top-report"></pre><script>
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
document.querySelector("#top-report").textContent = JSON.stringify(state);
localStorage.setItem("rea_storage_probe", "top-updated");
sessionStorage.setItem("rea_storage_probe", "top-session-updated");
</script></body></html>`;
};
const startStorageSite = async () => {
let port = 0;
const server = createServer((request, response) => {
const path = new URL(request.url ?? "/", "http://127.0.0.1").pathname;
const kind =
path === "/storage-main"
? "main"
: path === "/storage-frame"
? "frame"
: path === "/storage-debugger"
? "debugger"
: path === "/storage-top"
? "top"
: undefined;
if (kind === undefined) {
response.writeHead(404).end();
return;
}
response.setHeader("content-type", "text/html; charset=utf-8");
response.end(storagePage(kind, port));
});
server.listen(0, "::");
await once(server, "listening");
port = server.address().port;
return {
origin: `http://127.0.0.1:${port}`,
close: async () => {
server.close();
// Chrome can leave speculative TCP connections without an HTTP request;
// server.close() alone waits indefinitely for those owned sockets.
server.closeAllConnections();
await once(server, "close");
},
};
};
const domText = (step) => {
assert.equal(step.artifacts.dom.state, "captured");
return step.artifacts.dom.value.text;
};
const assertContainsState = (text, id, expected) => {
const match = new RegExp(`<pre id="${id}"[^>]*>(.*?)</pre>`, "su").exec(text);
assert.ok(match, `Captured DOM omitted #${id}: ${text}`);
assert.deepEqual(JSON.parse(match[1]), expected);
};
/** Verify one-time local/session storage initialization on a real Chromium. */
export async function verifyScenarioStorage({
executable,
endpoint,
targetId,
}) {
const site = await startStorageSite();
try {
await verifyLaunchStorage(executable, site.origin);
await verifyConnectedStorage(endpoint, targetId, site.origin);
return {
mocked: false,
launch: true,
connect: true,
oopifInitialization: true,
firstInlineScriptSeeded: true,
updatesAndDeletesPersistAcrossNavigation: true,
newScenarioGetsFreshSeed: true,
externalTargetSurvives: true,
cancellationCleanup: true,
};
} finally {
await site.close();
}
}
const verifyLaunchStorage = async (executable, origin) => {
const scenario = scenarioFor(
{ mode: "launch", executable_path: executable },
origin,
);
const profilesBefore = await scenarioProfiles();
const result = await runScenarioCli(scenario);
const capture = result.normalized_result;
assert.ok(capture, "Launch-mode storage scenario returned no result");
assertLaunchStorageStates(capture);
assert.equal(capture.browser.cleanup, "terminated-owned-process");
const profilesAfter = await scenarioProfiles();
assert.ok([...profilesAfter].every((profile) => profilesBefore.has(profile)));
const freshScenario = browserScenarioSchema.parse({
...scenario,
actions: [scenario.actions[0]],
});
const fresh = await createBrowserScenarioProvider(
process.env,
).captureScenario(freshScenario);
assert.equal(fresh.ok, true);
if (!fresh.ok) throw fresh.error;
assertContainsState(domText(fresh.value.steps[0]), "main-report", {
local: "main-initial",
session: "main-session-initial",
removed: "main-remove-seed",
empty: "main-empty-seed",
});
};
const assertLaunchStorageStates = (capture) => {
assertContainsState(domText(capture.steps[0]), "main-report", {
local: "main-initial",
session: "main-session-initial",
removed: "main-remove-seed",
empty: "main-empty-seed",
});
assertContainsState(domText(capture.steps[1]), "frame-report", {
local: "frame-initial",
session: "frame-session-initial",
removed: "frame-remove-seed",
empty: "frame-empty-seed",
});
assertContainsState(domText(capture.steps[2]), "top-report", {
local: "frame-updated",
session: "frame-session-updated",
removed: null,
empty: "",
});
assertContainsState(domText(capture.steps[3]), "main-report", {
local: "main-updated",
session: "main-session-updated",
removed: null,
empty: "",
});
assertContainsState(domText(capture.steps[3]), "frame-report", {
local: "top-updated",
session: "top-session-updated",
removed: null,
empty: "",
});
};
const verifyConnectedStorage = async (endpoint, targetId, origin) => {
const scenario = scenarioFor(
{ mode: "connect", cdp_endpoint: endpoint, target_id: targetId },
origin,
);
const targets = await (
await fetch(`${endpoint}/json/list`, { signal: AbortSignal.timeout(2_000) })
).json();
const target = targets.find(({ id }) => id === targetId);
assert.ok(target, "External browser target is missing");
const observer = await CdpConnection.connect(
target.webSocketDebuggerUrl,
"capture_browser_scenario",
);
try {
await assertStorageKeyPartition(observer, origin);
const result = await runScenarioCli(scenario);
const capture = result.normalized_result;
assert.ok(capture, "Connect-mode storage scenario returned no result");
assertContainsState(domText(capture.steps[0]), "main-report", {
local: "main-initial",
session: "main-session-initial",
removed: "main-remove-seed",
empty: "main-empty-seed",
});
assert.equal(capture.browser.cleanup, "disconnected-external");
await verifyCallerDebuggerPause(observer, origin, scenario);
await verifyCancelledStorage(observer, endpoint, targetId, scenario);
} finally {
await raceWithTimeout(restoreTarget(observer, target.url), 1_000).catch(
() => undefined,
);
await raceWithTimeout(observer.close(), 1_000).catch(() => undefined);
}
};
const assertStorageKeyPartition = async (connection, origin) => {
const attachedSessions = new Set();
const unsubscribe = connection.onEvent((event) => {
if (
event.method === "Target.attachedToTarget" &&
typeof event.params === "object" &&
event.params !== null &&
"sessionId" in event.params &&
typeof event.params.sessionId === "string" &&
"targetInfo" in event.params &&
typeof event.params.targetInfo === "object" &&
event.params.targetInfo !== null &&
"type" in event.params.targetInfo &&
event.params.targetInfo.type === "iframe"
)
attachedSessions.add(event.params.sessionId);
});
await connection.send("Page.enable");
await connection.send("Target.setAutoAttach", {
autoAttach: true,
waitForDebuggerOnStart: false,
flatten: true,
});
try {
const start = await connection.send("Runtime.evaluate", {
expression: "performance.timeOrigin",
returnByValue: true,
});
await connection.send("Page.navigate", {
url: `${origin}/storage-main`,
});
await waitForPagePath(connection, "/storage-main", start.result.value);
let childSession;
let childFrameId;
const deadline = Date.now() + 10_000;
while (Date.now() < deadline && childFrameId === undefined) {
for (const sessionId of attachedSessions) {
try {
const tree = await connection.send(
"Page.getFrameTree",
{},
sessionId,
);
const frame = tree.frameTree?.frame;
if (
typeof frame?.id === "string" &&
frame.url ===
`${origin.replace("127.0.0.1", "localhost")}/storage-frame`
) {
childSession = sessionId;
childFrameId = frame.id;
break;
}
} catch {
// The iframe may detach between target discovery and frame inspection.
}
}
if (childFrameId === undefined) await delay(25);
}
assert.ok(
childSession && childFrameId,
"Storage fixture OOPIF was not attached",
);
const child = await connection.send(
"Storage.getStorageKey",
{ frameId: childFrameId },
childSession,
);
const rootTree = await connection.send("Page.getFrameTree");
const beforeRoot = await connection.send("Storage.getStorageKey", {
frameId: rootTree.frameTree.frame.id,
});
await connection.send("Page.navigate", {
url: `${origin.replace("127.0.0.1", "localhost")}/storage-top`,
});
await waitForPagePath(connection, "/storage-top", start.result.value);
const afterTree = await connection.send("Page.getFrameTree");
const topLevel = await connection.send("Storage.getStorageKey", {
frameId: afterTree.frameTree.frame.id,
});
assert.notEqual(
child.storageKey,
topLevel.storageKey,
`Chrome did not expose distinct partition keys: child=${JSON.stringify(child)} top=${JSON.stringify(topLevel)} rootBefore=${JSON.stringify(beforeRoot)}`,
);
} finally {
unsubscribe();
await connection.send("Target.setAutoAttach", {
autoAttach: false,
waitForDebuggerOnStart: false,
flatten: true,
});
}
};
const waitForPagePath = async (connection, path, priorTimeOrigin) => {
const deadline = Date.now() + 10_000;
while (Date.now() < deadline) {
const result = await connection.send("Runtime.evaluate", {
expression: `JSON.stringify({ path: location.pathname, time: performance.timeOrigin })`,
returnByValue: true,
});
const current = JSON.parse(result.result.value);
if (current.path === path && current.time !== priorTimeOrigin) return;
await delay(25);
}
throw new Error(`External browser did not navigate to ${path}`);
};
const verifyCallerDebuggerPause = async (caller, origin, scenario) => {
await caller.send("Runtime.enable");
await caller.send("Debugger.enable");
const controller = new AbortController();
const debuggerUrl = `${origin}/storage-debugger`;
const scripts = new Map();
const protocolTrace = [];
let pauseObserved = false;
let resolvePaused;
const paused = new Promise((resolve) => {
resolvePaused = (callFrameId) => {
pauseObserved = true;
resolve(callFrameId);
};
});
const unsubscribe = caller.onEvent((event) => {
if (event.method === "Debugger.scriptParsed") {
const params = event.params;
if (
typeof params === "object" &&
params !== null &&
"scriptId" in params &&
typeof params.scriptId === "string" &&
"url" in params &&
typeof params.url === "string"
)
scripts.set(params.scriptId, params.url);
if (
typeof params === "object" &&
params !== null &&
"url" in params &&
typeof params.url === "string" &&
params.url.includes("storage")
)
protocolTrace.push({ method: event.method, url: params.url });
} else if (event.method === "Debugger.paused") {
const params = event.params;
const callFrames =
typeof params === "object" && params !== null && "callFrames" in params
? params.callFrames
: undefined;
const frame = Array.isArray(callFrames) ? callFrames[0] : undefined;
const scriptId =
typeof frame === "object" && frame !== null && "location" in frame
? frame.location?.scriptId
: undefined;
protocolTrace.push({
method: event.method,
url: typeof scriptId === "string" ? scripts.get(scriptId) : undefined,
reason:
typeof params === "object" && params !== null && "reason" in params
? params.reason
: undefined,
});
if (
typeof frame === "object" &&
frame !== null &&
typeof frame.callFrameId === "string" &&
typeof scriptId === "string" &&
scripts.get(scriptId) === debuggerUrl
)
resolvePaused(frame.callFrameId);
}
});
const debuggerScenario = browserScenarioSchema.parse({
...scenario,
start_url: { url: debuggerUrl },
actions: [
{ step_id: "after-debugger", action: "wait_for_timeout", duration_ms: 1 },
],
});
let settled = false;
const pending = createBrowserScenarioProvider(process.env)
.captureScenario(debuggerScenario, { signal: controller.signal })
.finally(() => {
settled = true;
unsubscribe();
});
try {
const callFrameId = await raceWithTimeout(paused, 10_000, () => {
throw new Error(
`Caller debugger pause was not observed at ${debuggerUrl}; settled=${String(settled)} events=${JSON.stringify(protocolTrace)}`,
);
});
const pausedState = await caller.send("Debugger.evaluateOnCallFrame", {
callFrameId,
expression: "window.reaDebuggerAfter === true",
returnByValue: true,
});
assert.equal(
pausedState.result.value,
false,
"Caller-owned debugger frame resumed before the caller inspected it",
);
await delay(100);
assert.equal(
settled,
false,
"Storage observer resumed a caller-owned debugger pause",
);
await caller.send("Debugger.resume");
pauseObserved = false;
const result = await pending;
if (!result.ok) throw result.error;
const debuggerState = await caller.send("Runtime.evaluate", {
expression: "window.reaDebuggerAfter === true",
returnByValue: true,
});
assert.equal(debuggerState.result.value, true);
} catch (cause) {
if (!pauseObserved) controller.abort(cause);
else await caller.send("Debugger.resume").catch(() => undefined);
const completion = await raceWithTimeout(
pending.catch((pendingCause) => ({ ok: false, error: pendingCause })),
5_000,
);
if (completion === undefined)
throw new Error("Capture remained pending after cancellation", { cause });
const completionError =
typeof completion === "object" &&
completion !== null &&
"error" in completion
? completion.error
: typeof completion === "object" &&
completion !== null &&
"ok" in completion &&
completion.ok === false
? completion.error
: undefined;
const completionDetail =
completionError === undefined
? "capture did not report an error"
: (() => {
const causes = [];
let current = completionError;
for (
let depth = 0;
depth < 4 && current !== undefined;
depth += 1
) {
causes.push({
name: current.name,
tag: current._tag,
reason: current.reason,
message: current.message,
});
current = current.cause;
}
return JSON.stringify(causes);
})();
throw new Error(`${String(cause)}; capture=${completionDetail}`, {
cause,
});
} finally {
unsubscribe();
await raceWithTimeout(caller.send("Debugger.disable"), 1_000).catch(
() => undefined,
);
}
};
const verifyCancelledStorage = async (
observer,
endpoint,
targetId,
scenario,
) => {
const controller = new AbortController();
const cancelledScenario = browserScenarioSchema.parse({
...scenario,
actions: [
{
step_id: "cancelled-wait",
action: "wait_for",
locator: { kind: "css", selector: "#never-storage-ready" },
state: "visible",
timeout_ms: 30_000,
},
],
});
const priorDocument = await observer.send("Runtime.evaluate", {
expression: "performance.timeOrigin",
returnByValue: true,
});
const pending = createBrowserScenarioProvider(process.env).captureScenario(
cancelledScenario,
{ signal: controller.signal },
);
await waitForNewDocument(
observer,
"/storage-main",
priorDocument.result.value,
);
const state = await observer.send("Runtime.evaluate", {
expression:
"JSON.parse(document.querySelector('#main-report').textContent)",
returnByValue: true,
});
assert.deepEqual(state.result.value, {
local: "main-initial",
session: "main-session-initial",
removed: "main-remove-seed",
empty: "main-empty-seed",
});
controller.abort(new Error("cancel storage seed scenario"));
const result = await pending;
assert.ok(
!result.ok ||
result.value.steps.some(({ status }) => status === "cancelled"),
"Cancellation unexpectedly completed every scenario step",
);
const targets = await (await fetch(`${endpoint}/json/list`)).json();
assert.ok(
targets.some(({ id }) => id === targetId),
"Cancelled connect-mode storage cleanup closed the external target",
);
};
const waitForNewDocument = async (connection, path, priorTimeOrigin) => {
const deadline = Date.now() + 10_000;
while (Date.now() < deadline) {
const result = await connection.send("Runtime.evaluate", {
expression: `JSON.stringify({ path: location.pathname, time: performance.timeOrigin, state: document.querySelector('#main-report')?.textContent })`,
returnByValue: true,
});
const current = JSON.parse(result.result.value);
if (
current.path === path &&
current.time !== priorTimeOrigin &&
current.state !== undefined
)
return;
await new Promise((resolve) => setTimeout(resolve, 25));
}
throw new Error(`External browser did not navigate to ${path}`);
};
const restoreTarget = async (connection, url) => {
await connection.send("Page.navigate", { url });
const deadline = Date.now() + 10_000;
while (Date.now() < deadline) {
const result = await connection.send("Runtime.evaluate", {
expression: "location.href",
returnByValue: true,
});
if (result.result.value === url) return;
await new Promise((resolve) => setTimeout(resolve, 25));
}
throw new Error("External browser did not return to its original URL");
};
+18 -12
View File
@@ -65,7 +65,11 @@ function assertObservationEvents(result) {
throw new Error( throw new Error(
"Real Chrome attach-window console observation was missing", "Real Chrome attach-window console observation was missing",
); );
if (result.network.websocket_events.length < 1) if (
!result.network.websocket_connections.some(
(connection) => connection.events.length > 0,
)
)
throw new Error("Real Chrome WebSocket metadata was missing"); throw new Error("Real Chrome WebSocket metadata was missing");
} }
@@ -77,8 +81,8 @@ function assertObservationPrivacy(result) {
result.network.requests.some( result.network.requests.some(
(request) => request.body_shapes.status !== "not_approved", (request) => request.body_shapes.status !== "not_approved",
) || ) ||
result.network.websocket_events.some( result.network.websocket_connections.some((connection) =>
(event) => event.payload_shape !== null, connection.events.some((event) => event.payload_shape !== null),
) )
) )
throw new Error( throw new Error(
@@ -246,15 +250,17 @@ export function assertSensitiveShapes(result) {
) )
throw new Error("Real Chrome selected console text was not preserved"); throw new Error("Real Chrome selected console text was not preserved");
if ( if (
!result.network.websocket_events.some( !result.network.websocket_connections.some((connection) =>
(event) => connection.events.some(
event.payload_shape?.format === "json" && (event) =>
event.payload_shape.json_shape?.properties.some( event.payload_shape?.format === "json" &&
({ path }) => event.payload_shape.json_shape?.properties.some(
path.length === 1 && ({ path }) =>
path[0].kind === "property" && path.length === 1 &&
path[0].name === "token", path[0].kind === "property" &&
), path[0].name === "token",
),
),
) )
) )
throw new Error("Real Chrome WebSocket JSON shape was missing"); throw new Error("Real Chrome WebSocket JSON shape was missing");
@@ -62,6 +62,36 @@ export async function verifyGhidraAddressBoundaries(client, entry, cli) {
await query("read_bytes", { address: padded, length: 1 }), await query("read_bytes", { address: padded, length: 1 }),
await query("read_bytes", { address: entry, length: 1 }), await query("read_bytes", { address: entry, length: 1 }),
); );
const aliasKeys = [entry, padded];
assert.deepEqual(
await query("set_addresses_names", {
names: {
[entry]: "rea_same_address_alias",
[padded]: "rea_same_address_alias",
},
}),
Object.fromEntries(aliasKeys.map((address) => [address, true])),
"Equivalent address spellings with one requested name share one edit",
);
const nameBeforeAliasedBatch = await query("address_name", {
address: entry,
});
assert.equal(nameBeforeAliasedBatch, "rea_same_address_alias");
assert.deepEqual(
await query("set_addresses_names", {
names: {
[entry]: "rea_first_address_alias",
[padded]: "rea_second_address_alias",
},
}),
Object.fromEntries(aliasKeys.map((address) => [address, false])),
"Conflicting spellings of one Ghidra address must be rejected together",
);
assert.equal(
await query("address_name", { address: entry }),
nameBeforeAliasedBatch,
"A rejected alias collision must leave the preexisting target name unchanged",
);
for (const [command, flags, operation] of [ for (const [command, flags, operation] of [
["read-bytes", ["--length", "1"], "read_bytes"], ["read-bytes", ["--length", "1"], "read_bytes"],
[ [
+15
View File
@@ -58,6 +58,15 @@ export async function verifyGhidraBoundaries(
`${name}: ${JSON.stringify(validate.errors)}`, `${name}: ${JSON.stringify(validate.errors)}`,
); );
assert.deepEqual(reply.structuredContent, JSON.parse(mcpTextValue(reply))); assert.deepEqual(reply.structuredContent, JSON.parse(mcpTextValue(reply)));
if (name === "list_strings" || name === "search_strings")
assert.ok(
reply.structuredContent.limitations.some((limitation) =>
/Only Ghidra-defined string Data.*empty search does not establish/u.test(
limitation,
),
),
`${name} omitted defined-string coverage`,
);
successfulCalls++; successfulCalls++;
return result; return result;
}; };
@@ -378,6 +387,12 @@ export async function verifyGhidraBoundaries(
).some((item) => item.address === marker.address), ).some((item) => item.address === marker.address),
false, false,
); );
assert.deepEqual(
await call("search_strings", {
pattern: `REA_GHIDRA_ABSENT_${randomUUID()}`,
}),
[],
);
assert.equal( assert.equal(
(await call("inspect_native_instruction", { address: strings[0].address })) (await call("inspect_native_instruction", { address: strings[0].address }))
.status, .status,
+3 -1
View File
@@ -183,7 +183,9 @@ export async function verifyGhidraLargeResults({
assert.equal(dossier.procedure.name, "rea_frame_probe"); assert.equal(dossier.procedure.name, "rea_frame_probe");
assert.ok(dossier.comments.some(({ text }) => text === comment)); assert.ok(dossier.comments.some(({ text }) => text === comment));
assert.ok( assert.ok(
dossierRecord.raw_result.comments.some(({ text }) => text === comment), dossierRecord.raw_result.value.comments.some(
({ text }) => text === comment,
),
); );
const { stdout } = await exec( const { stdout } = await exec(
process.execPath, process.execPath,
@@ -62,6 +62,7 @@ export async function verifyGhidraNamespaceAnnotations({
assert.ok(symbol, `Missing demangled fixture global ${namespace}::value`); assert.ok(symbol, `Missing demangled fixture global ${namespace}::value`);
return { namespace, address: symbol.address }; return { namespace, address: symbol.address };
}); });
await verifyDuplicateAddressNames(call, procedures, names);
const ambiguous = await reject("procedure_address", { procedure: "same" }); const ambiguous = await reject("procedure_address", { procedure: "same" });
assert.equal(ambiguous.code, "invalid_request"); assert.equal(ambiguous.code, "invalid_request");
for (const { address } of entries) for (const { address } of entries)
@@ -133,7 +134,8 @@ export async function verifyGhidraNamespaceAnnotations({
}); });
assert.deepEqual(unchanged.annotations, qualified.annotations); assert.deepEqual(unchanged.annotations, qualified.annotations);
} }
await verifyAddressNames(call, [entries[0], entries[2], ...globals]); await verifyAddressNames(call, [entries[0], entries[2]], "function");
await verifyAddressNames(call, globals, "label");
for (const { namespace, address } of entries) { for (const { namespace, address } of entries) {
assert.equal( assert.equal(
await call("set_address_name", { await call("set_address_name", {
@@ -177,24 +179,107 @@ export async function verifyGhidraNamespaceAnnotations({
} }
} }
async function verifyAddressNames(call, symbols) { async function verifyDuplicateAddressNames(call, procedures, names) {
for (const { namespace, address } of symbols) { const functionEntries = [
"rea_duplicate_function_a",
"rea_duplicate_function_b",
].map((name) => {
const procedure = procedures.find(
({ value }) => value === name || value === `_${name}`,
);
assert.ok(procedure, `Missing duplicate-name fixture function ${name}`);
return { address: procedure.address, original: procedure.value };
});
const labelEntries = ["rea_duplicate_label_a", "rea_duplicate_label_b"].map(
(name) => {
const symbol = names.find(
(item) =>
(item.value === name || item.value === `_${name}`) &&
item.symbol.primary,
);
assert.ok(symbol, `Missing duplicate-name fixture label ${name}`);
return { address: symbol.address, original: symbol.value };
},
);
for (const [kind, entries] of [
["function", functionEntries],
["label", labelEntries],
]) {
const [owner, duplicate] = entries;
const name = `rea_duplicate_${kind}`;
assert.equal( assert.equal(
await call("set_address_name", { address, name: "address_renamed" }), await call("set_address_name", { address: owner.address, name }),
true, true,
); );
const qualified = `${namespace}::address_qualified`; assert.equal(
await call("set_address_name", { address: owner.address, name }),
true,
"Renaming the current owner to its existing name remains idempotent",
);
assert.equal(
await call("set_address_name", { address: duplicate.address, name }),
false,
);
assert.equal(
await call("address_name", { address: duplicate.address }),
duplicate.original,
"A rejected duplicate leaves the other address unchanged",
);
const batchName = `rea_batch_duplicate_${kind}`;
assert.deepEqual(
await call("set_addresses_names", {
names: {
[owner.address]: batchName,
[duplicate.address]: batchName,
},
}),
{ [owner.address]: true, [duplicate.address]: false },
);
assert.equal(
await call("address_name", { address: owner.address }),
batchName,
);
assert.equal(
await call("address_name", { address: duplicate.address }),
duplicate.original,
"A failed batch member rolls back only its own address",
);
const owners = (await call("list_names")).filter(
(item) => item.value === batchName && item.symbol.primary,
);
assert.deepEqual(
owners.map(({ address }) => address),
[owner.address],
);
}
}
async function verifyAddressNames(call, symbols, kind) {
for (const { namespace, address } of symbols) {
assert.equal(
await call("set_address_name", {
address,
name: `address_renamed_${kind}`,
}),
true,
);
const qualified = `${namespace}::address_qualified_${kind}`;
assert.equal( assert.equal(
await call("set_address_name", { address, name: qualified }), await call("set_address_name", { address, name: qualified }),
true, true,
); );
assert.equal(await call("address_name", { address }), qualified); assert.equal(await call("address_name", { address }), qualified);
assert.equal( assert.equal(
await call("set_address_name", { address, name: "other::literal" }), await call("set_address_name", {
address,
name: `other::literal_${kind}`,
}),
true, true,
); );
const literal = await call("address_name", { address }); const literal = await call("address_name", { address });
assert.equal(literal, `${namespace}::other::literal`); assert.equal(literal, `${namespace}::other::literal_${kind}`);
assert.equal( assert.equal(
await call("set_address_name", { address, name: literal }), await call("set_address_name", { address, name: literal }),
true, true,
@@ -235,7 +320,7 @@ async function verifyAddressNames(call, symbols) {
assert.deepEqual( assert.deepEqual(
await call("set_addresses_names", { await call("set_addresses_names", {
names: { names: {
[valid.address]: `${valid.namespace}::batch_renamed`, [valid.address]: `${valid.namespace}::batch_renamed_${kind}`,
[invalid.address]: `${invalid.namespace}::`, [invalid.address]: `${invalid.namespace}::`,
}, },
}), }),
@@ -243,18 +328,18 @@ async function verifyAddressNames(call, symbols) {
); );
assert.equal( assert.equal(
await call("address_name", { address: valid.address }), await call("address_name", { address: valid.address }),
`${valid.namespace}::batch_renamed`, `${valid.namespace}::batch_renamed_${kind}`,
); );
assert.equal( assert.equal(
await call("address_name", { address: invalid.address }), await call("address_name", { address: invalid.address }),
`${invalid.namespace}::address_qualified`, `${invalid.namespace}::address_qualified_${kind}`,
); );
const names = await call("list_names"); const names = await call("list_names");
for (const { namespace, address } of symbols) { for (const { namespace, address } of symbols) {
const expected = const expected =
address === valid.address address === valid.address
? `${namespace}::batch_renamed` ? `${namespace}::batch_renamed_${kind}`
: `${namespace}::address_qualified`; : `${namespace}::address_qualified_${kind}`;
assert.ok( assert.ok(
names.some( names.some(
(item) => (item) =>
+149
View File
@@ -0,0 +1,149 @@
import { z } from "zod";
import {
releaseAuthorSchema as author,
releaseIssueSchema as issue,
releasePullRequestSchema,
} from "./release-notes.mjs";
const pageInfo = z.object({
hasNextPage: z.boolean(),
endCursor: z.string().nullable(),
});
const connection = (node) => z.object({ nodes: z.array(node), pageInfo });
const associated = z.object({
number: z.number().int().positive(),
mergeCommit: z.object({ oid: z.string() }).nullable(),
});
const pr = releasePullRequestSchema.omit({ issues: true }).extend({
closingIssuesReferences: connection(issue),
});
const PAGE = "pageInfo { hasNextPage endCursor }";
const ACTOR = "login __typename";
const AUTHORS = `authors(first:100, after:$cursor) { nodes { name email user { ${ACTOR} } } ${PAGE} }`;
const ASSOCIATED = `associatedPullRequests(first:100, after:$cursor) { nodes { number mergeCommit { oid } } ${PAGE} }`;
const ISSUES = `closingIssuesReferences(first:100, after:$cursor) { nodes { number url author { ${ACTOR} } repository { nameWithOwner } } ${PAGE} }`;
async function query(run, repository, selection, cursor = null) {
const [owner, name] = repository.split("/");
const response = JSON.parse(
await run(
"gh",
["api", "graphql", "--input", "-"],
JSON.stringify({
query: `query($owner:String!,$name:String!,$cursor:String) { repository(owner:$owner,name:$name) { ${selection} } }`,
variables: { owner, name, cursor },
}),
),
);
if (response.errors?.length)
throw new Error(
`GitHub inventory query failed: ${JSON.stringify(response.errors)}`,
);
if (!response.data?.repository)
throw new Error(`GitHub repository unavailable: ${repository}`);
return response.data.repository;
}
async function remainingPages(run, repository, selection, field) {
const initial = field.initial;
const nodes = [...initial.nodes];
let page = initial;
const cursors = new Set();
while (page.pageInfo.hasNextPage) {
const cursor = page.pageInfo.endCursor;
if (!cursor || cursors.has(cursor))
throw new Error(`GitHub returned a non-advancing ${field.name} cursor`);
cursors.add(cursor);
const response = await query(run, repository, selection, cursor);
page = field.schema.parse(response.item?.[field.name]);
nodes.push(...page.nodes);
}
return nodes;
}
/** GitHub's commit authors include co-authors; unresolved identities remain explicit. */
export async function collectGitHubCredits(run, repository, commits) {
const shipped = new Set(commits.map((commit) => commit.sha));
const numbers = new Set();
const result = [];
// Small query batches limit transient GraphQL work, never the history examined.
for (let offset = 0; offset < commits.length; offset += 20) {
const batch = commits.slice(offset, offset + 20);
const selection = batch
.map(
(commit, index) =>
`c${index}: object(oid:${JSON.stringify(commit.sha)}) { ... on Commit { oid ${AUTHORS} ${ASSOCIATED} } }`,
)
.join("\n");
const response = await query(run, repository, selection);
for (const [index, commit] of batch.entries()) {
const value = z
.object({
oid: z.literal(commit.sha),
authors: connection(author),
associatedPullRequests: connection(associated),
})
.parse(response[`c${index}`]);
const select = (fields) =>
`item: object(oid:${JSON.stringify(commit.sha)}) { ... on Commit { ${fields} } }`;
const authors = await remainingPages(run, repository, select(AUTHORS), {
name: "authors",
schema: connection(author),
initial: value.authors,
});
const associations = await remainingPages(
run,
repository,
select(ASSOCIATED),
{
name: "associatedPullRequests",
schema: connection(associated),
initial: value.associatedPullRequests,
},
);
const pullRequests = [
...new Set(
associations
.filter((item) => shipped.has(item.mergeCommit?.oid))
.map((item) => item.number),
),
];
for (const number of pullRequests) numbers.add(number);
result.push({ ...commit, authors, pullRequests });
}
}
const pullRequests = [];
const sorted = [...numbers].sort((a, b) => a - b);
for (let offset = 0; offset < sorted.length; offset += 20) {
const batch = sorted.slice(offset, offset + 20);
const fields = `number title url author { ${ACTOR} } mergeCommit { oid } ${ISSUES}`;
const response = await query(
run,
repository,
batch
.map(
(number, index) =>
`p${index}: pullRequest(number:${number}) { ${fields} }`,
)
.join("\n"),
);
for (const [index, number] of batch.entries()) {
const value = pr.parse(response[`p${index}`]);
if (value.number !== number || !shipped.has(value.mergeCommit.oid))
throw new Error(`PR #${number} changed outside the selected history`);
const issues = await remainingPages(
run,
repository,
`item: pullRequest(number:${number}) { ${ISSUES} }`,
{
name: "closingIssuesReferences",
schema: connection(issue),
initial: value.closingIssuesReferences,
},
);
const { closingIssuesReferences: _connection, ...metadata } = value;
pullRequests.push({ ...metadata, issues });
}
}
return { commits: result, pullRequests };
}
+118
View File
@@ -0,0 +1,118 @@
import { valid } from "semver";
import { z } from "zod";
import {
releaseInventorySchema,
verifyInventoryHistory,
verifyReleaseNotes,
} from "./release-notes.mjs";
const sha = z.string().regex(/^[a-f0-9]{40}$/u);
const repository = z.string().regex(/^[\w.-]+\/[\w.-]+$/u);
const pullRequest = z.object({
number: z.number().int().positive(),
body: z.string(),
merged_at: z.string().nullable(),
merge_commit_sha: sha.nullable(),
head: z.object({
ref: z.string(),
repo: z.object({ full_name: z.string() }).nullable(),
}),
labels: z.array(z.object({ name: z.string() })),
});
/** Read the reviewed Git snapshot and live PR before Release Please can mutate tags. */
export async function verifyNotesPublication(run, options) {
const repo = repository.parse(options.repository);
const source = sha.parse(
(
await run("git", [
"rev-parse",
"--verify",
"--end-of-options",
`${options.source}^{commit}`,
])
).trim(),
);
const read = (path) => run("git", ["show", `${source}:${path}`]);
const { version } = z
.object({ version: z.string() })
.parse(JSON.parse(await read("package.json")));
if (valid(version) !== version || version.includes("+"))
throw new Error("Invalid reviewed package version");
let candidates;
if (options.pr !== undefined) {
const number = z.coerce.number().int().positive().parse(options.pr);
candidates = [
pullRequest.parse(
JSON.parse(await run("gh", ["api", `repos/${repo}/pulls/${number}`])),
),
];
} else {
candidates = z
.array(z.array(pullRequest))
.parse(
JSON.parse(
await run("gh", [
"api",
"--paginate",
"--slurp",
`repos/${repo}/commits/${source}/pulls?per_page=100`,
]),
),
)
.flat();
}
const matching = candidates.filter(
(pr) =>
pr.merged_at !== null &&
pr.merge_commit_sha === source &&
pr.head.repo?.full_name.toLowerCase() === repo.toLowerCase() &&
pr.head.ref.startsWith("release-please--branches--"),
);
if (matching.length !== 1)
throw new Error(
"Expected exactly one merged Release Please PR for the reviewed source",
);
const pr = matching[0];
const path = `docs/releases/${version}.contributions.json`;
const present =
(
await run("git", ["ls-tree", "--name-only", source, "--", path])
).trim() === path;
if (!present) {
if (pr.labels.some((label) => label.name === "rea:release-notes-finalized"))
throw new Error(
"Finalized release PR is missing its contribution inventory",
);
return { status: "legacy", version, pullRequest: pr.number };
}
const inventory = releaseInventorySchema.parse(JSON.parse(await read(path)));
if (
inventory.version !== version ||
inventory.repository.toLowerCase() !== repo.toLowerCase()
)
throw new Error(
"Contribution inventory differs from the reviewed release version or repository",
);
const baseline = z
.object({ ".": z.string() })
.parse(
JSON.parse(
await run("git", ["show", `${source}^1:.release-please-manifest.json`]),
),
)["."];
if (valid(baseline) !== baseline || baseline.includes("+"))
throw new Error("Invalid release baseline version");
const baselineSha = (
await run("git", ["rev-parse", `refs/tags/rea-agents-${baseline}^{commit}`])
).trim();
if (inventory.base !== baselineSha)
throw new Error(
"Contribution inventory does not start at the published baseline tag",
);
await verifyInventoryHistory(run, inventory, source);
return {
status: "verified",
...verifyReleaseNotes(inventory, await read("CHANGELOG.md"), pr.body),
};
}
+415
View File
@@ -0,0 +1,415 @@
import { valid } from "semver";
import { z } from "zod";
const sha = z.string().regex(/^[a-f0-9]{40}$/u);
const actor = z.object({
login: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9-]*(?:\[bot\])?$/u),
__typename: z.string(),
});
const author = z.object({
name: z.string(),
email: z.string(),
user: actor.nullable(),
});
const commit = z.object({
sha,
subject: z.string(),
body: z.string(),
parents: z.array(sha),
authors: z.array(author),
pullRequests: z.array(z.number().int().positive()),
});
const issue = z.object({
number: z.number().int().positive(),
url: z.url(),
author: actor.nullable(),
repository: z.object({ nameWithOwner: z.string() }),
});
const pullRequest = z.object({
number: z.number().int().positive(),
title: z.string(),
url: z.url(),
author: actor.nullable(),
mergeCommit: z.object({ oid: sha }),
issues: z.array(issue),
});
export {
author as releaseAuthorSchema,
issue as releaseIssueSchema,
pullRequest as releasePullRequestSchema,
};
export const releaseInventorySchema = z.object({
schemaVersion: z.literal(1),
repository: z.string().regex(/^[\w.-]+\/[\w.-]+$/u),
version: z
.string()
.refine(
(value) => valid(value) === value && !value.includes("+"),
"Expected exact release SemVer",
),
base: sha,
target: sha,
commits: z.array(commit),
pullRequests: z.array(pullRequest),
});
export async function readReleaseHistory(run, base, target) {
if (
(await run("git", ["rev-parse", "--is-shallow-repository"])).trim() !==
"false"
)
throw new Error(
"Release inventory requires complete Git history; fetch --unshallow first",
);
const resolve = async (ref) =>
sha.parse(
(
await run("git", [
"rev-parse",
"--verify",
"--end-of-options",
`${ref}^{commit}`,
])
).trim(),
);
const baseSha = await resolve(base);
const targetSha = await resolve(target);
await run("git", ["merge-base", "--is-ancestor", baseSha, targetSha]);
const text = await run("git", [
"log",
"--reverse",
"--topo-order",
"-z",
"--format=%H%x00%s%x00%B%x00%P",
`${baseSha}..${targetSha}`,
]);
const fields = text.split("\0");
if (fields.pop() !== "" || fields.length % 4)
throw new Error("Malformed Git release history");
const commits = [];
for (let index = 0; index < fields.length; index += 4) {
commits.push({
sha: sha.parse(fields[index]),
subject: fields[index + 1],
body: fields[index + 2],
parents: fields[index + 3].split(" ").filter(Boolean),
});
}
return { base: baseSha, target: targetSha, commits };
}
// REA history maps Anthropic and Cursor's agent trailers to ordinary User actors.
// Keep those source identities in the inventory while excluding automation credit.
const AGENT_ACCOUNTS = new Set(["claude", "cursoragent"]);
const human = (value) =>
value?.__typename === "User" &&
!value.login.endsWith("[bot]") &&
!AGENT_ACCOUNTS.has(value.login.toLowerCase())
? value.login
: undefined;
const unique = (values) =>
[...new Set(values.filter(Boolean))].sort((a, b) =>
a.toLowerCase().localeCompare(b.toLowerCase()),
);
function prCredits(inventory, pr) {
return unique([
human(pr.author),
...pr.issues.map((item) => human(item.author)),
...inventory.commits
.filter(
(item) =>
item.parents.length === 1 && item.pullRequests.includes(pr.number),
)
.flatMap((item) => item.authors.map((identity) => human(identity.user))),
]);
}
function thanks(handles) {
return handles.length
? ` Thanks ${handles.map((handle) => `@${handle}`).join(", ")}.`
: "";
}
export function contributionRecord(raw) {
const inventory = releaseInventorySchema.parse(raw);
const lines = [
"### Complete contribution record",
"",
`History: \`${inventory.base}..${inventory.target}\`; ${inventory.pullRequests.length} merged PRs, ${inventory.commits.length} commits examined.`,
"",
];
for (const pr of inventory.pullRequests) {
const issues = pr.issues
.map(
(item) =>
`[${item.repository.nameWithOwner}#${item.number}](${item.url})`,
)
.join(", ");
lines.push(
`- **PR [#${pr.number}](${pr.url})**${issues ? ` Related ${issues}.` : ""}${thanks(prCredits(inventory, pr))}`,
);
}
const direct = inventory.commits.filter(
(item) => item.pullRequests.length === 0 && item.parents.length <= 1,
);
if (direct.length) {
lines.push("", "#### Direct contributions", "");
for (const item of direct)
lines.push(
`- [${item.sha.slice(0, 7)}](https://github.com/${inventory.repository}/commit/${item.sha})${thanks(unique(item.authors.map((identity) => human(identity.user))))}`,
);
}
return `${lines.join("\n")}\n`;
}
export function selectedReleaseNotes(text, version) {
const headings = [...text.matchAll(/^## \[([^\]]+)\].*$/gmu)];
if (headings[0]?.[1] !== version)
throw new Error(`Notes must start with release [${version}]`);
return text
.slice(headings[0].index, headings[1]?.index ?? text.length)
.trim();
}
// Notes have one Markdown list item per line; fenced examples are never credit sources.
function proseBullets(text) {
let fence;
return text.split("\n").filter((line) => {
const marker = /^\s*(`{3,}|~{3,})/u.exec(line)?.[1];
if (marker) {
if (!fence) fence = marker;
else if (marker[0] === fence[0] && marker.length >= fence.length)
fence = undefined;
return false;
}
return !fence && /^[-*] /u.test(line);
});
}
function bulletPrs(bullet, repository) {
// Resolve explicit links first so cross-repository numbers cannot acquire local credit.
const local = [];
const withoutLinks = bullet.replace(
/\[([^\]]*)\]\(https:\/\/github\.com\/([^/]+\/[^/]+)\/(?:pull|issues)\/(\d+)\)/gu,
(_match, _label, repo, number) => {
if (repo.toLowerCase() === repository.toLowerCase())
local.push(Number(number));
return "";
},
);
for (const match of withoutLinks.matchAll(/(?<![\w/])#(\d+)\b/gu))
local.push(Number(match[1]));
return new Set(local);
}
function requiredCredits(inventory, bullet, references) {
const required = [];
for (const pr of inventory.pullRequests.filter((item) =>
references.has(item.number),
)) {
for (const handle of prCredits(inventory, pr))
required.push({ handle, context: `PR #${pr.number}` });
}
for (const issue of inventory.pullRequests.flatMap((item) => item.issues)) {
const mentioned =
(issue.repository.nameWithOwner.toLowerCase() ===
inventory.repository.toLowerCase() &&
references.has(issue.number)) ||
bullet.includes(`](${issue.url})`);
const handle = human(issue.author);
if (mentioned && handle)
required.push({
handle,
context: `issue ${issue.repository.nameWithOwner}#${issue.number}`,
});
}
for (const match of bullet.matchAll(
/https:\/\/github\.com\/([^/]+\/[^/]+)\/commit\/([a-f0-9]{7,40})\b/gu,
)) {
if (match[1].toLowerCase() !== inventory.repository.toLowerCase()) continue;
const matches = inventory.commits.filter((item) =>
item.sha.startsWith(match[2]),
);
if (matches.length !== 1)
throw new Error(`Unknown or ambiguous release commit ${match[2]}`);
if (matches[0].parents.length > 1) continue;
for (const identity of matches[0].authors) {
const handle = human(identity.user);
if (handle) required.push({ handle, context: `commit ${match[2]}` });
}
}
return required;
}
function verifyEditorialCredits(inventory, prose) {
const known = new Set(inventory.pullRequests.map((pr) => pr.number));
const issues = new Set(
inventory.pullRequests.flatMap((pr) =>
pr.issues
.filter(
(item) =>
item.repository.nameWithOwner.toLowerCase() ===
inventory.repository.toLowerCase(),
)
.map((item) => item.number),
),
);
const eligible = new Set(
unique([
...inventory.pullRequests.flatMap((pr) => prCredits(inventory, pr)),
...inventory.commits.flatMap((item) =>
item.authors.map((identity) => human(identity.user)),
),
]).map((handle) => handle.toLowerCase()),
);
const errors = [];
for (const bullet of proseBullets(prose)) {
const references = bulletPrs(bullet, inventory.repository);
const start = bullet.indexOf("Thanks ");
const thanked = new Set(
[
...bullet
.slice(start < 0 ? bullet.length : start)
.matchAll(/@([A-Za-z0-9-]+(?:\[bot\])?)/gu),
].map((match) => match[1].toLowerCase()),
);
for (const number of references)
if (!known.has(number) && !issues.has(number))
errors.push(`Unknown release reference #${number}`);
for (const { handle, context } of requiredCredits(
inventory,
bullet,
references,
)) {
if (!thanked.has(handle.toLowerCase()))
errors.push(`Missing Thanks @${handle} in bullet for ${context}`);
}
for (const handle of thanked)
if (!eligible.has(handle))
errors.push(`Unverified human credit @${handle}`);
}
return errors;
}
export function verifyReleaseNotes(raw, text, body) {
const inventory = releaseInventorySchema.parse(raw);
const notes = selectedReleaseNotes(text, inventory.version);
// Release Please's inline issue/PR body limit (pull-request-overflow-handler).
if (notes.length > 65_536 || (body !== undefined && body.length > 65_536))
throw new Error(
"Release notes exceed GitHub's inline PR body limit; overflow-link bodies are not supported by this checker",
);
if (notes.length > 125_000 || Buffer.byteLength(notes) > 125_000)
throw new Error(
"Release notes exceed GitHub's body budget; shorten prose without dropping credit",
);
const record = contributionRecord(inventory).trim();
const start = notes.indexOf("### Complete contribution record");
if (start < 0 || notes.slice(start).trim() !== record)
throw new Error(
"Contribution record differs from inventory; regenerate it with release-notes record",
);
const prose = notes.slice(0, start);
const errors = verifyEditorialCredits(inventory, prose);
for (const heading of ["Highlights", "Changes", "Fixes"])
if (!prose.includes(`### ${heading}\n`))
errors.push(`Missing ${heading} section`);
if (body !== undefined) {
const bodyNotes = selectedReleaseNotes(body, inventory.version)
.replace(
/\n---\nThis PR was generated with \[Release Please\][\s\S]*$/u,
"",
)
.trim();
if (bodyNotes !== notes)
errors.push("Release PR body differs from CHANGELOG.md");
}
if (errors.length) throw new Error(errors.join("\n"));
return {
version: inventory.version,
pullRequests: inventory.pullRequests.length,
commits: inventory.commits.length,
unresolvedAuthors: inventory.commits.flatMap((item) =>
item.authors
.filter((identity) => identity.user === null)
.map((identity) => ({ sha: item.sha, ...identity })),
),
};
}
/** Reject stale inventories and product changes made after the editorial target. */
export async function verifyInventoryHistory(run, raw, source) {
const inventory = releaseInventorySchema.parse(raw);
const history = await readReleaseHistory(
run,
inventory.base,
inventory.target,
);
const original = inventory.commits.map(
({ authors: _authors, pullRequests: _prs, ...item }) => item,
);
if (JSON.stringify(history.commits) !== JSON.stringify(original))
throw new Error("Inventory does not match complete local Git history");
const shas = new Set(history.commits.map((item) => item.sha));
const numbers = new Set();
for (const pr of inventory.pullRequests) {
if (numbers.has(pr.number) || !shas.has(pr.mergeCommit.oid))
throw new Error(`Duplicate or out-of-range PR #${pr.number}`);
numbers.add(pr.number);
}
for (const item of inventory.commits)
for (const number of item.pullRequests)
if (!numbers.has(number))
throw new Error(`Commit refers to missing PR #${number}`);
if (source !== undefined) {
const resolved = sha.parse(
(
await run("git", [
"rev-parse",
"--verify",
"--end-of-options",
`${source}^{commit}`,
])
).trim(),
);
const parent = (await run("git", ["rev-parse", `${resolved}^1`])).trim();
if (parent !== inventory.target)
throw new Error(
"Release base moved after notes target; regenerate inventory for the reviewed merge's first parent",
);
await run("git", [
"merge-base",
"--is-ancestor",
inventory.target,
resolved,
]);
const allowed = new Set([
"CHANGELOG.md",
".release-please-manifest.json",
"package.json",
"package-lock.json",
"server.json",
"src/generatedPackageMetadata.ts",
"docs/installation.md",
`docs/releases/${inventory.version}.contributions.json`,
]);
const paths = (
await run("git", [
"diff",
"--name-only",
"--no-renames",
"-z",
inventory.target,
resolved,
])
)
.split("\0")
.filter(Boolean);
const product = paths.filter((path) => !allowed.has(path));
if (product.length)
throw new Error(
`Product changed after notes target; regenerate inventory: ${product.join(", ")}`,
);
}
}
+16 -1
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env node #!/usr/bin/env node
import { access } from "node:fs/promises"; import { access } from "node:fs/promises";
import { dirname, resolve } from "node:path"; import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
// Route production MCP before importing Incur. Incur owns registration helpers // Route production MCP before importing Incur. Incur owns registration helpers
@@ -15,6 +15,21 @@ const isMcpMode =
args.length === 1 && (args[0] === "--mcp" || args[0] === "mcp"); args.length === 1 && (args[0] === "--mcp" || args[0] === "mcp");
const isMcpDoctorMode = args[0] === "mcp" && args[1] === "doctor"; const isMcpDoctorMode = args[0] === "mcp" && args[1] === "doctor";
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
// Playwright's registry assumes one of its three desktop host platforms when
// it chooses the browser cache directory. Termux reports `android`, although
// its Node runtime and filesystem are POSIX-compatible. Keep the cache path
// explicit so REA can start on Termux. Browser launch still requires a
// caller-supplied Android-compatible Chromium executable.
if (
process.platform === "android" &&
process.env.PLAYWRIGHT_BROWSERS_PATH === undefined
) {
const cacheRoot =
process.env.XDG_CACHE_HOME ?? join(process.env.HOME ?? ".", ".cache");
process.env.PLAYWRIGHT_BROWSERS_PATH = join(cacheRoot, "ms-playwright");
}
const runtimeFiles = isMcpMode const runtimeFiles = isMcpMode
? ["dist/main.js"] ? ["dist/main.js"]
: isMcpDoctorMode : isMcpDoctorMode
+135
View File
@@ -0,0 +1,135 @@
#!/usr/bin/env node
import { execFile } from "node:child_process";
import { readFile, writeFile } from "node:fs/promises";
import { parseArgs, promisify } from "node:util";
import { collectGitHubCredits } from "./lib/release-notes-github.mjs";
import { verifyNotesPublication } from "./lib/release-notes-publication.mjs";
import {
contributionRecord,
readReleaseHistory,
releaseInventorySchema,
selectedReleaseNotes,
verifyInventoryHistory,
verifyReleaseNotes,
} from "./lib/release-notes.mjs";
const execFileAsync = promisify(execFile);
async function run(command, args, input) {
const pending = execFileAsync(command, args, {
encoding: "utf8",
maxBuffer: 64 * 1024 * 1024,
});
pending.child.stdin.end(input);
return (await pending).stdout;
}
async function createInventory(values) {
if (!values.repository || !values.base || !values.target || !values.version)
throw new Error(
"inventory requires --repository, --base, --target and --version",
);
// Parse caller-selected boundaries before making network requests.
const history = await readReleaseHistory(run, values.base, values.target);
releaseInventorySchema.omit({ commits: true, pullRequests: true }).parse({
schemaVersion: 1,
repository: values.repository,
version: values.version,
base: history.base,
target: history.target,
});
const credits = await collectGitHubCredits(
run,
values.repository,
history.commits,
);
const inventory = releaseInventorySchema.parse({
schemaVersion: 1,
repository: values.repository,
version: values.version,
base: history.base,
target: history.target,
...credits,
});
await verifyInventoryHistory(run, inventory);
await writeFile(values.manifest, `${JSON.stringify(inventory, null, 2)}\n`, {
flag: "wx",
});
process.stdout.write(
`Inventoried ${inventory.commits.length} commits and ${inventory.pullRequests.length} merged PRs. Saved ${values.manifest}\n`,
);
}
async function main() {
const { values, positionals } = parseArgs({
allowPositionals: true,
options: {
repository: { type: "string" },
pr: { type: "string" },
base: { type: "string" },
target: { type: "string" },
version: { type: "string" },
manifest: { type: "string" },
notes: { type: "string", default: "CHANGELOG.md" },
"pr-body": { type: "string" },
source: { type: "string" },
output: { type: "string" },
},
});
const [command] = positionals;
if (command === "publication" && positionals.length === 1) {
if (!values.repository || !values.source)
throw new Error(
"publication requires --repository and --source; --pr selects the merged release PR",
);
process.stdout.write(
`${JSON.stringify(await verifyNotesPublication(run, values), null, 2)}\n`,
);
return;
}
if (
positionals.length !== 1 ||
!["inventory", "record", "check", "render"].includes(command) ||
!values.manifest
)
throw new Error(
"Usage: release-notes inventory|record|check|render --manifest FILE [--repository OWNER/REPO --base TAG --target SHA --version VERSION] [--notes FILE --pr-body FILE --source SHA --output FILE]",
);
if (command === "inventory") {
await createInventory(values);
return;
}
const inventory = releaseInventorySchema.parse(
JSON.parse(await readFile(values.manifest, "utf8")),
);
await verifyInventoryHistory(run, inventory, values.source);
if (command === "record") {
if (values.output)
await writeFile(values.output, contributionRecord(inventory), {
flag: "wx",
});
else process.stdout.write(contributionRecord(inventory));
return;
}
const notes = await readFile(values.notes, "utf8");
const body = values["pr-body"]
? await readFile(values["pr-body"], "utf8")
: undefined;
const report = verifyReleaseNotes(inventory, notes, body);
if (command === "check") {
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
return;
}
if (!values.output)
throw new Error(
"render requires --output (an existing file is never overwritten)",
);
const rendered = selectedReleaseNotes(notes, inventory.version);
await writeFile(values.output, `${rendered}\n`, { flag: "wx" });
}
await main().catch((error) => {
process.stderr.write(
`Release notes failed: ${error instanceof Error ? error.message : String(error)}\n`,
);
process.exitCode = 1;
});
@@ -175,6 +175,7 @@ const callerDirectory = join(workspace, "caller cwd with spaces");
const scriptCollisions = [ const scriptCollisions = [
"ReaGhidraBridge.java", "ReaGhidraBridge.java",
"ReaGhidraPrepareCom.java", "ReaGhidraPrepareCom.java",
"ReaGhidraApplySeeds.java",
"ReaGhidraNoReturnFix.java", "ReaGhidraNoReturnFix.java",
]; ];
await mkdir(callerDirectory); await mkdir(callerDirectory);
+17 -1
View File
@@ -45,6 +45,7 @@ import { verifyBrowserDomDestinations } from "./lib/browser-dom-destinations-e2e
import { verifyBrowserCaptureMetadataBudget } from "./lib/browser-capture-metadata-budget-e2e.mjs"; import { verifyBrowserCaptureMetadataBudget } from "./lib/browser-capture-metadata-budget-e2e.mjs";
import { artifactCliEvidence, artifactMcpResult } from "./lib/artifact-e2e.mjs"; import { artifactCliEvidence, artifactMcpResult } from "./lib/artifact-e2e.mjs";
import { verifyScenarioEnvironment } from "./lib/browser-scenario-environment-e2e.mjs"; import { verifyScenarioEnvironment } from "./lib/browser-scenario-environment-e2e.mjs";
import { verifyScenarioStorage } from "./lib/browser-scenario-storage-e2e.mjs";
const REAL_BROWSER_STARTUP_TIMEOUT_MS = 60_000; const REAL_BROWSER_STARTUP_TIMEOUT_MS = 60_000;
const SCENARIO_SECRET_VALUE = "rea-browser-verifier-secret"; const SCENARIO_SECRET_VALUE = "rea-browser-verifier-secret";
@@ -64,6 +65,7 @@ try {
executable, executable,
[ [
"--headless=new", "--headless=new",
"--password-store=basic",
...(process.env.REA_BROWSER_NO_SANDBOX === "true" ...(process.env.REA_BROWSER_NO_SANDBOX === "true"
? ["--no-sandbox"] ? ["--no-sandbox"]
: []), : []),
@@ -275,11 +277,18 @@ try {
throw new Error("Scenario attachment terminated its external browser"); throw new Error("Scenario attachment terminated its external browser");
const profilesBefore = await scenarioProfiles(); const profilesBefore = await scenarioProfiles();
process.stderr.write("Browser verifier: scenario environment\n");
const scenarioEnvironment = await verifyScenarioEnvironment( const scenarioEnvironment = await verifyScenarioEnvironment(
endpoint, endpoint,
target, target,
site.origin, site.origin,
); );
process.stderr.write("Browser verifier: scenario storage\n");
const scenarioStorage = await verifyScenarioStorage({
executable,
endpoint,
targetId: target,
});
const launchedScenario = await createBrowserScenarioProvider( const launchedScenario = await createBrowserScenarioProvider(
process.env, process.env,
).captureScenario( ).captureScenario(
@@ -332,7 +341,9 @@ try {
`Browser scenario included unselected structured value: ${unselectedValue}`, `Browser scenario included unselected structured value: ${unselectedValue}`,
); );
process.stderr.write("Browser verifier: large screenshot\n");
const largeScreenshot = await verifyLargeScreenshotE2e(endpoint, site.origin); const largeScreenshot = await verifyLargeScreenshotE2e(endpoint, site.origin);
process.stderr.write("Browser verifier: popup coverage\n");
const popupEvents = await verifyPopupEventCoverage(executable); const popupEvents = await verifyPopupEventCoverage(executable);
const networkEvidence = await verifyBrowserNetworkEvidence( const networkEvidence = await verifyBrowserNetworkEvidence(
executable, executable,
@@ -354,7 +365,11 @@ try {
scripts: observed.value.scripts.items.length, scripts: observed.value.scripts.items.length,
networkRequests: observed.value.network.requests.length, networkRequests: observed.value.network.requests.length,
consoleEvents: observed.value.console.events.length, consoleEvents: observed.value.console.events.length,
websocketEvents: observed.value.network.websocket_events.length, websocketConnections: observed.value.network.websocket_connections.length,
websocketFrames: observed.value.network.websocket_connections.reduce(
(total, connection) => total + connection.events.length,
0,
),
bundleScripts: bundle.capture.scripts_analyzed, bundleScripts: bundle.capture.scripts_analyzed,
sourceMaps: bundle.observations.source_maps.processed, sourceMaps: bundle.observations.source_maps.processed,
bundle_cli_and_stdio_mcp: true, bundle_cli_and_stdio_mcp: true,
@@ -373,6 +388,7 @@ try {
browserScenarioLaunchCleanup: "terminated-owned-process", browserScenarioLaunchCleanup: "terminated-owned-process",
scenarioFailure, scenarioFailure,
scenarioEnvironment, scenarioEnvironment,
scenarioStorage,
verified: true, verified: true,
}; };
} finally { } finally {
+164
View File
@@ -21,6 +21,10 @@ import {
} from "./verify-real-ghidra-assertions.mjs"; } from "./verify-real-ghidra-assertions.mjs";
async function functionCall(client, operation, parameters) { async function functionCall(client, operation, parameters) {
return (await functionObservationCall(client, operation, parameters)).value;
}
async function functionObservationCall(client, operation, parameters) {
const input = parseGhidraFunctionInput(operation, parameters); const input = parseGhidraFunctionInput(operation, parameters);
if (!input.ok) throw input.error; if (!input.ok) throw input.error;
const called = await client.callTool(operation, input.value); const called = await client.callTool(operation, input.value);
@@ -153,6 +157,57 @@ export async function verifyDebugFunctionOperations(
const denseSwitchDossier = await functionCall(client, "analyze_function", { const denseSwitchDossier = await functionCall(client, "analyze_function", {
procedure: denseSwitchProcedure.address, procedure: denseSwitchProcedure.address,
}); });
const terminalProcedure = requireProcedure(
procedures,
"rea_ghidra_inventory_terminal_call",
);
const terminalObservations = await Promise.all(
[
"procedure_info",
"procedure_assembly",
"read_function_instructions",
"procedure_pseudo_code",
"analyze_function",
].map((operation) =>
functionObservationCall(client, operation, {
...(operation === "analyze_function" ? {} : { document: null }),
procedure: terminalProcedure.address,
}),
),
);
const terminalLimitations = terminalObservations.map((result) =>
result.limitations.find(
(limitation) =>
/terminal call at 0x[0-9a-f]+/i.test(limitation) &&
/ to 0x[0-9a-f]+/i.test(limitation) &&
limitation.includes("abort") &&
limitation.includes("hasNoReturn=true"),
),
);
if (
terminalLimitations.some((limitation) => limitation === undefined) ||
terminalLimitations.some(
(limitation) =>
!limitation.includes("external=") ||
!limitation.includes("thunk=") ||
!limitation.includes("fallthrough is excluded") ||
!limitation.includes("not an independent verification"),
)
)
throw new Error(
`Ghidra no-return boundary evidence was missing callee, address, or flag provenance: ${JSON.stringify(terminalObservations.map(({ limitations }) => limitations))}`,
);
const returningObservation = await functionObservationCall(
client,
"procedure_info",
{ document: null, procedure: leaf.address },
);
if (
returningObservation.limitations.some((limitation) =>
limitation.includes("puts"),
)
)
throw new Error("Ghidra marked an ordinary returning call as terminal");
// The debug fixture is compiled -O0, and GCC 16 and later lower dense // The debug fixture is compiled -O0, and GCC 16 and later lower dense
// switches to compare chains at -O0, so this dossier asserts only its // switches to compare chains at -O0, so this dossier asserts only its
// complete shape; the optimized jump-table variant proves dense jump-table // complete shape; the optimized jump-table variant proves dense jump-table
@@ -179,6 +234,8 @@ export async function verifyDebugFunctionOperations(
instruction_window: { instruction_window: {
returned: instructionWindow.instructions.length, returned: instructionWindow.instructions.length,
}, },
terminal_call_limitations: terminalLimitations,
returning_call_limitations: returningObservation.limitations,
cancellation, cancellation,
timeout, timeout,
concurrency: concurrent, concurrency: concurrent,
@@ -617,6 +674,113 @@ export async function verifyRelativeSwitch(client, procedures, entrySize) {
}; };
} }
/**
* Oracle for relative-switch-signed.S: signed halfword byte offsets from a base
* inside the case bodies, a CBZ route for the zero entry, and a B.CS bound.
*/
const SIGNED_RELATIVE_SWITCH_CASES = [
{ value: 0, immediate: 200 },
{ value: 1, immediate: 201 },
{ value: 2, immediate: 202 },
{ value: 3, immediate: 250 },
{ value: 4, immediate: 204 },
];
export async function verifyRelativeSignedSwitch(client, selectorBias = 0) {
const names = await inventoryCall(client, "list_names", {
document: null,
address: null,
});
const procedure = requireProcedure(names, "rea_relative_switch_signed");
const dossier = await functionCall(client, "analyze_function", {
procedure: procedure.address,
});
const table = dossier.native_api?.jump_tables.find((item) =>
item.data_sources.some(
(source) =>
source.entry_size_bytes === 2 &&
source.entry_count === SIGNED_RELATIVE_SWITCH_CASES.length,
),
);
if (table === undefined)
throw new Error(
`Exact signed halfword relative table unavailable: ${JSON.stringify(dossier.native_api)}`,
);
const expectedValues = SIGNED_RELATIVE_SWITCH_CASES.map(
({ value }) => value + selectorBias,
);
if (
table.mappings.some(
(mapping) =>
mapping.case_value !== null &&
!expectedValues.includes(mapping.case_value),
)
)
throw new Error(
`Signed relative switch emitted an unsupported case label: ${JSON.stringify(table)}`,
);
if (table.mappings.length !== SIGNED_RELATIVE_SWITCH_CASES.length)
throw new Error(
`Signed relative switch has unexpected mappings: ${JSON.stringify(table)}`,
);
if (selectorBias !== 0) {
if (
!table.limitations.some((limitation) =>
limitation.includes("table indices are not proven"),
) ||
!table.mappings.some((mapping) => mapping.case_value === null) ||
table.mappings.some((mapping) => mapping.case_value === selectorBias + 3)
)
throw new Error(
`Normalized selector must retain unknown case evidence: ${JSON.stringify(table)}`,
);
}
for (const { value: tableIndex, immediate } of SIGNED_RELATIVE_SWITCH_CASES) {
const value = tableIndex + selectorBias;
// Without a proven relationship between the machine index and the typed
// switch selector, the CBZ-only case must stay unknown rather than being
// assigned the normalized index or Ghidra's unreachable base destination.
if (selectorBias !== 0 && tableIndex === 3) continue;
const mapping = table.mappings.find((item) => item.case_value === value);
if (
mapping === undefined ||
mapping.confidence !== "high" ||
(selectorBias !== 0 &&
!mapping.evidence.some(
(evidence) => evidence.source === "ghidra-clang-case-token",
))
)
throw new Error(
`Signed relative switch case ${value} was not verified: ${JSON.stringify(table)}`,
);
const instruction = await functionCall(
client,
"inspect_native_instruction",
{ address: mapping.target_address },
);
if (
instruction.status !== "decoded" ||
!instruction.operands.some((operand) =>
operand.components.some(
(component) =>
component.kind === "immediate" &&
component.value === `0x${immediate.toString(16)}`,
),
)
)
throw new Error(
`Signed relative switch target does not implement source case ${value}: ${JSON.stringify(instruction)}`,
);
}
return {
entry_size: 2,
signed: true,
selector_bias: selectorBias,
dispatch_address: table.dispatch_address,
mappings: table.mappings.length,
};
}
export async function verifyNativeValueTrace(client, procedures, target) { export async function verifyNativeValueTrace(client, procedures, target) {
const entry = requireProcedure(procedures, "rea_ghidra_inventory_entry"); const entry = requireProcedure(procedures, "rea_ghidra_inventory_entry");
const names = await inventoryCall(client, "list_names", { const names = await inventoryCall(client, "list_names", {
+227
View File
@@ -0,0 +1,227 @@
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { parseBinaryTarget } from "../dist/application/BinaryTargetResolver.js";
import { parseConfig } from "../dist/config/parseConfig.js";
import { GhidraClient } from "../dist/ghidra/GhidraClient.js";
import { inspectGhidraInstallation } from "../dist/ghidra/GhidraInstallation.js";
import { SUPPORTED_GHIDRA_VERSION } from "../dist/ghidra/GhidraInstallationPolicy.js";
import { GhidraProvider } from "../dist/ghidra/GhidraProvider.js";
import { silentLogger } from "../dist/logger.js";
import { cleanupOwnedProcessGroup } from "../dist/process/ProcessOwnership.js";
import {
ProviderProcessSupervisor,
spawnOwnedProviderProcess,
} from "../dist/process/ProviderProcess.js";
import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs";
const exec = promisify(execFile);
const run = createVerifierRun();
const procedureName = (name) =>
process.platform === "darwin" ? `_${name}` : name;
assert.ok(
process.platform === "linux" || process.platform === "darwin",
"This regression lane requires Linux or macOS",
);
assert.ok(
process.arch === "x64" || process.arch === "arm64",
"This regression lane requires x86-64 or arm64",
);
const config = parseConfig(process.env);
if (!config.ok) throw config.error;
const installation = inspectGhidraInstallation({
environment: process.env,
installDir: config.value.ghidraInstallDir,
javaHome: config.value.ghidraJavaHome,
});
assert.equal(installation.status, "available", JSON.stringify(installation));
assert.equal(installation.providerVersion, SUPPORTED_GHIDRA_VERSION);
const fixtures = fileURLToPath(
new URL("../tests/conformance/ghidra", import.meta.url),
);
const bridge = fileURLToPath(new URL("../bridge/ghidra", import.meta.url));
const workspace = await mkdtemp(join(tmpdir(), "rea-ghidra-noreturn-"));
let cleaned = true;
let report;
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
try {
const targetPath = join(
workspace,
process.platform === "darwin" ? "noreturn.dylib" : "noreturn.so",
);
const targetOptions =
process.platform === "darwin"
? [
"-dynamiclib",
"-undefined",
"dynamic_lookup",
"-U_FORTIFY_SOURCE",
"-D_FORTIFY_SOURCE=0",
]
: ["-shared"];
await exec(process.env.REA_CC ?? "cc", [
...targetOptions,
"-fPIC",
"-O0",
"-fno-builtin",
join(fixtures, "no-return.c"),
"-o",
targetPath,
]);
const targetDigest = digest(await readFile(targetPath));
const sourceDigests = Object.fromEntries(
await Promise.all(
["ReaGhidraBridge.java", "ReaGhidraNoReturnFix.java"].map(
async (name) => [name, digest(await readFile(join(bridge, name)))],
),
),
);
const project = join(workspace, "project");
await mkdir(project);
const launch = await spawnOwnedProviderProcess({
command: installation.analyzeHeadlessPath,
arguments: [
project,
"NoReturn",
"-import",
targetPath,
"-readOnly",
"-deleteProject",
"-scriptPath",
`${fixtures};${bridge}`,
"-postScript",
join(fixtures, "ReaNoReturnProbe.java"),
bridge,
],
runId: run.run_id,
expectedCommand: null,
});
const supervisor = new ProviderProcessSupervisor({
...launch,
ownsProcessLifetime: true,
cleanup: () => cleanupOwnedProcessGroup(launch.ownership),
});
let probe;
try {
assert.ok(
await supervisor.waitForExit(120000),
"No-return probe timed out",
);
assert.ok(
await supervisor.waitForOutputClose(1000),
"No-return probe output did not close",
);
const snapshot = supervisor.snapshot();
const output = `${snapshot.stdout.text}\n${snapshot.stderr.text}`;
assert.equal(snapshot.exitCode, 0, output);
const reports = [
...output.matchAll(/REA_NO_RETURN_PROBE_JSON (\{[^\r\n]*\})/gu),
];
assert.equal(reports.length, 1, output);
probe = JSON.parse(reports[0][1]);
assert.equal(probe.status, "passed");
assert.equal(probe.ghidra_version, SUPPORTED_GHIDRA_VERSION);
assert.equal(probe.target_sha256, targetDigest);
for (const [name, expected] of Object.entries(sourceDigests))
assert.equal(probe[name], expected);
assert.ok(probe.checks.includes("independent_entry_preserved"));
assert.ok(probe.checks.includes("propagated_flag_repaired_rea_tls_outer"));
assert.ok(probe.checks.includes("recovered_pseudocode"));
} finally {
const stopped = await supervisor.stop();
cleaned = stopped.status === "verified-cleanup";
assert.ok(
cleaned,
`Owned cleanup incomplete; retained ${workspace}: ${JSON.stringify(stopped)}`,
);
}
// Exercise the actual Java server, authenticated transport, request queue and Evidence
// projection. This read-only TCP lane also runs where AF_UNIX is unavailable; it does
// not claim Windows authority or validation of the default Linux Unix-socket transport.
const provider = new GhidraProvider(
config.value,
silentLogger,
process.env,
undefined,
(options) =>
new GhidraClient({ ...options, transport: "authenticated-loopback-tcp" }),
);
const target = await parseBinaryTarget(targetPath);
if (!target.ok) throw target.error;
const profile = await provider.resolveAnalysisProfile(target.value);
if (!profile.ok) throw profile.error;
assert.ok(profile.value.profile);
assert.equal(
profile.value.profile.parameters.no_return_repair,
"returning-imports-decoded-return-v2",
);
const client = provider.createClient(target.value, profile.value.profile);
cleaned = false;
const observations = [];
const failures = [];
try {
for (const operation of [
"procedure_pseudo_code",
"procedure_assembly",
"procedure_info",
"read_function_instructions",
"analyze_function",
]) {
const result = await client.execute(operation, {
procedure: procedureName("rea_abort"),
});
if (!result.ok) throw result.error;
const warnings = result.value.limitations.filter((value) =>
/terminal call/u.test(value),
);
assert.ok(
warnings.some(
(value) => /abort/u.test(value) && /fallthrough/u.test(value),
),
operation,
);
observations.push({ operation, warnings });
}
const healthy = await client.execute("procedure_pseudo_code", {
procedure: procedureName("rea_tls"),
});
if (!healthy.ok) throw healthy.error;
assert.ok(
!healthy.value.limitations.some((value) => /terminal call/u.test(value)),
);
} catch (cause) {
failures.push(cause);
}
const closed = await client.close();
cleaned = closed.ok;
if (!closed.ok) failures.push(closed.error);
if (failures.length > 0)
throw new AggregateError(
failures,
"No-return Evidence verification failed",
);
assert.equal(digest(await readFile(targetPath)), targetDigest);
report = {
status: "passed",
probe,
observations,
scope:
"Real Ghidra ELF or Mach-O analysis with injected flags; real read-only TCP Evidence projection. No target execution; exact Amethyst artifact and Windows are unverified.",
};
} finally {
if (cleaned) await rm(workspace, { recursive: true, force: true });
}
console.log(
JSON.stringify({
...report,
cleanup: "complete",
verifier_run: await completeVerifierRun(run),
}),
);
+1 -1
View File
@@ -322,7 +322,7 @@ async function functionOperation(operation, parameters) {
const result = parseGhidraFunctionResult(operation, called.value); const result = parseGhidraFunctionResult(operation, called.value);
if (!result.ok) throw result.error; if (!result.ok) throw result.error;
observed.add(operation); observed.add(operation);
return result.value; return result.value.value;
} }
function assertSession(session, targetSha256, profileDigest) { function assertSession(session, targetSha256, profileDigest) {
+53 -17
View File
@@ -38,6 +38,7 @@ import {
verifyJumpTableDenseSwitch, verifyJumpTableDenseSwitch,
verifyNativeTypeLayout, verifyNativeTypeLayout,
verifyRelativeSwitch, verifyRelativeSwitch,
verifyRelativeSignedSwitch,
verifyNativeValueTrace, verifyNativeValueTrace,
} from "./verify-real-ghidra-function.mjs"; } from "./verify-real-ghidra-function.mjs";
const exec = promisify(execFile); const exec = promisify(execFile);
@@ -173,6 +174,36 @@ try {
]); ]);
} }
} }
const signedRelativeSource = fileURLToPath(
new URL(
"../tests/conformance/ghidra/relative-switch-signed.S",
import.meta.url,
),
);
for (const selectorBias of [0, 2, 10]) {
for (const format of [
"elf",
...(expectedNativeTarget.architecture === "arm64" ? ["mach-o"] : []),
]) {
const targetPath = join(
fixtureRoot,
`relative-signed-${selectorBias}-${format}.o`,
);
await exec(clang, [
...(format === "elf" ? ["--target=aarch64-linux-gnu"] : []),
`-DREA_SELECTOR_BIAS=${selectorBias}`,
"-c",
signedRelativeSource,
"-o",
targetPath,
]);
crossTargets.push([
targetPath,
`relative-signed-${selectorBias}`,
{ format, architecture: "arm64" },
]);
}
}
await exec(clang, [ await exec(clang, [
"--target=aarch64-linux-gnu", "--target=aarch64-linux-gnu",
@@ -522,25 +553,30 @@ async function verifyTarget(targetPath, variant, expectedTarget = null) {
client, client,
}) })
: null; : null;
const probes = variant.startsWith("relative-") const probes = variant.startsWith("relative-signed-")
? await verifyRelativeSwitch( ? await verifyRelativeSignedSwitch(
client, client,
procedures, Number(variant.split("-").at(-1)),
Number(variant.slice(-1)),
) )
: variant === "type-layout" : variant.startsWith("relative-")
? await verifyNativeTypeLayout(client, names) ? await verifyRelativeSwitch(
: variant === "jump-table" client,
? await verifyJumpTableDenseSwitch(client, procedures) procedures,
: variant === "custom" || variant === "aarch64-jump-table" Number(variant.slice(-1)),
? null )
: await verifyInventoryOperations({ : variant === "type-layout"
client, ? await verifyNativeTypeLayout(client, names)
variant, : variant === "jump-table"
procedures, ? await verifyJumpTableDenseSwitch(client, procedures)
names, : variant === "custom" || variant === "aarch64-jump-table"
strings, ? null
}); : await verifyInventoryOperations({
client,
variant,
procedures,
names,
strings,
});
const nativeValues = const nativeValues =
variant === "debug" variant === "debug"
? await verifyNativeValueTrace(client, procedures, parsedTarget.value) ? await verifyNativeValueTrace(client, procedures, parsedTarget.value)
+59 -6
View File
@@ -26,22 +26,49 @@ export async function verifyElectronToolContracts({
await mkdtemp(join(tmpdir(), "rea-electron-e2e-")), await mkdtemp(join(tmpdir(), "rea-electron-e2e-")),
); );
const main = join(root, "main.cjs"); const main = join(root, "main.cjs");
const preload = join(root, "preload.cjs");
const ready = join(root, "ready"); const ready = join(root, "ready");
const entrypoint = join(repositoryRoot, "scripts", "rea.mjs"); const entrypoint = join(repositoryRoot, "scripts", "rea.mjs");
const runId = randomUUID(); const runId = randomUUID();
await writeFile( await writeFile(
main, main,
`const {app,BrowserWindow}=require("electron"); `const electron = require("electron");
app.whenReady().then(()=>{const window=new BrowserWindow(); const path = require("node:path");
window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringify(ready)},"ready");});`, const { app, BrowserWindow, ipcMain } = electron;
const Window = BrowserWindow;
const mainIpc = ipcMain;
app.whenReady().then(() => {
const window = new Window({
webPreferences: { preload: path.join(__dirname, "preload.cjs") },
});
window.loadFile("renderer.html");
require("node:fs").writeFileSync(${JSON.stringify(ready)}, "ready");
});
mainIpc.handle("fixture:echo", (_event, value) => ({ value }));`,
); );
await writeFile( await writeFile(
join(root, "renderer.html"), join(root, "renderer.html"),
'<!doctype html><button id="run">Run</button><script src="renderer.js"></script>', `<!doctype html>
<button id="run">Run</button>
<output id="result"></output>
<script src="renderer.js"></script>`,
);
await writeFile(
preload,
`const electron = require("electron");
const { contextBridge, ipcRenderer } = electron;
const bridge = contextBridge;
const rendererIpc = ipcRenderer;
bridge.exposeInMainWorld("fixture", {
echo: (value) => rendererIpc.invoke("fixture:echo", value),
});`,
); );
await writeFile( await writeFile(
join(root, "renderer.js"), join(root, "renderer.js"),
'document.querySelector("#run").addEventListener("click",()=>console.log("fixture-click"));', `document.querySelector("#run").addEventListener("click", async () => {
const result = await window.fixture.echo("renderer");
document.querySelector("#result").textContent = result.value;
});`,
); );
const transport = new StdioClientTransport({ const transport = new StdioClientTransport({
command: process.execPath, command: process.execPath,
@@ -135,6 +162,25 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
const staticAnalysis = await call("analyze_javascript_application", { const staticAnalysis = await call("analyze_javascript_application", {
input_path: root, input_path: root,
}); });
assert.equal(staticAnalysis.normalized_result.summary.browser_windows, 1);
assert.equal(
staticAnalysis.normalized_result.summary.preload_entrypoints,
1,
);
assert.equal(
staticAnalysis.normalized_result.summary.context_bridge_apis,
1,
);
assert.equal(staticAnalysis.normalized_result.summary.ipc.main_handlers, 1);
assert.equal(
staticAnalysis.normalized_result.summary.ipc.renderer_transmissions,
1,
);
assert.equal(
staticAnalysis.normalized_result.summary.ipc
.paired_renderer_transmissions,
1,
);
const reconciled = await call("reconcile_javascript_runtime", { const reconciled = await call("reconcile_javascript_runtime", {
static_layers: [ static_layers: [
{ {
@@ -169,7 +215,7 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
executable_path: executable, executable_path: executable,
application_path: main, application_path: main,
application_root: root, application_root: root,
actions: [{ step_id: "settle", kind: "wait", duration_ms: 50 }], actions: [{ step_id: "ipc", kind: "click", selector: "#run" }],
}; };
const active = await client.callTool( const active = await client.callTool(
{ name: "capture_electron_scenario", arguments: input }, { name: "capture_electron_scenario", arguments: input },
@@ -192,6 +238,13 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
assert.equal(capture.application.cleanup, "terminated-owned-process"); assert.equal(capture.application.cleanup, "terminated-owned-process");
} }
assert.equal(capture.actions[0].status, "completed"); assert.equal(capture.actions[0].status, "completed");
assert.ok(
capture.ipc.events.some(
({ kind, channel }) =>
kind === "main-handler-invocation" && channel === "fixture:echo",
),
JSON.stringify(capture.ipc),
);
assert.equal( assert.equal(
capture.coverage.status, capture.coverage.status,
"partial_attach", "partial_attach",
+97
View File
@@ -0,0 +1,97 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { isAbsolute, join } from "node:path";
import { fileURLToPath } from "node:url";
import {
spawnOwnedProviderProcess,
ProviderProcessSupervisor,
} from "../../../dist/process/ProviderProcess.js";
import { cleanupOwnedProcessGroup } from "../../../dist/process/ProcessOwnership.js";
import { createVerifierRun } from "../../lib/verifier-run.mjs";
const installation = process.env.GHIDRA_INSTALL_DIR;
assert.ok(
installation && isAbsolute(installation),
"Set GHIDRA_INSTALL_DIR to a local Ghidra installation",
);
assert.notEqual(
process.platform,
"win32",
"This direct headless regression requires a POSIX host; Windows uses its controlled provider lane",
);
const workspace = await mkdtemp(join(tmpdir(), "rea-ghidra-seed-regression-"));
const run = createVerifierRun();
let supervisor;
try {
const target = join(workspace, "seed-functions.bin");
const seeds = join(workspace, "seeds.tsv");
// Falling through from the lower entry reaches the higher entry. Creating
// the higher function first is what keeps the two function bodies separate.
await writeFile(target, Buffer.from([0x90, 0x90, 0xc3]));
await writeFile(
seeds,
[
"0x0\tfunction\tseed_low",
"0x100000000\tfunction\tout_of_range",
"0x1\tfunction\tseed_high",
"0x1000\tlabel\tseed_global",
"0x1000\tcode",
"",
].join("\n"),
);
const launch = await spawnOwnedProviderProcess({
command: join(installation, "support", "analyzeHeadless"),
arguments: [
workspace,
"seed-regression",
"-import",
target,
"-loader",
"BinaryLoader",
"-processor",
"x86:LE:32:default",
"-noanalysis",
"-readOnly",
"-deleteProject",
"-scriptPath",
[
fileURLToPath(new URL("../../../bridge/ghidra", import.meta.url)),
fileURLToPath(
new URL("../../../tests/conformance/ghidra", import.meta.url),
),
].join(";"),
"-postScript",
"ReaSeedRegressionProbe.java",
seeds,
],
runId: run.run_id,
expectedCommand: null,
});
supervisor = new ProviderProcessSupervisor({
...launch,
ownsProcessLifetime: true,
cleanup: () => cleanupOwnedProcessGroup(launch.ownership),
});
assert.ok(
await supervisor.waitForExit(180_000),
"Ghidra seed regression exceeded its startup deadline",
);
const snapshot = supervisor.snapshot();
const output = `${snapshot.stdout.text}\n${snapshot.stderr.text}`;
assert.equal(snapshot.exitCode, 0, output);
// Headless can exit zero after a script fails, so require the probe's marker.
assert.ok(output.includes("REA_SEED_REGRESSION_OK "), output);
} finally {
if (supervisor !== undefined) {
const stopped = await supervisor.stop();
assert.notEqual(stopped.status, "incomplete", JSON.stringify(stopped));
supervisor.dispose();
}
await rm(workspace, { recursive: true, force: true });
}
console.log(
"Real Ghidra seed regression passed: descending function order, out-of-range seed, BSS label, BSS code rejection, and owned-process cleanup.",
);
+57 -11
View File
@@ -56,7 +56,7 @@ try {
process.env.REA_JAVASCRIPT_FIXTURE_TOOLS, process.env.REA_JAVASCRIPT_FIXTURE_TOOLS,
); );
await client.connect(transport); await client.connect(transport);
for (const fixture of fixtures.files) { for (const fixture of [...fixtures.files, fixtures.codeSplit]) {
for (const surface of ["cli", "stdio_mcp"]) { for (const surface of ["cli", "stdio_mcp"]) {
const output = join(root, `${fixture.name}-${surface}`); const output = join(root, `${fixture.name}-${surface}`);
const flags = const flags =
@@ -75,11 +75,18 @@ try {
fixture.path, fixture.path,
output, output,
...flags, ...flags,
...(fixture.additional_paths ?? []).flatMap((path) => [
"--additional-path",
path,
]),
) )
: await call("recover_javascript_sources", { : await call("recover_javascript_sources", {
path: fixture.path, path: fixture.path,
output_directory: output, output_directory: output,
...fixture.options, ...fixture.options,
...(fixture.additional_paths
? { additional_paths: fixture.additional_paths }
: {}),
}); });
const result = javascriptRecoveryResultSchema.parse( const result = javascriptRecoveryResultSchema.parse(
evidence.normalized_result, evidence.normalized_result,
@@ -97,6 +104,25 @@ try {
analysis.normalized_result.statistics.parsed_javascript_files, analysis.normalized_result.statistics.parsed_javascript_files,
result.modules.length, result.modules.length,
); );
const imports = analysis.normalized_result.graph.edges.filter(
(edge) =>
edge.relation === "imports" &&
edge.properties.resolution_status === "resolved",
);
if (fixture.additional_paths) {
assert.ok(
imports.length >= 4,
"Expected cross-chunk imports in the recovered application graph",
);
const recoveredPaths = new Set(
result.modules.map((module) => module.reported_filename),
);
for (const edge of imports)
assert.ok(
recoveredPaths.has(edge.properties.resolved_path),
`Expected an emitted import target: ${String(edge.properties.resolved_path)}`,
);
}
const recoveredEntry = const recoveredEntry =
result.modules.find( result.modules.find(
(module) => module.reported_filename === "entry.js", (module) => module.reported_filename === "entry.js",
@@ -104,7 +130,8 @@ try {
assert.ok(recoveredEntry); assert.ok(recoveredEntry);
const oracle = const oracle =
'await import((await import("node:url")).pathToFileURL(process.argv[1])); const f=globalThis.recoveryFixture; console.log(JSON.stringify([f.greet(null),f.greet("REA"),f.total([1,2,3])]));'; 'await import((await import("node:url")).pathToFileURL(process.argv[1])); const f=globalThis.recoveryFixture; console.log(JSON.stringify([f.greet(null),f.greet("REA"),f.total([1,2,3])]));';
for (const selected of fixture.options?.extraction_mode === "inspection" for (const selected of fixture.additional_paths ||
fixture.options?.extraction_mode === "inspection"
? [] ? []
: [fixture.path, recoveredEntry.artifact.path]) { : [fixture.path, recoveredEntry.artifact.path]) {
const execution = await run( const execution = await run(
@@ -125,7 +152,9 @@ try {
maps: result.modules.filter((module) => module.source_map !== null) maps: result.modules.filter((module) => module.source_map !== null)
.length, .length,
finite_fixture_oracle: finite_fixture_oracle:
!fixture.additional_paths &&
fixture.options?.extraction_mode !== "inspection", fixture.options?.extraction_mode !== "inspection",
cross_chunk_imports: fixture.additional_paths ? imports.length : null,
}); });
} }
} }
@@ -157,9 +186,15 @@ async function assertRecoveredArtifacts(evidence, result, fixture) {
assert.equal(result.status, "complete"); assert.equal(result.status, "complete");
assert.equal(result.reported_format, fixture.expectedFormat); assert.equal(result.reported_format, fixture.expectedFormat);
assert.equal(result.modules.length, fixture.expectedModules); assert.equal(result.modules.length, fixture.expectedModules);
assert.equal(result.source.sha256, fingerprint(await readFile(fixture.path))); const sources = [result.source, ...(result.additional_sources ?? [])];
assert.deepEqual(
sources.map((source) => source.path),
[fixture.path, ...(fixture.additional_paths ?? [])],
);
for (const source of sources)
assert.equal(source.sha256, fingerprint(await readFile(source.path)));
for (const file of [ for (const file of [
result.source.published_copy, ...sources.map((source) => source.published_copy),
result.report, result.report,
result.provenance, result.provenance,
result.manifest, result.manifest,
@@ -173,16 +208,27 @@ async function assertRecoveredArtifacts(evidence, result, fixture) {
assert.equal(fingerprint(bytes), file.sha256); assert.equal(fingerprint(bytes), file.sha256);
} }
for (const module of result.modules) { for (const module of result.modules) {
assert.equal(module.provenance.original_sha256, result.source.sha256); const source = sources.find(
assert.equal(module.provenance.reported_input, result.source.snapshot_path); (source) => source.snapshot_path === module.provenance.reported_input,
for (const range of module.provenance.byte_ranges) );
assert.ok(range.start <= range.end && range.end <= result.source.bytes); assert.ok(source, "Module must name a selected input snapshot");
assert.equal(module.provenance.original_path, source.path);
assert.equal(module.provenance.original_sha256, source.sha256);
for (const range of [
...module.provenance.byte_ranges,
...module.provenance.context_byte_ranges,
])
assert.ok(range.start <= range.end && range.end <= source.bytes);
if (module.source_map) { if (module.source_map) {
const map = JSON.parse(await readFile(module.source_map.path, "utf8")); const map = JSON.parse(await readFile(module.source_map.path, "utf8"));
for (const source of map.sources) for (const source of map.sources)
assert.equal( assert.ok(
resolve(dirname(module.source_map.path), source), sources.some(
result.source.published_copy.path, (selected) =>
selected.published_copy.path ===
resolve(dirname(module.source_map.path), source),
),
"Map source must name a published input snapshot",
); );
} }
} }
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createServer } from "node:http";
import { promisify } from "node:util";
const execute = promisify(execFile);
assert.equal(process.platform, "android", "Run this verifier inside Termux");
assert.equal(process.env.PLAYWRIGHT_BROWSERS_PATH, undefined);
// Import no browser provider here: the public dispatcher must establish its
// Android cache fallback before Playwright initializes in each subprocess.
const { stdout: doctorOutput } = await execute(
process.execPath,
["scripts/rea.mjs", "mcp", "doctor", "--format", "json"],
{ timeout: 120_000, maxBuffer: 8 * 1024 * 1024 },
);
assert.equal(JSON.parse(doctorOutput).healthy, true, doctorOutput);
const executable = `${process.env.PREFIX}/bin/chromium`;
const { stdout: chromiumVersion } = await execute(executable, ["--version"], {
timeout: 30_000,
});
const marker = "rea-termux-browser-smoke";
const server = createServer((_request, response) => {
response.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
response.end(`<!doctype html><html><body><h1>${marker}</h1></body></html>`);
});
await new Promise((resolve, reject) => {
server.once("error", reject);
server.listen(0, "127.0.0.1", resolve);
});
try {
const url = `http://127.0.0.1:${server.address().port}/smoke`;
const scenario = {
browser: { mode: "launch", executable_path: executable },
start_url: { url },
actions: [
{
step_id: "ready",
action: "wait_for",
locator: { kind: "css", selector: "h1" },
state: "visible",
},
],
capture: { at_end: ["url", "dom"] },
};
const { stdout } = await execute(
process.execPath,
[
"scripts/rea.mjs",
"capture-browser-scenario",
JSON.stringify(scenario),
"--json",
],
{ timeout: 180_000, maxBuffer: 16 * 1024 * 1024 },
);
const evidence = JSON.parse(stdout);
assert.equal(evidence.error, undefined, stdout);
const capture = evidence.normalized_result;
assert.ok(
capture.steps.every((step) => step.status === "completed"),
stdout,
);
const artifacts = capture.steps.at(-1).artifacts;
assert.equal(artifacts.url.state, "captured", stdout);
assert.equal(artifacts.url.value.url, url);
assert.equal(artifacts.dom.state, "captured", stdout);
assert.ok(artifacts.dom.value.text.includes(`<h1>${marker}</h1>`), stdout);
assert.equal(capture.browser.cleanup, "terminated-owned-process");
console.log(
JSON.stringify({
platform: process.platform,
architecture: process.arch,
node: process.version,
chromium: chromiumVersion.trim(),
mcp_doctor: "healthy",
url_capture: "passed",
dom_capture: "passed",
cleanup: capture.browser.cleanup,
}),
);
} finally {
await new Promise((resolve, reject) =>
server.close((cause) => (cause ? reject(cause) : resolve())),
);
}
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env bash
# Host-side orchestration. Root is used only to stage/read files and send the
# intent; TermuxService launches all package/build/browser work as the app UID.
set -euo pipefail
staging="${RUNNER_TEMP:?}/rea-termux"
app_home=/data/data/com.termux/files/home
prefix=/data/data/com.termux/files/usr
mkdir -p "$staging/diagnostics"
collect_diagnostics() {
adb logcat -d > "$staging/diagnostics/logcat.txt" 2>&1 || true
adb exec-out cat "$app_home/rea-ci/run.log" > "$staging/diagnostics/termux.log" 2>&1 || true
adb exec-out cat "$app_home/rea-ci/status" > "$staging/diagnostics/status.txt" 2>&1 || true
}
trap collect_diagnostics EXIT
adb root
adb wait-for-device
adb install "$staging/termux-app_v0.118.3+github-debug_x86_64.apk"
adb shell am start -n com.termux/.app.TermuxActivity
# First launch unpacks the APK's bundled bootstrap. Do not replace it with a
# root-installed userland: that would miss Termux's own initialization.
deadline=$((SECONDS + 180))
until adb shell test -x "$prefix/bin/bash"; do
if (( SECONDS >= deadline )); then
echo 'Termux bootstrap did not finish within 180 seconds' >&2
exit 1
fi
sleep 2
done
git archive --format=tar HEAD > "$staging/source.tar"
adb push "$staging/source.tar" /data/local/tmp/rea-source.tar
adb push scripts/verify/termux/in-app.sh /data/local/tmp/rea-in-app.sh
adb shell chmod 644 /data/local/tmp/rea-source.tar /data/local/tmp/rea-in-app.sh
app_uid=$(adb shell stat -c %u /data/data/com.termux | tr -d '\r')
[[ "$app_uid" =~ ^[0-9]+$ && "$app_uid" != 0 ]]
adb shell mkdir -p "$app_home/rea-ci" "$app_home/.termux"
adb shell "echo allow-external-apps=true > $app_home/.termux/termux.properties"
adb shell cp /data/local/tmp/rea-in-app.sh "$app_home/rea-ci/in-app.sh"
adb shell cp /data/local/tmp/rea-source.tar "$app_home/rea-ci/source.tar"
adb shell chown -R "$app_uid:$app_uid" "$app_home/rea-ci" "$app_home/.termux"
adb shell restorecon -RF "$app_home/rea-ci" "$app_home/.termux"
# Reload the external-command policy in a fresh app process before sending it.
adb shell am force-stop com.termux
adb shell am start -n com.termux/.app.TermuxActivity
# Root may send this permission-protected intent; the service executes as the
# application, rather than running Node/Chromium through adb shell or su.
adb shell am startservice \
-n com.termux/.app.RunCommandService \
-a com.termux.RUN_COMMAND \
--es com.termux.RUN_COMMAND_PATH "$prefix/bin/bash" \
--esa com.termux.RUN_COMMAND_ARGUMENTS "-l,$app_home/rea-ci/in-app.sh" \
--es com.termux.RUN_COMMAND_WORKDIR "$app_home" \
--ez com.termux.RUN_COMMAND_BACKGROUND true
deadline=$((SECONDS + 2100))
until adb shell test -f "$app_home/rea-ci/status"; do
if (( SECONDS >= deadline )); then
echo 'Termux verification exceeded 35 minutes' >&2
exit 1
fi
sleep 5
done
adb exec-out cat "$app_home/rea-ci/run.log"
status=$(adb shell cat "$app_home/rea-ci/status" | tr -d '\r')
[[ "$status" == 0 ]]
+33
View File
@@ -0,0 +1,33 @@
#!/data/data/com.termux/files/usr/bin/bash
set -euo pipefail
report="$HOME/rea-ci"
exec > "$report/run.log" 2>&1
# Publish the result only after all commands have settled. The host reads it as
# root; no shared-storage permission or helper Android application is needed.
trap 'result=$?; printf "%s\n" "$result" > "$report/status"' EXIT
set -x
[[ "$(id -u)" != 0 ]]
export DEBIAN_FRONTEND=noninteractive
export HUSKY=0
pkg update -y
pkg upgrade -y
pkg install -y nodejs-lts npm git x11-repo termux-exec
pkg install -y chromium
mkdir -p "$report/source"
tar -xf "$report/source.tar" -C "$report/source"
cd "$report/source"
node -e 'if (process.platform !== "android") throw new Error("Expected Android Node.js")'
node -e 'const expected = require("node:fs").readFileSync(".nvmrc", "utf8").trim().replace(/^v/, ""); if (process.versions.node !== expected) throw new Error(`Termux Node ${process.versions.node} differs from .nvmrc ${expected}`)'
# Android uses Termux's Bionic-linked Node distribution; the desktop .nvmrc
# binary cannot run here. Keep npm at the repository's exact pinned version.
npm install --prefix "$report/toolchain" --no-package-lock --ignore-scripts \
"$(node -p 'JSON.parse(require("node:fs").readFileSync("package.json", "utf8")).packageManager')"
export PATH="$report/toolchain/node_modules/.bin:$PATH"
node --version
npm --version
dpkg-query -W nodejs-lts chromium termux-exec
npm ci
npm run build:termux
unset PLAYWRIGHT_BROWSERS_PATH npm_config_playwright_browsers_path npm_package_config_playwright_browsers_path
node scripts/verify/termux/browser.mjs
+1 -1
View File
@@ -22,7 +22,7 @@ describe("doctor", () => {
), ),
).toEqual({ ).toEqual({
node: `Install ${SUPPORTED_NODE_VERSION_PROSE}.`, node: `Install ${SUPPORTED_NODE_VERSION_PROSE}.`,
host: "REA supports macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, and the experimental Windows x64 Ghidra P0 boundary.", host: "REA supports macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, Omarchy, and the experimental Windows x64 Ghidra P0 boundary.",
hopper: "Run rea setup to install Hopper, or set HOPPER_LAUNCHER_PATH.", hopper: "Run rea setup to install Hopper, or set HOPPER_LAUNCHER_PATH.",
target: "Supply a readable local app or program path.", target: "Supply a readable local app or program path.",
}); });
+1 -1
View File
@@ -113,7 +113,7 @@ const hostCheck = async (host: DoctorHost): Promise<DoctorCheck> => {
: `${linuxDistribution.id} ${linuxDistribution.versionId ?? "unknown"}`); : `${linuxDistribution.id} ${linuxDistribution.versionId ?? "unknown"}`);
return check("host", supported, detail, { return check("host", supported, detail, {
remediation: remediation:
"REA supports macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, and the experimental Windows x64 Ghidra P0 boundary.", "REA supports macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, Omarchy, and the experimental Windows x64 Ghidra P0 boundary.",
classification: "unsupported_host", classification: "unsupported_host",
}); });
}; };
+12 -6
View File
@@ -60,12 +60,16 @@ describe("Linux Hopper host classification", () => {
['ID=nobara\nVERSION_ID="44"\nID_LIKE="rhel centos fedora"\n', "rpm"], ['ID=nobara\nVERSION_ID="44"\nID_LIKE="rhel centos fedora"\n', "rpm"],
["ID=arch\n", "arch"], ["ID=arch\n", "arch"],
["ID=cachyos\nID_LIKE=arch\n", "arch"], ["ID=cachyos\nID_LIKE=arch\n", "arch"],
] as const)("accepts an official Hopper distribution", (document, family) => { ['ID=omarchy\nID_LIKE=arch\nVERSION_ID="4.0.4"\n', "arch"],
expect(parseLinuxDistribution(document)).toMatchObject({ ] as const)(
packageFamily: family, "accepts an explicitly supported distribution",
supported: true, (document, family) => {
}); expect(parseLinuxDistribution(document)).toMatchObject({
}); packageFamily: family,
supported: true,
});
},
);
it("detects unresolved or loader-reported shared libraries", () => { it("detects unresolved or loader-reported shared libraries", () => {
expect( expect(
@@ -93,6 +97,8 @@ describe("Linux Hopper host classification", () => {
'ID=debian\nVERSION_ID="13"\nID_LIKE=debian\n', 'ID=debian\nVERSION_ID="13"\nID_LIKE=debian\n',
"ID=manjaro\nID_LIKE=arch\n", "ID=manjaro\nID_LIKE=arch\n",
"ID=garuda\nID_LIKE=arch\n", "ID=garuda\nID_LIKE=arch\n",
"ID=omarchy\n",
"ID=omarchy\nID_LIKE=debian\n",
])("rejects unsupported vendor/version combinations", (document) => { ])("rejects unsupported vendor/version combinations", (document) => {
expect(parseLinuxDistribution(document).supported).toBe(false); expect(parseLinuxDistribution(document).supported).toBe(false);
}); });
+1
View File
@@ -154,6 +154,7 @@ export const parseLinuxDistribution = (text: string): LinuxDistribution => {
packageFamily !== undefined && packageFamily !== undefined &&
(id === "arch" || (id === "arch" ||
id === "cachyos" || id === "cachyos" ||
(id === "omarchy" && packageFamily === "arch") ||
(id === "ubuntu" && versionAtLeast(versionId, 24)) || (id === "ubuntu" && versionAtLeast(versionId, 24)) ||
(id === "fedora" && versionAtLeast(versionId, 41)) || (id === "fedora" && versionAtLeast(versionId, 41)) ||
(id === "nobara" && versionAtLeast(versionId, 44))); (id === "nobara" && versionAtLeast(versionId, 44)));
+1 -1
View File
@@ -267,7 +267,7 @@ describe("setup scoped readiness", () => {
expect(result.status).toBe("needs_human"); expect(result.status).toBe("needs_human");
expect(host.hopperInstalls).toBe(0); expect(host.hopperInstalls).toBe(0);
expect(result.remediation).toBe( expect(result.remediation).toBe(
"Automated Hopper setup supports Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, and CachyOS; configure an existing supported provider instead.", "Automated Hopper setup supports Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, and Omarchy; configure an existing supported provider instead.",
); );
}); });
+1 -1
View File
@@ -90,7 +90,7 @@ export const hostRemediation = async (
: undefined; : undefined;
} }
if ((await host.linuxDistribution())?.supported === true) return undefined; if ((await host.linuxDistribution())?.supported === true) return undefined;
return "Automated Hopper setup supports Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, and CachyOS; configure an existing supported provider instead."; return "Automated Hopper setup supports Ubuntu 24.04+, Fedora 41+, Nobara 44+, 64-bit Arch Linux, CachyOS, and Omarchy; configure an existing supported provider instead.";
}; };
/** Production setup effects for Hopper, agent configuration, and the canonical skill directory. */ /** Production setup effects for Hopper, agent configuration, and the canonical skill directory. */
@@ -6,14 +6,9 @@ import {
analyzeJavaScriptApplicationInputSchema, analyzeJavaScriptApplicationInputSchema,
parseOwnedJavaScriptApplicationAnalysisSteps, parseOwnedJavaScriptApplicationAnalysisSteps,
} from "../../domain/javascript/javascriptApplicationAnalysis.js"; } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { import { AnalysisOutputError } from "../../domain/analysisErrorCore.js";
AnalysisAccessDeniedError,
AnalysisInputError,
AnalysisOutputError,
AnalysisUnsupportedTargetError,
} from "../../domain/analysisErrorCore.js";
import { ArtifactOperationError } from "../../domain/artifactOperationError.js"; import { ArtifactOperationError } from "../../domain/artifactOperationError.js";
import { type AnalysisError } from "../../domain/analysisErrorBase.js"; import { AnalysisError } from "../../domain/analysisErrorBase.js";
import type { Evidence } from "../../domain/evidence.js"; import type { Evidence } from "../../domain/evidence.js";
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js"; import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
import { err, ok, type Result } from "../../domain/result.js"; import { err, ok, type Result } from "../../domain/result.js";
@@ -64,6 +59,12 @@ export const analyzeJavaScriptApplicationValidated = async (
input_path: input.input_path, input_path: input.input_path,
format: input.format, format: input.format,
integrity_policy: input.integrity_policy, integrity_policy: input.integrity_policy,
...(input.max_heap_mb === undefined
? {}
: { max_heap_mb: input.max_heap_mb }),
...(input.analysis_timeout_ms === undefined
? {}
: { analysis_timeout_ms: input.analysis_timeout_ms }),
}, },
scope, scope,
options.signal, options.signal,
@@ -110,13 +111,8 @@ export const analyzeJavaScriptApplicationValidated = async (
assertJavaScriptAnalysisActive(options.signal); assertJavaScriptAnalysisActive(options.signal);
return ok(evidence); return ok(evidence);
} catch (cause: unknown) { } catch (cause: unknown) {
// Input selection failures are already typed by their cause. // Preserve execution constraints and their completed partial Evidence.
if ( if (cause instanceof AnalysisError) return err(cause);
cause instanceof AnalysisInputError ||
cause instanceof AnalysisUnsupportedTargetError ||
cause instanceof AnalysisAccessDeniedError
)
return err(cause);
if (cause instanceof ArtifactReaderFailure) if (cause instanceof ArtifactReaderFailure)
return err(artifactFailureToOperationError(cause)); return err(artifactFailureToOperationError(cause));
if (cause instanceof z.ZodError) if (cause instanceof z.ZodError)
@@ -8,6 +8,7 @@ import {
classifyParsedJavaScriptOpenReceiversSteps, classifyParsedJavaScriptOpenReceiversSteps,
} from "../../domain/javascript/javascriptSemanticAnalysis.js"; } from "../../domain/javascript/javascriptSemanticAnalysis.js";
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js"; import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
import { projectJavaScriptModuleSemantics } from "../../domain/javascript/javascriptModuleSemanticIr.js";
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js"; import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js"; import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js";
import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js"; import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js";
@@ -35,6 +36,15 @@ import type {
import { analyzeJavaScriptJsonModule } from "./JavaScriptJsonModules.js"; import { analyzeJavaScriptJsonModule } from "./JavaScriptJsonModules.js";
import { completeJavaScriptAnalysisSteps } from "./JavaScriptAnalysisControl.js"; import { completeJavaScriptAnalysisSteps } from "./JavaScriptAnalysisControl.js";
import { htmlArtifactReferences } from "../../domain/javascript/htmlArtifactReferences.js"; import { htmlArtifactReferences } from "../../domain/javascript/htmlArtifactReferences.js";
import type { JavaScriptSourceAnalysisPort } from "../../domain/javascript/javascriptSourceAnalysis.js";
import type { JavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import {
AnalysisCancelledError,
AnalysisResourceConstraintError,
AnalysisTimeoutError,
} from "../../domain/analysisErrorCore.js";
interface MutableArtifactAnalysis< interface MutableArtifactAnalysis<
SemanticIr extends JavaScriptModuleSemanticIr, SemanticIr extends JavaScriptModuleSemanticIr,
@@ -95,27 +105,7 @@ export const analyzeAndProjectJavaScriptArtifactFiles = async (
): Generator<void, JavaScriptModuleSemanticIr> { ): Generator<void, JavaScriptModuleSemanticIr> {
const projection = projectSemantics(file, ir); const projection = projectSemantics(file, ir);
if (projection !== undefined) while (projection.next().done !== true) yield; if (projection !== undefined) while (projection.next().done !== true) yield;
const programScopes = ir.scopes.filter(({ kind }) => kind === "program"); return projectJavaScriptModuleSemantics(ir);
const programScopeId = programScopes[0]?.scopeId;
const localNames = new Set(
ir.moduleLinks.map(({ localName }) => localName),
);
const callableIds = new Set(
ir.moduleLinks.map(({ callableId }) => callableId),
);
return {
scopes: programScopes,
bindings: ir.bindings.filter(
({ name, scopeId }) =>
scopeId === programScopeId && localNames.has(name),
),
callables: ir.callables.filter(({ callableId }) =>
callableIds.has(callableId),
),
moduleLinks: ir.moduleLinks,
coverage: ir.coverage,
limitations: ir.limitations,
};
}; };
const context = { state, projectSemantics: projectFileSemantics }; const context = { state, projectSemantics: projectFileSemantics };
for (const [index, file] of fileSet.files.entries()) { for (const [index, file] of fileSet.files.entries()) {
@@ -130,6 +120,174 @@ export const analyzeAndProjectJavaScriptArtifactFiles = async (
return finalizeArtifactAnalysis(state); return finalizeArtifactAnalysis(state);
}; };
/** Analyze sources behind a hard heap boundary, retaining collected facts on interruption. */
export const analyzeAndAdoptJavaScriptArtifactFiles = async (
fileSet: JavaScriptArtifactFileSet,
worker: Pick<JavaScriptSourceAnalysisPort, "analyze">,
projection: JavaScriptSemanticGraphProjection,
beforeFile: (
file: JavaScriptArtifactFile,
completed: number,
total: number,
) => Promise<void>,
signal?: AbortSignal,
): Promise<{
readonly analysis: JavaScriptModuleArtifactAnalysis;
readonly interruption: AnalysisError | null;
}> => {
const state = emptyArtifactAnalysis<JavaScriptModuleSemanticIr>();
let interruption: AnalysisError | null = null;
for (const [index, file] of fileSet.files.entries()) {
try {
await setImmediate();
await beforeFile(file, index, fileSet.files.length);
if (signal?.aborted === true)
throw new AnalysisCancelledError("analyze_javascript_application");
if (file.kind !== "javascript" || !file.text.included) {
const steps = analyzeArtifactFileSteps(file, {
state,
projectSemantics: function* (_file, ir) {
yield;
return projectJavaScriptModuleSemantics(ir);
},
});
await completeJavaScriptAnalysisSteps(steps, signal);
continue;
}
const result = await worker.analyze(
file,
projection.remainingFileNodeBudget(),
signal,
);
const facts = result.ok ? result.value : result.error;
const applicationProjectionFailure = facts.applicationProjectionFailure;
if (facts.projection !== null)
projection.adoptFileProjection(facts.projection);
const error = result.ok ? null : result.error.error;
const limits =
error instanceof AnalysisResourceConstraintError
? error.reportedLimits
: error instanceof AnalysisTimeoutError
? { worker_timeout_ms: error.timeoutMs }
: null;
const failure =
error === null
? undefined
: {
reason: `${file.path}: ${error.message}`,
error: projectAnalysisError(error),
limits,
};
state.files.push({
file,
javascript: facts.javascript,
semantic: facts.module === null ? null : { ir: facts.module },
...(failure === undefined ? {} : { analysis_failure: failure }),
...(applicationProjectionFailure === undefined
? {}
: {
application_projection_failure: {
reason: applicationProjectionFailure.message,
error: projectAnalysisError(applicationProjectionFailure),
limits: applicationProjectionFailure.reportedLimits,
},
}),
});
if (failure !== undefined && facts.projection === null)
projection.recordUnavailableFile(
file,
failure.reason,
failure.limits,
error instanceof AnalysisResourceConstraintError ||
error instanceof AnalysisTimeoutError
? "resource-limit"
: "incomplete-module",
);
if (facts.javascript !== null) {
state.visitedNodes += facts.javascript.visited_ast_nodes;
state.findings +=
findingCount(facts.javascript) +
(facts.module?.moduleLinks.length ?? 0);
state.modules += facts.javascript.bundler_registrations.reduce(
(count, item) => count + item.modules.length,
0,
);
if (facts.javascript.parse_status === "failed")
state.parseFailures += 1;
}
if (
error !== null &&
(!(error instanceof AnalysisResourceConstraintError) ||
error.reportedLimits?.parent_result_budget_exhausted === true)
) {
interruption = error;
break;
}
} catch (cause: unknown) {
if (signal?.aborted !== true && !(cause instanceof AnalysisError))
throw cause;
interruption =
signal?.aborted === true
? new AnalysisCancelledError("analyze_javascript_application")
: cause instanceof AnalysisError
? cause
: null;
if (interruption !== null) {
state.files.push({
file,
javascript: null,
semantic: null,
analysis_failure: {
reason: `${file.path}: ${interruption.message}`,
error: projectAnalysisError(interruption),
limits: null,
},
});
projection.recordUnavailableFile(
file,
interruption.message,
null,
"incomplete-module",
);
}
break;
}
}
if (interruption !== null) {
const analyzedPaths = new Set(state.files.map(({ file }) => file.path));
const failure = {
reason: `Source analysis was interrupted: ${interruption.message}`,
error: projectAnalysisError(interruption),
limits:
interruption instanceof AnalysisResourceConstraintError
? interruption.reportedLimits
: interruption instanceof AnalysisTimeoutError
? { worker_timeout_ms: interruption.timeoutMs }
: null,
};
for (const file of fileSet.files) {
if (analyzedPaths.has(file.path)) continue;
state.files.push({
file,
javascript: null,
semantic: null,
analysis_failure: failure,
});
if (file.kind === "javascript")
projection.recordUnavailableFile(
file,
failure.reason,
failure.limits,
interruption instanceof AnalysisResourceConstraintError ||
interruption instanceof AnalysisTimeoutError
? "resource-limit"
: "incomplete-module",
);
}
}
return { analysis: finalizeArtifactAnalysis(state), interruption };
};
const analyzeArtifactFiles = <SemanticIr extends JavaScriptModuleSemanticIr>( const analyzeArtifactFiles = <SemanticIr extends JavaScriptModuleSemanticIr>(
fileSet: JavaScriptArtifactFileSet, fileSet: JavaScriptArtifactFileSet,
projectSemantics: ArtifactAnalysisContext<SemanticIr>["projectSemantics"], projectSemantics: ArtifactAnalysisContext<SemanticIr>["projectSemantics"],
@@ -161,6 +319,12 @@ const finalizeArtifactAnalysis = <
limitations: [ limitations: [
"JavaScript and HTML were parsed as inert text; bundle bootstrap code was never executed.", "JavaScript and HTML were parsed as inert text; bundle bootstrap code was never executed.",
"Static paths and relationships may remain unresolved when expressions are dynamic or obfuscated.", "Static paths and relationships may remain unresolved when expressions are dynamic or obfuscated.",
...state.files.flatMap(
({ analysis_failure, application_projection_failure }) =>
[analysis_failure, application_projection_failure].flatMap(
(failure) => (failure === undefined ? [] : [failure.reason]),
),
),
], ],
}; };
}; };
@@ -4,7 +4,9 @@ import type {
JavaScriptStaticAnalysis, JavaScriptStaticAnalysis,
} from "../../domain/javascript/javascriptStaticAnalysisTypes.js"; } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js"; import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
import type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js"; import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import type { JsonValue } from "../../domain/jsonValue.js";
interface JavaScriptStructuredObservation { interface JavaScriptStructuredObservation {
readonly path: string; readonly path: string;
@@ -89,15 +91,7 @@ export type JavaScriptJsonModuleObservation = JavaScriptStructuredObservation &
); );
/** Module facts needed after file-local semantic projection has finished. */ /** Module facts needed after file-local semantic projection has finished. */
export type JavaScriptModuleSemanticIr = Pick< export type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
JavaScriptSemanticIr,
| "scopes"
| "bindings"
| "callables"
| "moduleLinks"
| "coverage"
| "limitations"
>;
/** One relevant file plus optional AST-only JavaScript facts. */ /** One relevant file plus optional AST-only JavaScript facts. */
export interface AnalyzedJavaScriptArtifactFile< export interface AnalyzedJavaScriptArtifactFile<
@@ -108,6 +102,18 @@ export interface AnalyzedJavaScriptArtifactFile<
readonly semantic: { readonly semantic: {
readonly ir: SemanticIr; readonly ir: SemanticIr;
} | null; } | null;
/** Operational incompleteness, distinct from a malformed source file. */
readonly analysis_failure?: {
readonly reason: string;
readonly error: JsonValue;
readonly limits: Readonly<Record<string, JsonValue>> | null;
};
/** Facts retained inline when their expanded graph exceeds available memory. */
readonly application_projection_failure?: {
readonly reason: string;
readonly error: JsonValue;
readonly limits: Readonly<Record<string, JsonValue>> | null;
};
} }
/** Complete static-analysis projection before graph construction. */ /** Complete static-analysis projection before graph construction. */
@@ -12,7 +12,7 @@ export class JavaScriptArtifactGraphAccumulator {
readonly #nodes = new Map<string, ApplicationNode>(); readonly #nodes = new Map<string, ApplicationNode>();
readonly #edges = new Map<string, ApplicationEdge>(); readonly #edges = new Map<string, ApplicationEdge>();
/** Create or merge one canonical node. */ /** Create or merge one canonical node, retaining a stable live construction reference. */
addNode<Input extends DisplayableNodeInput>(input: Input): ApplicationNode { addNode<Input extends DisplayableNodeInput>(input: Input): ApplicationNode {
const created = createJavaScriptApplicationNode(displayableNode(input)); const created = createJavaScriptApplicationNode(displayableNode(input));
const existing = this.#nodes.get(created.node_id); const existing = this.#nodes.get(created.node_id);
@@ -23,19 +23,23 @@ export class JavaScriptArtifactGraphAccumulator {
const bySemanticId = new Map( const bySemanticId = new Map(
[...existing.observations, ...created.observations].map((observation) => [ [...existing.observations, ...created.observations].map((observation) => [
observation.observation_id, observation.observation_id,
observationInput(observation), observation,
]), ]),
); );
const observations = [...bySemanticId.entries()] const observations = [...bySemanticId.values()].sort((left, right) =>
.sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0)) left.observation_id < right.observation_id
.map(([, observation]) => observation); ? -1
const merged = createJavaScriptApplicationNode({ : left.observation_id > right.observation_id
kind: existing.kind, ? 1
identity: existing.identity, : 0,
observations, );
}); // Both factories already validated and identified these observations.
this.#nodes.set(merged.node_id, merged); // Replacing the node retained every previous observation array through
return merged; // path/chunk/IPC indexes when multiple paths shared an entity identity.
existing.observations.length = 0;
for (const observation of observations)
existing.observations.push(observation);
return existing;
} }
/** Create or replace one canonical edge by semantic identifier. */ /** Create or replace one canonical edge by semantic identifier. */
@@ -56,17 +60,6 @@ export class JavaScriptArtifactGraphAccumulator {
} }
} }
const observationInput = (
observation: ApplicationNode["observations"][number],
) => {
const {
observation_id: _observationId,
identifier_strategy: _strategy,
...input
} = observation;
return input;
};
/** Node fields whose display form is normalized before validation. */ /** Node fields whose display form is normalized before validation. */
interface DisplayableNodeInput { interface DisplayableNodeInput {
readonly observations: readonly { readonly label: string | null }[]; readonly observations: readonly { readonly label: string | null }[];
@@ -1,3 +1,4 @@
import { ELECTRON_IDENTITY_LIMITATION } from "../../domain/javascript/javascriptElectronMemberWrites.js";
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js"; import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import { import {
createJavaScriptApplicationGraph, createJavaScriptApplicationGraph,
@@ -107,19 +108,33 @@ function* buildJavaScriptArtifactGraphInputSteps(
yield; yield;
addJavaScriptArtifactFiles(context); addJavaScriptArtifactFiles(context);
yield; yield;
const packageRoots = addJavaScriptPackageNodes(context); // Raw observations remain on the file nodes. Only admitted expansions feed
// the relationship builders, including their Electron and bundler callers.
const projectionContext = {
...context,
analysis: {
...analysis,
files: analysis.files.map((file) =>
file.application_projection_failure === undefined
? file
: { ...file, javascript: null, semantic: null },
),
},
};
const packageRoots = addJavaScriptPackageNodes(projectionContext);
yield; yield;
addJavaScriptSourceModules(context); addJavaScriptSourceModules(projectionContext);
yield; yield;
const bundlerLimitations = addJavaScriptBundlerNodes(context); const bundlerLimitations = addJavaScriptBundlerNodes(projectionContext);
const relationshipOmissions = const relationshipOmissions =
yield* addJavaScriptModuleRelationshipsSteps(context); yield* addJavaScriptModuleRelationshipsSteps(projectionContext);
const findingLimitations = yield* addJavaScriptStaticFindingsSteps(context); const findingLimitations =
addElectronBoundaries(context); yield* addJavaScriptStaticFindingsSteps(projectionContext);
addElectronBoundaries(projectionContext);
yield; yield;
addJavaScriptHtmlRoles(context); addJavaScriptHtmlRoles(projectionContext);
yield; yield;
addJavaScriptSourceMapOriginals(context); addJavaScriptSourceMapOriginals(projectionContext);
yield; yield;
const coverage = graphCoverage(context); const coverage = graphCoverage(context);
return { return {
@@ -153,14 +168,34 @@ const graphCoverage = (context: JavaScriptArtifactGraphContext) => {
javascript !== null && javascript.parse_status === "partial", javascript !== null && javascript.parse_status === "partial",
); );
const unknownGap = const unknownGap =
context.analysis.files.some(
({ analysis_failure, application_projection_failure }) =>
analysis_failure !== undefined ||
application_projection_failure !== undefined,
) ||
context.analysis.parse_failures > 0 || context.analysis.parse_failures > 0 ||
context.fileSet.invalid_utf8_files > 0 || context.fileSet.invalid_utf8_files > 0 ||
sourceMapPolicyGap || sourceMapPolicyGap ||
malformedStructuredData || malformedStructuredData ||
partialJavaScript; partialJavaScript;
if (resourceLimits.length > 0) const executionLimits = context.analysis.files.flatMap(
({ analysis_failure, application_projection_failure }) =>
[analysis_failure, application_projection_failure].flatMap((failure) =>
failure === undefined
? []
: javaScriptAnalysisCoverageLimits(failure.limits),
),
);
if (resourceLimits.length > 0 || executionLimits.length > 0)
return partialApplicationCoverage( return partialApplicationCoverage(
semanticResourceLimitCoverage(resourceLimits), [
...new Map(
[
...semanticResourceLimitCoverage(resourceLimits),
...executionLimits,
].map((limit) => [JSON.stringify(limit), limit]),
).values(),
],
null, null,
); );
if (unknownGap) return partialApplicationCoverage([], null); if (unknownGap) return partialApplicationCoverage([], null);
@@ -221,6 +256,11 @@ const graphLimitations = (
"CommonJS and ESM binding relationships were recovered from inert syntax and resolved only within the inventoried artifact container.", "CommonJS and ESM binding relationships were recovered from inert syntax and resolved only within the inventoried artifact container.",
"Webpack/Rspack factories were recovered from AST literals; REA did not invoke push handlers or bundle bootstrap code.", "Webpack/Rspack factories were recovered from AST literals; REA did not invoke push handlers or bundle bootstrap code.",
"Static imports, entrypoints, workers, endpoints, and storage relationships do not prove runtime execution.", "Static imports, entrypoints, workers, endpoints, and storage relationships do not prove runtime execution.",
...(context.analysis.files.some(({ javascript }) =>
javascript?.limitations.includes(ELECTRON_IDENTITY_LIMITATION),
)
? [ELECTRON_IDENTITY_LIMITATION]
: []),
...(electronFindings === 0 ...(electronFindings === 0
? [] ? []
: [ : [
@@ -260,3 +300,4 @@ const graphLimitations = (
]), ]),
]; ];
}; };
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
@@ -1,4 +1,5 @@
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js"; import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import type { JsonValue } from "../../domain/jsonValue.js";
import { compositeKey } from "../../domain/unicodeCodePointOrder.js"; import { compositeKey } from "../../domain/unicodeCodePointOrder.js";
import { import {
completeApplicationCoverage, completeApplicationCoverage,
@@ -99,6 +100,8 @@ export interface UnavailableParseInput {
readonly asset: ApplicationNode; readonly asset: ApplicationNode;
readonly operation: string; readonly operation: string;
readonly limitation: string; readonly limitation: string;
readonly failure?: JsonValue;
readonly limits?: JavaScriptArtifactGraphCoverage["limits"];
} }
/** Add a direct artifact-containment edge. */ /** Add a direct artifact-containment edge. */
@@ -182,12 +185,15 @@ export const addUnavailableStaticParseScope = (
path: input.file.path, path: input.file.path,
operation: input.operation, operation: input.operation,
limitation: input.limitation, limitation: input.limitation,
...(input.failure === undefined
? {}
: { analysis_failure: input.failure }),
}, },
evidence: unavailableAstEvidence({ evidence: unavailableAstEvidence({
sha256: input.file.sha256, sha256: input.file.sha256,
path: input.file.path, path: input.file.path,
operation: input.operation, operation: input.operation,
coverage: partialApplicationCoverage([], null), coverage: partialApplicationCoverage(input.limits ?? [], null),
limitation: input.limitation, limitation: input.limitation,
}), }),
}, },
@@ -2,6 +2,9 @@ import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySn
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js"; import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
import { completeApplicationCoverage } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js"; import { completeApplicationCoverage } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
import type { JavaScriptModuleArtifactAnalysis } from "./JavaScriptArtifactAnalysisTypes.js"; import type { JavaScriptModuleArtifactAnalysis } from "./JavaScriptArtifactAnalysisTypes.js";
import { jsonValueSchema } from "../../domain/jsonValue.js";
import { freezeJsonSnapshot } from "../../domain/immutableJson.js";
import type { JavaScriptSourceRange } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
import type { import type {
JavaScriptArtifactContainer, JavaScriptArtifactContainer,
JavaScriptArtifactFile, JavaScriptArtifactFile,
@@ -12,15 +15,18 @@ import {
addUnavailableStaticParseScope, addUnavailableStaticParseScope,
artifactFileNodeKind, artifactFileNodeKind,
artifactLocalIdentity, artifactLocalIdentity,
javascriptAnalysisCoverage,
createElectronRoleNode, createElectronRoleNode,
linkElectronRoleToAsset, linkElectronRoleToAsset,
type JavaScriptArtifactGraphContext, type JavaScriptArtifactGraphContext,
} from "./JavaScriptArtifactGraphContext.js"; } from "./JavaScriptArtifactGraphContext.js";
import { import {
artifactObservationEvidence, artifactObservationEvidence,
astObservationEvidence,
staticInferenceEvidence, staticInferenceEvidence,
} from "./JavaScriptArtifactGraphEvidence.js"; } from "./JavaScriptArtifactGraphEvidence.js";
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js"; import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
import { compareUnicodeCodePoints } from "../../domain/unicodeCodePointOrder.js";
interface PackageRoleInput { interface PackageRoleInput {
readonly packageNode: ApplicationNode; readonly packageNode: ApplicationNode;
@@ -127,6 +133,40 @@ export const addJavaScriptArtifactFiles = (
analyzed.javascript, analyzed.javascript,
jsonValue, jsonValue,
); );
if (
analyzed.application_projection_failure !== undefined &&
analyzed.javascript !== null
)
context.accumulator.addNode({
kind: target.kind,
identity: target.identity,
observations: [
{
label: file.path,
properties: {
static_analysis: freezeJsonSnapshot(
jsonValueSchema.parse(analyzed.javascript),
),
semantic_module_analysis:
analyzed.semantic === null
? null
: freezeJsonSnapshot(
jsonValueSchema.parse(analyzed.semantic.ir),
),
application_projection_failure:
analyzed.application_projection_failure.error,
},
evidence: astObservationEvidence({
sha256: file.sha256,
path: file.path,
operation: "retain-unexpanded-static-findings",
range: sourceFileRange(file),
coverage: javascriptAnalysisCoverage(analyzed.javascript),
limitations: [analyzed.application_projection_failure.reason],
}),
},
],
});
context.fileNodes.set(file.path, target); context.fileNodes.set(file.path, target);
if (file.kind === "javascript") context.assetNodes.set(file.path, target); if (file.kind === "javascript") context.assetNodes.set(file.path, target);
const entry = createAsarEntry(context, file); const entry = createAsarEntry(context, file);
@@ -159,7 +199,18 @@ export const addJavaScriptArtifactFiles = (
}), }),
}); });
} }
if ( if (analyzed.analysis_failure !== undefined)
addUnavailableStaticParseScope(context, {
file,
asset: target,
operation: "analyze-javascript-source",
limitation: analyzed.analysis_failure.reason,
failure: analyzed.analysis_failure.error,
limits: javaScriptAnalysisCoverageLimits(
analyzed.analysis_failure.limits,
),
});
else if (
file.kind === "javascript" && file.kind === "javascript" &&
(analyzed.javascript === null || (analyzed.javascript === null ||
analyzed.javascript.parse_status === "failed") analyzed.javascript.parse_status === "failed")
@@ -172,6 +223,17 @@ export const addJavaScriptArtifactFiles = (
? "JavaScript syntax could not be parsed." ? "JavaScript syntax could not be parsed."
: `JavaScript text was unavailable: ${file.text.reason}.`, : `JavaScript text was unavailable: ${file.text.reason}.`,
}); });
if (analyzed.application_projection_failure !== undefined)
addUnavailableStaticParseScope(context, {
file,
asset: target,
operation: "project-javascript-application",
limitation: analyzed.application_projection_failure.reason,
failure: analyzed.application_projection_failure.error,
limits: javaScriptAnalysisCoverageLimits(
analyzed.application_projection_failure.limits,
),
});
const packageValue = packagesByPath.get(file.path); const packageValue = packagesByPath.get(file.path);
if (packageValue !== undefined && packageValue.status !== "included") if (packageValue !== undefined && packageValue.status !== "included")
addUnavailableStaticParseScope(context, { addUnavailableStaticParseScope(context, {
@@ -198,11 +260,29 @@ export const addJavaScriptArtifactFiles = (
} }
}; };
const sourceFileRange = (
file: JavaScriptArtifactFile,
): JavaScriptSourceRange => {
if (!file.text.included)
throw new TypeError("Retained static facts require their source text");
let line = 1;
let lastLineStart = 0;
for (const match of file.text.value.matchAll(/\r\n|[\n\r\u2028\u2029]/gu)) {
line += 1;
lastLineStart = match.index + match[0].length;
}
return {
start: { line: 1, column: 0 },
end: { line, column: file.text.value.length - lastLineStart },
};
};
/** Project package metadata and its declared Electron roles. */ /** Project package metadata and its declared Electron roles. */
export const addJavaScriptPackageNodes = ( export const addJavaScriptPackageNodes = (
context: JavaScriptArtifactGraphContext, context: JavaScriptArtifactGraphContext,
): ApplicationNode[] => { ): ApplicationNode[] => {
const roots: ApplicationNode[] = []; const roots: ApplicationNode[] = [];
const rootPath = applicationManifestPath(context);
for (const packageValue of context.analysis.packages) { for (const packageValue of context.analysis.packages) {
const file = context.filesByPath.get(packageValue.path); const file = context.filesByPath.get(packageValue.path);
if (file === undefined) continue; if (file === undefined) continue;
@@ -231,7 +311,7 @@ export const addJavaScriptPackageNodes = (
}, },
], ],
}); });
if (roots.length === 0) { if (file.path === rootPath) {
roots.push(node); roots.push(node);
context.accumulator.addEdge({ context.accumulator.addEdge({
source_node_id: node.node_id, source_node_id: node.node_id,
@@ -252,6 +332,7 @@ export const addJavaScriptPackageNodes = (
file, file,
operation: "inventory-package", operation: "inventory-package",
}); });
if (dependencyManifest(file.path)) continue;
addPackageRole(context, { addPackageRole(context, {
packageNode: node, packageNode: node,
packageFile: file, packageFile: file,
@@ -268,6 +349,24 @@ export const addJavaScriptPackageNodes = (
return roots; return roots;
}; };
/** Dependency-only artifacts retain their inventory root. */
const applicationManifestPath = (
context: JavaScriptArtifactGraphContext,
): string | undefined =>
context.analysis.packages
.map(({ path }) => path)
.filter(
(path) => context.filesByPath.has(path) && !dependencyManifest(path),
)
.sort(
(left, right) =>
left.split("/").length - right.split("/").length ||
compareUnicodeCodePoints(left, right),
)[0];
const dependencyManifest = (path: string): boolean =>
path.split("/").slice(0, -1).includes("node_modules");
const createFileTarget = ( const createFileTarget = (
context: JavaScriptArtifactGraphContext, context: JavaScriptArtifactGraphContext,
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
@@ -407,3 +506,4 @@ const containerProperties = (container: JavaScriptArtifactContainer) => ({
bytes: container.bytes, bytes: container.bytes,
inventory_artifact_id: container.inventory_artifact_id, inventory_artifact_id: container.inventory_artifact_id,
}); });
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
@@ -1015,6 +1015,7 @@ describe("contextual JavaScript module identity", () => {
it("retains CommonJS and ESM file-base identity during inert extraction", () => { it("retains CommonJS and ESM file-base identity during inert extraction", () => {
const analysis = analyzeJavaScriptStaticSource( const analysis = analyzeJavaScriptStaticSource(
` `
import { BrowserWindow } from "electron";
new BrowserWindow({ webPreferences: { new BrowserWindow({ webPreferences: {
preload: path.resolve("ignored", __dirname, "preload.cjs") preload: path.resolve("ignored", __dirname, "preload.cjs")
}}); }});
@@ -11,9 +11,15 @@ import { createJavaScriptArtifactReader as createReader } from "../../artifacts/
import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js"; import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js";
import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js"; import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js";
import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { parseOwnedJavaScriptApplicationAnalysisSteps } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js"; import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import type { ArtifactInventoryPartialObservation } from "../../domain/artifactPartialObservation.js"; import type { ArtifactInventoryPartialObservation } from "../../domain/artifactPartialObservation.js";
import { analyzeAndProjectJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js"; import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
import { analyzeAndAdoptJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js";
import { createJavaScriptSourceAnalysis } from "../../artifacts/javascript/JavaScriptSourceAnalysis.js";
import { JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
import { createOwnedJavaScriptApplicationEvidenceCooperatively } from "./JavaScriptApplicationEvidence.js";
import { readJavaScriptArtifactFiles } from "../../artifacts/javascript/JavaScriptArtifactFiles.js"; import { readJavaScriptArtifactFiles } from "../../artifacts/javascript/JavaScriptArtifactFiles.js";
import { buildImmutableJavaScriptArtifactGraphSteps } from "./JavaScriptArtifactGraphBuilder.js"; import { buildImmutableJavaScriptArtifactGraphSteps } from "./JavaScriptArtifactGraphBuilder.js";
import { import {
@@ -21,7 +27,6 @@ import {
type JavaScriptArtifactReconstructionInput, type JavaScriptArtifactReconstructionInput,
} from "./JavaScriptArtifactReconstructionInput.js"; } from "./JavaScriptArtifactReconstructionInput.js";
import { scanCanonicalArtifactInventoryInScope } from "../../artifacts/inventory/scanCanonical.js"; import { scanCanonicalArtifactInventoryInScope } from "../../artifacts/inventory/scanCanonical.js";
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
import { summarizeElectronBoundaries } from "./ElectronBoundaryAnalysis.js"; import { summarizeElectronBoundaries } from "./ElectronBoundaryAnalysis.js";
import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js"; import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
import type { ProgressReporter } from "../ProgressReporter.js"; import type { ProgressReporter } from "../ProgressReporter.js";
@@ -63,12 +68,19 @@ export const reconstructJavaScriptArtifact = async (
progress?: ProgressReporter, progress?: ProgressReporter,
): Promise<JavaScriptArtifactReconstructionResult> => ): Promise<JavaScriptArtifactReconstructionResult> =>
scope.run(async () => { scope.run(async () => {
let effectiveSignal = signal;
let effectiveProgress = progress;
const reportPhase = async ( const reportPhase = async (
phase: string, phase: string,
message: string, message: string,
): Promise<void> => { ): Promise<void> => {
await progress?.report({ phase, completed: 0, total: 1, message }); await effectiveProgress?.report({
await checkpointJavaScriptAnalysis(signal); phase,
completed: 0,
total: 1,
message,
});
await checkpointJavaScriptAnalysis(effectiveSignal);
}; };
const input = javascriptArtifactReconstructionInputSchema.parse(rawInput); const input = javascriptArtifactReconstructionInputSchema.parse(rawInput);
abortIfNeeded(signal); abortIfNeeded(signal);
@@ -85,7 +97,13 @@ export const reconstructJavaScriptArtifact = async (
`Artifact inventory classified ${path} as ${snapshot.manifest.root_format}, not ${format}`, `Artifact inventory classified ${path} as ${snapshot.manifest.root_format}, not ${format}`,
); );
const reader = createReader(path, format); const reader = createReader(path, format);
let outcome: const readerOwner = {
kind: "reader" as const,
reader,
resource: `JavaScript artifact reader for ${path}`,
};
let readerOpen = true;
let reconstructionOutcome:
| { | {
readonly kind: "completed"; readonly kind: "completed";
readonly result: JavaScriptArtifactReconstructionResult; readonly result: JavaScriptArtifactReconstructionResult;
@@ -102,28 +120,75 @@ export const reconstructJavaScriptArtifact = async (
scope, scope,
signal, signal,
); );
// Release acquisition before analysis consumes these owned file facts.
const earlyCleanup = await scope.release(readerOwner);
readerOpen = false;
if (earlyCleanup.kind === "failed")
throw ArtifactReaderFailure.withCleanup(
earlyCleanup.cause,
ArtifactReaderFailure.cleanupObservation(
earlyCleanup.cause,
readerOwner.resource,
),
{ kind: "artifact-inventory", inventory: snapshot },
);
await reportPhase( await reportPhase(
"parse_javascript_sources", "parse_javascript_sources",
`Parsing and projecting ${String(files.files.length)} application source files`, `Parsing and projecting ${String(files.files.length)} application source files`,
); );
const semanticProjection = createJavaScriptSemanticGraphProjection(); const semanticProjection = createJavaScriptSemanticGraphProjection();
const analysis = await analyzeAndProjectJavaScriptArtifactFiles( const worker = createJavaScriptSourceAnalysis(
files, {
semanticProjection.projectFileSteps, heapMb:
async (file, completed, total) => { input.max_heap_mb ?? JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS.heapMb,
abortIfNeeded(signal); timeoutMs:
await progress?.report({ input.analysis_timeout_ms ??
phase: "parse_javascript_source", JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS.timeoutMs,
completed: 0,
total: 1,
message: `Parsing and projecting ${file.path} (${String(completed + 1)}/${String(total)})`,
});
abortIfNeeded(signal);
}, },
signal, files.files.filter(
(file) => file.kind === "javascript" && file.text.included,
).length,
); );
await checkpointJavaScriptAnalysis(signal); let outcome;
abortIfNeeded(signal); let cleanupFailure: AnalysisError | undefined;
try {
outcome = await analyzeAndAdoptJavaScriptArtifactFiles(
files,
worker,
semanticProjection,
async (file, completed, total) => {
abortIfNeeded(signal);
await progress?.report({
phase: "parse_javascript_source",
completed: 0,
total: 1,
message: `Parsing and projecting ${file.path} (${String(completed + 1)}/${String(total)})`,
});
abortIfNeeded(signal);
},
signal,
);
} finally {
try {
const cleanup = await scope.release({
kind: "javascript-source-analysis",
analysis: worker,
resource: `JavaScript source analyzer for ${path}`,
});
if (cleanup.kind === "failed") throw cleanup.cause;
} catch (cause: unknown) {
if (!(cause instanceof AnalysisError)) throw cause;
cleanupFailure = cause;
}
}
const analysis = outcome.analysis;
const interruption = cleanupFailure ?? outcome.interruption;
if (interruption !== null && interruption !== undefined) {
effectiveSignal = undefined;
effectiveProgress = undefined;
}
await checkpointJavaScriptAnalysis(effectiveSignal);
abortIfNeeded(effectiveSignal);
await reportPhase( await reportPhase(
"build_javascript_application_graph", "build_javascript_application_graph",
"Constructing application and Electron boundary relationships", "Constructing application and Electron boundary relationships",
@@ -139,7 +204,7 @@ export const reconstructJavaScriptArtifact = async (
); );
const graph = await completeJavaScriptAnalysisSteps( const graph = await completeJavaScriptAnalysisSteps(
applicationGraphSteps, applicationGraphSteps,
signal, effectiveSignal,
); );
await reportPhase( await reportPhase(
"build_javascript_semantic_graph", "build_javascript_semantic_graph",
@@ -155,63 +220,83 @@ export const reconstructJavaScriptArtifact = async (
); );
const semanticGraph = await completeJavaScriptAnalysisSteps( const semanticGraph = await completeJavaScriptAnalysisSteps(
semanticGraphSteps, semanticGraphSteps,
signal, effectiveSignal,
); );
await checkpointJavaScriptAnalysis(signal); await checkpointJavaScriptAnalysis(effectiveSignal);
outcome = { const completed = {
kind: "completed", input_path: path,
result: { format,
input_path: path, root_artifact_sha256: snapshot.manifest.root_sha256,
format, inventory_manifest_id: snapshot.manifest.manifest_id,
root_artifact_sha256: snapshot.manifest.root_sha256, inventory_graph_sha256: snapshot.manifest.graph_sha256,
inventory_manifest_id: snapshot.manifest.manifest_id, integrity_contradictions: snapshot.integrity_contradictions,
inventory_graph_sha256: snapshot.manifest.graph_sha256, graph,
integrity_contradictions: snapshot.integrity_contradictions, semantic_graph: semanticGraph,
graph, electron_summary: summarizeElectronBoundaries(analysis),
semantic_graph: semanticGraph, statistics: {
electron_summary: summarizeElectronBoundaries(analysis), relevant_files: files.files.length,
statistics: { nested_asar_containers: files.containers.length,
relevant_files: files.files.length, text_bytes_read: files.text_bytes_read,
nested_asar_containers: files.containers.length, invalid_utf8_files: files.invalid_utf8_files,
text_bytes_read: files.text_bytes_read, parsed_javascript_files: analysis.files.filter(
invalid_utf8_files: files.invalid_utf8_files, ({ javascript }) => javascript !== null,
parsed_javascript_files: analysis.files.filter( ).length,
({ javascript }) => javascript !== null, visited_ast_nodes: analysis.visited_ast_nodes,
).length, findings: analysis.findings,
visited_ast_nodes: analysis.visited_ast_nodes, modules: analysis.modules,
findings: analysis.findings, parse_failures: analysis.parse_failures,
modules: analysis.modules,
parse_failures: analysis.parse_failures,
},
limitations: analysis.limitations,
}, },
limitations: analysis.limitations,
}; };
if (interruption !== null && interruption !== undefined) {
const { electron_summary: summary, ...application } = completed;
const partial = await completeJavaScriptAnalysisSteps(
parseOwnedJavaScriptApplicationAnalysisSteps({
...application,
summary,
limitations: completed.graph.limitations,
}),
);
const { max_heap_mb, analysis_timeout_ms, ...coreInput } = input;
const parameters = {
...coreInput,
...(max_heap_mb === undefined ? {} : { max_heap_mb }),
...(analysis_timeout_ms === undefined ? {} : { analysis_timeout_ms }),
};
const evidence =
await createOwnedJavaScriptApplicationEvidenceCooperatively(
parameters,
partial,
);
throw interruption.retainPartialObservation(evidence);
}
reconstructionOutcome = { kind: "completed", result: completed };
} catch (cause: unknown) { } catch (cause: unknown) {
outcome = { kind: "failed", cause }; reconstructionOutcome = { kind: "failed", cause };
} }
const owner = { if (readerOpen) {
kind: "reader" as const, const cleanupAttempt = await scope.release(readerOwner);
reader, if (cleanupAttempt.kind === "failed") {
resource: `JavaScript artifact reader for ${path}`, const cleanup = ArtifactReaderFailure.cleanupObservation(
}; cleanupAttempt.cause,
const cleanupAttempt = await scope.release(owner); readerOwner.resource,
if (cleanupAttempt.kind === "failed") { );
const cleanup = ArtifactReaderFailure.cleanupObservation( const partialObservation: ArtifactInventoryPartialObservation = {
cleanupAttempt.cause, kind: "artifact-inventory",
owner.resource, inventory: snapshot,
); };
const partialObservation: ArtifactInventoryPartialObservation = { throw ArtifactReaderFailure.withCleanup(
kind: "artifact-inventory", reconstructionOutcome.kind === "failed"
inventory: snapshot, ? reconstructionOutcome.cause
}; : cleanupAttempt.cause,
throw ArtifactReaderFailure.withCleanup( cleanup,
outcome.kind === "failed" ? outcome.cause : cleanupAttempt.cause, partialObservation,
cleanup, );
partialObservation, }
);
} }
if (outcome.kind === "failed") throw outcome.cause; if (reconstructionOutcome.kind === "failed")
return outcome.result; throw reconstructionOutcome.cause;
return reconstructionOutcome.result;
}); });
const OPERATION = "analyze_javascript_application"; const OPERATION = "analyze_javascript_application";
@@ -1,11 +1,13 @@
import { z } from "zod"; import { z } from "zod";
import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js"; import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js";
import { javaScriptAnalysisResourceControlsSchema } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
/** Local ASAR/directory reconstruction request. */ /** Local ASAR/directory reconstruction request. */
export const javascriptArtifactReconstructionInputSchema = z.strictObject({ export const javascriptArtifactReconstructionInputSchema = z.strictObject({
input_path: z.string().min(1), input_path: z.string().min(1),
format: z.enum(["auto", "asar", "directory"]).default("auto"), format: z.enum(["auto", "asar", "directory"]).default("auto"),
integrity_policy: artifactIntegrityPolicySchema, integrity_policy: artifactIntegrityPolicySchema,
...javaScriptAnalysisResourceControlsSchema.shape,
}); });
/** Parsed local reconstruction request. */ /** Parsed local reconstruction request. */
@@ -129,4 +129,50 @@ describe("JavaScript recovery workflow", () => {
if (cancelled.ok) throw new Error("expected cancellation"); if (cancelled.ok) throw new Error("expected cancellation");
expect(cancelled.error._tag).toBe("AnalysisCancelledError"); expect(cancelled.error._tag).toBe("AnalysisCancelledError");
}); });
it("rejects additional_paths repeating path or each other", async () => {
const service = new JavaScriptRecoveryService({
recover: async () => {
throw new Error("port must not execute");
},
});
for (const additional_paths of [
[artifact.path],
["/tmp/dir/../bundle.js", "/tmp/bundle.js"],
["/tmp/shared.js", "/tmp/shared.js"],
]) {
const result = await service.recover({
...input,
additional_paths,
});
if (result.ok) throw new Error("expected duplicate rejection");
expect(result.error._tag).toBe("AnalysisInputError");
expect(result.error).toMatchObject({
issues: [{ path: ["additional_paths"], reason: "invalid_format" }],
});
}
});
it("rejects relative additional_paths before acquiring the port", async () => {
let invoked = false;
const service = new JavaScriptRecoveryService({
recover: async () => {
invoked = true;
return err(
new AnalysisCapabilityUnavailableError(
"fixture",
"recover_javascript_sources",
"engine missing",
),
);
},
});
const result = await service.recover({
...input,
additional_paths: ["relative-chunk.js"],
});
expect(invoked).toBe(false);
if (result.ok) throw new Error("expected invalid path");
expect(result.error).toMatchObject({
issues: [{ path: ["additional_paths"], reason: "invalid_format" }],
});
});
}); });
@@ -1,4 +1,4 @@
import { isAbsolute } from "node:path"; import { isAbsolute, resolve } from "node:path";
import { import {
AnalysisCancelledError, AnalysisCancelledError,
@@ -11,7 +11,7 @@ import {
javascriptRecoveryInputSchema, javascriptRecoveryInputSchema,
javascriptRecoveryResultSchema, javascriptRecoveryResultSchema,
} from "../../domain/javascript/javascriptRecovery.js"; } from "../../domain/javascript/javascriptRecovery.js";
import { jsonObjectSchema } from "../../domain/jsonValue.js"; import { jsonObjectSchema, jsonValueSchema } from "../../domain/jsonValue.js";
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js"; import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
import { err, ok, type Result } from "../../domain/result.js"; import { err, ok, type Result } from "../../domain/result.js";
import type { ExecutionOptions } from "../AnalysisProvider.js"; import type { ExecutionOptions } from "../AnalysisProvider.js";
@@ -44,17 +44,38 @@ export class JavaScriptRecoveryService {
const invalidPaths = (["path", "output_directory"] as const).filter( const invalidPaths = (["path", "output_directory"] as const).filter(
(field) => !isAbsolute(parsed.data[field]), (field) => !isAbsolute(parsed.data[field]),
); );
if (invalidPaths.length > 0) const invalidAdditional = (parsed.data.additional_paths ?? []).filter(
(additional) => !isAbsolute(additional),
);
if (invalidPaths.length > 0 || invalidAdditional.length > 0)
return err( return err(
new AnalysisInputError( new AnalysisInputError(OPERATION, undefined, [
OPERATION, ...invalidPaths.map((field) => ({
undefined,
invalidPaths.map((field) => ({
path: [field], path: [field],
reason: "invalid_format", reason: "invalid_format" as const,
message: `${field} must be an absolute filesystem path on this host.`, message: `${field} must be an absolute filesystem path on this host.`,
})), })),
), ...invalidAdditional.map((additional) => ({
path: ["additional_paths"],
reason: "invalid_format" as const,
message: `additional_paths entries must be absolute filesystem paths on this host: ${additional}`,
})),
]),
);
const selectedInputs = new Set(
[parsed.data.path, ...(parsed.data.additional_paths ?? [])].map((path) =>
resolve(path),
),
);
if (selectedInputs.size !== 1 + (parsed.data.additional_paths ?? []).length)
return err(
new AnalysisInputError(OPERATION, undefined, [
{
path: ["additional_paths"],
reason: "invalid_format",
message: "additional_paths must not repeat path or each other.",
},
]),
); );
const execution = await this.provider.recover(parsed.data, options); const execution = await this.provider.recover(parsed.data, options);
if (!execution.ok) return execution; if (!execution.ok) return execution;
@@ -70,6 +91,11 @@ export class JavaScriptRecoveryService {
); );
if ( if (
result.data.source.path !== parsed.data.path || result.data.source.path !== parsed.data.path ||
(result.data.additional_sources ?? []).length !==
(parsed.data.additional_paths ?? []).length ||
(parsed.data.additional_paths ?? []).some(
(path, index) => result.data.additional_sources?.[index]?.path !== path,
) ||
execution.value.subject.path !== result.data.source.path || execution.value.subject.path !== result.data.source.path ||
execution.value.subject.sha256 !== result.data.source.sha256 || execution.value.subject.sha256 !== result.data.source.sha256 ||
execution.value.provider.id !== result.data.engine.id || execution.value.provider.id !== result.data.engine.id ||
@@ -85,7 +111,7 @@ export class JavaScriptRecoveryService {
createEvidence(execution.value.subject, execution.value.provider, { createEvidence(execution.value.subject, execution.value.provider, {
operation: OPERATION, operation: OPERATION,
parameters: jsonObjectSchema.parse(parsed.data), parameters: jsonObjectSchema.parse(parsed.data),
result: result.data, result: jsonValueSchema.parse(result.data),
rawResult: execution.value.rawResult, rawResult: execution.value.rawResult,
locations: execution.value.locations, locations: execution.value.locations,
limitations: execution.value.limitations, limitations: execution.value.limitations,
@@ -1,5 +1,6 @@
import { import {
createJavaScriptSemanticGraph, createJavaScriptSemanticGraph,
createJavaScriptSemanticGraphUnknown,
sealTransferredJavaScriptSemanticGraphSteps, sealTransferredJavaScriptSemanticGraphSteps,
type JavaScriptSemanticGraph, type JavaScriptSemanticGraph,
} from "../../domain/javascript/javascriptSemanticGraph.js"; } from "../../domain/javascript/javascriptSemanticGraph.js";
@@ -10,6 +11,7 @@ import type {
import { import {
JAVASCRIPT_SEMANTIC_RELATION_FAMILIES, JAVASCRIPT_SEMANTIC_RELATION_FAMILIES,
JAVASCRIPT_SEMANTIC_RELATION_FAMILY, JAVASCRIPT_SEMANTIC_RELATION_FAMILY,
JAVASCRIPT_SEMANTIC_RELATIONS,
} from "../../domain/javascript/javascriptSemanticGraphSchemas.js"; } from "../../domain/javascript/javascriptSemanticGraphSchemas.js";
import type { import type {
JavaScriptSemanticCallArgument, JavaScriptSemanticCallArgument,
@@ -33,6 +35,13 @@ import { projectSemanticResources } from "./JavaScriptSemanticGraphResourceProje
import { projectSemanticObjects } from "./JavaScriptSemanticGraphObjectProjection.js"; import { projectSemanticObjects } from "./JavaScriptSemanticGraphObjectProjection.js";
import { projectSemanticValues } from "./JavaScriptSemanticGraphValueProjection.js"; import { projectSemanticValues } from "./JavaScriptSemanticGraphValueProjection.js";
import { inferredSemanticEvidenceAt } from "./JavaScriptSemanticGraphEvidence.js"; import { inferredSemanticEvidenceAt } from "./JavaScriptSemanticGraphEvidence.js";
import { observedSemanticEvidence } from "./JavaScriptSemanticGraphEvidence.js";
import {
partialApplicationCoverage,
type ApplicationCoverage,
} from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
import type { JsonValue } from "../../domain/jsonValue.js";
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
import { projectSemanticFunctionFingerprints } from "./JavaScriptSemanticGraphFingerprintProjection.js"; import { projectSemanticFunctionFingerprints } from "./JavaScriptSemanticGraphFingerprintProjection.js";
import { import {
projectSemanticClosureCaptures, projectSemanticClosureCaptures,
@@ -55,6 +64,7 @@ import {
bindSemanticGraphApplicationNodes, bindSemanticGraphApplicationNodes,
type SemanticGraphProjectionState as BuilderState, type SemanticGraphProjectionState as BuilderState,
} from "./JavaScriptSemanticGraphConstruction.js"; } from "./JavaScriptSemanticGraphConstruction.js";
import type { JavaScriptSemanticFileProjection } from "../../domain/javascript/javascriptSemanticFileProjection.js";
interface BuilderInput { interface BuilderInput {
readonly rootArtifactSha256: string; readonly rootArtifactSha256: string;
@@ -126,6 +136,19 @@ export const buildJavaScriptSemanticGraph = ({
/** File-local semantic projection that does not retain consumed source IR. */ /** File-local semantic projection that does not retain consumed source IR. */
export interface JavaScriptSemanticGraphProjection { export interface JavaScriptSemanticGraphProjection {
/** Node capacity offered to the next independently analyzed source file. */
readonly remainingFileNodeBudget: () => number;
/** Adopt decoded file-local facts; final validation and proof remain in this realm. */
readonly adoptFileProjection: (
projection: JavaScriptSemanticFileProjection,
) => void;
/** Preserve source-bound unknowns for an execution or result-retention failure. */
readonly recordUnavailableFile: (
file: JavaScriptArtifactFile,
reason: string,
limits: Readonly<Record<string, JsonValue>> | null,
unknownReason: "resource-limit" | "incomplete-module",
) => void;
readonly projectFile: ( readonly projectFile: (
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
ir: JavaScriptSemanticIr, ir: JavaScriptSemanticIr,
@@ -151,6 +174,12 @@ export const createJavaScriptSemanticGraphProjection =
const state = emptyState(); const state = emptyState();
const fingerprints: JavaScriptSemanticFingerprint[] = []; const fingerprints: JavaScriptSemanticFingerprint[] = [];
let truncatedFiles = 0; let truncatedFiles = 0;
let unavailableFiles = 0;
let resourceTruncatedFiles = 0;
const executionLimits = new Map<
string,
ApplicationCoverage["limits"][number]
>();
const projectSourceSteps = function* ( const projectSourceSteps = function* (
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
ir: JavaScriptSemanticIr, ir: JavaScriptSemanticIr,
@@ -207,12 +236,15 @@ export const createJavaScriptSemanticGraphProjection =
fingerprints: [...fingerprints], fingerprints: [...fingerprints],
unknowns, unknowns,
coverage: { coverage: {
status: truncatedFiles > 0 ? "partial" : "unknown", status:
truncated: truncatedFiles > 0, truncatedFiles > 0 || unavailableFiles > 0 ? "partial" : "unknown",
omitted_nodes: truncatedFiles > 0 ? null : 0, truncated: truncatedFiles > 0 || resourceTruncatedFiles > 0,
omitted_relations: truncatedFiles > 0 ? null : 0, omitted_nodes: truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
limits: omitted_relations:
truncatedFiles > 0 truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
limits: [
...executionLimits.values(),
...(truncatedFiles > 0
? [ ? [
{ {
name: "semantic_graph_node_ceiling", name: "semantic_graph_node_ceiling",
@@ -220,12 +252,14 @@ export const createJavaScriptSemanticGraphProjection =
unit: "items" as const, unit: "items" as const,
}, },
] ]
: [], : []),
],
families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({ families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({
family, family,
status: semanticFamilyStatus(family), status: semanticFamilyStatus(family),
retained_relations: retainedByFamily.get(family) ?? 0, retained_relations: retainedByFamily.get(family) ?? 0,
omitted_relations: truncatedFiles > 0 ? null : 0, omitted_relations:
truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
unknown_ids: unknownIdsByFamily.get(family) ?? [], unknown_ids: unknownIdsByFamily.get(family) ?? [],
})), })),
}, },
@@ -252,6 +286,87 @@ export const createJavaScriptSemanticGraphProjection =
return graph; return graph;
}; };
return { return {
remainingFileNodeBudget: () =>
Math.min(
SEMANTIC_GRAPH_FILE_NODE_CEILING,
Math.max(0, SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size),
),
adoptFileProjection: (projection) => {
const newNodes = projection.nodes.filter(
({ node_id }) => !state.nodes.has(node_id),
);
if (
newNodes.length >
Math.min(
SEMANTIC_GRAPH_FILE_NODE_CEILING,
SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size,
)
)
throw new RangeError(
"Transferred semantic nodes exceed the offered tree capacity",
);
for (const context of projection.evidenceContexts)
state.evidenceContexts.adopt(context);
for (const node of newNodes) state.nodes.set(node.node_id, node);
for (const relation of projection.relations)
state.relations.set(relation.relation_id, relation);
for (const unknown of projection.unknowns)
state.unknowns.set(unknown.unknown_id, unknown);
for (const root of projection.roots) state.roots.add(root);
fingerprints.push(...projection.fingerprints);
if (projection.truncated) truncatedFiles += 1;
},
recordUnavailableFile: (file, reason, limits, unknownReason) => {
const retainedLimits = javaScriptAnalysisCoverageLimits(limits);
for (const limit of retainedLimits) {
executionLimits.set(
JSON.stringify([limit.name, limit.value, limit.unit]),
limit,
);
}
state.fileNodeBudget = Math.min(
SEMANTIC_GRAPH_FILE_NODE_CEILING,
Math.max(0, SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size),
);
const module = retainNode(state, file, {
kind: "module",
roleKey: "module",
location: null,
label: file.path,
functionNodeId: null,
});
state.fileNodeBudget = null;
if (module !== null) state.roots.add(module.node_id);
const evidence = {
...observedSemanticEvidence(file, null),
authority: "unknown" as const,
state: "unknown" as const,
confidence: "unknown" as const,
coverage: partialApplicationCoverage(retainedLimits, null),
limitations: [reason],
};
for (const family of JAVASCRIPT_SEMANTIC_RELATION_FAMILIES) {
const unknown = createJavaScriptSemanticGraphUnknown(
{
node_id: module?.node_id ?? null,
family,
relation_kinds: JAVASCRIPT_SEMANTIC_RELATIONS.filter(
(relation) =>
JAVASCRIPT_SEMANTIC_RELATION_FAMILY[relation] === family,
),
reason: unknownReason,
detail: reason,
candidate_node_ids: [],
evidence,
},
state.evidenceContexts,
);
state.unknowns.set(unknown.unknown_id, unknown);
}
unavailableFiles += 1;
if (unknownReason === "resource-limit" && retainedLimits.length > 0)
resourceTruncatedFiles += 1;
},
projectFile: (file, ir) => { projectFile: (file, ir) => {
const steps = projectSourceSteps(file, ir); const steps = projectSourceSteps(file, ir);
while (steps.next().done !== true); while (steps.next().done !== true);
@@ -272,6 +387,35 @@ export const createJavaScriptSemanticGraphProjection =
}; };
}; };
/** Project one source with the parent's remaining node capacity, retaining no full IR. */
export function* projectJavaScriptSemanticFileSteps(
file: JavaScriptArtifactFile,
ir: JavaScriptSemanticIr,
nodeBudget: number,
): Generator<void, JavaScriptSemanticFileProjection> {
if (
!Number.isSafeInteger(nodeBudget) ||
nodeBudget < 0 ||
nodeBudget > SEMANTIC_GRAPH_FILE_NODE_CEILING
)
throw new RangeError(
"File semantic projection requires a valid offered node capacity",
);
const state = emptyState();
state.fileNodeBudget = nodeBudget;
state.fileNodesDropped = false;
const fingerprints = yield* projectFileSteps(file, ir, state);
return {
roots: [...state.roots],
evidenceContexts: state.evidenceContexts.contexts,
nodes: [...state.nodes.values()],
relations: [...state.relations.values()],
unknowns: [...state.unknowns.values()],
fingerprints,
truncated: Boolean(state.fileNodesDropped),
};
}
function* projectFileSteps( function* projectFileSteps(
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
ir: JavaScriptSemanticIr, ir: JavaScriptSemanticIr,
@@ -229,6 +229,211 @@ describe("native value parameter-use node membership", () => {
}); });
}); });
const nativeFlowDossier = (
base: ReturnType<typeof functionDossierSchema.parse>,
facts: {
address: string;
name: string;
operations: ReturnType<typeof node>[];
parameters: { ordinal: number; name: string; data_type: string }[];
},
) => {
const { address, name, operations, parameters } = facts;
return jsonValueSchema.parse({
...base,
procedure: {
address,
name,
classification: null,
body: {
available: false,
reason: "The provider did not report complete function body ranges.",
},
signature: null,
locals: [],
},
native_value_flow: {
available: true,
provenance: "ghidra-high-pcode",
operations,
def_use: [],
effects: [],
parameters,
parameter_uses: [],
truncated: false,
omitted_operations_lower_bound: 0,
known_omitted_inputs: 0,
known_omitted_edges: 0,
limitations: [],
},
});
};
const callBindingAnalysis = (
callerAddress: string,
calleeAddress: string,
caller: ReturnType<typeof jsonValueSchema.parse>,
callee: ReturnType<typeof jsonValueSchema.parse>,
): AnalysisOperationPort => ({
execute: async (operation, parameters) => {
if (operation === "resolve_native_call_targets")
return ok(
createAnalysisExecution(
{
call_site: String(parameters.address),
procedure: calleeAddress,
status: "direct",
mechanism: "direct",
targets: [
{
address: calleeAddress,
procedure: calleeAddress,
status: "direct",
basis: "provider-reference",
references: [],
},
],
limitations: [],
},
provider,
{ subject },
),
);
if (operation !== "analyze_function")
throw new Error(`Unexpected native trace operation: ${operation}`);
if (
parameters.procedure !== callerAddress &&
parameters.procedure !== calleeAddress
)
throw new Error(`Unexpected procedure: ${String(parameters.procedure)}`);
return ok(
createAnalysisExecution(
parameters.procedure === callerAddress ? caller : callee,
provider,
{ subject },
),
);
},
});
describe("native value call-binding fan-out", () => {
it("preserves binding order and edge-budget behavior for repeated callees", async () => {
const callCount = 128;
const callerAddress = "0x1000";
const calleeAddress = "0x2000";
const base = functionDossierSchema.parse(ghidraFunctionDossier());
const caller = nativeFlowDossier(base, {
address: callerAddress,
name: "caller",
operations: Array.from({ length: callCount }, (_, index) => ({
...node(
`call-${index}`,
`0x${(0x1000 + index * 4).toString(16)}`,
"CALL",
),
sequence: index,
inputs: [scalar, scalar],
})),
parameters: [],
});
const callee = nativeFlowDossier(base, {
address: calleeAddress,
name: "callee",
operations: [node("return", calleeAddress, "RETURN")],
parameters: [{ ordinal: 0, name: "value", data_type: "/int" }],
});
const fanoutAnalysis = callBindingAnalysis(
callerAddress,
calleeAddress,
caller,
callee,
);
const expectedEdges = [
...Array.from({ length: callCount }, (_, index) => ({
source: `${callerAddress}/call-${index}`,
target: calleeAddress,
input_index: null,
kind: "call",
})),
...Array.from({ length: callCount }, (_, index) => [
{
source: `${callerAddress}/call-${index}`,
target: `${calleeAddress}/parameter:0`,
input_index: 1,
kind: "argument-binding",
},
{
source: `${calleeAddress}/return`,
target: `${callerAddress}/call-${index}`,
input_index: 1,
kind: "return-binding",
},
]).flat(),
];
const request = {
procedure: callerAddress,
max_functions: 2,
max_call_sites: callCount,
max_nodes: callCount + 2,
max_edges: callCount * 3,
limit: callCount + 2,
};
const complete = await traceNativeValues(fanoutAnalysis, request);
if (!complete.ok) throw complete.error;
const completeResult = nativeValueTraceSchema.parse(complete.value);
const edgeShape = (edges: typeof completeResult.edges) =>
edges.map(({ source, target, input_index, kind }) => ({
source,
target,
input_index,
kind,
}));
expect(completeResult).toMatchObject({
total_nodes: callCount + 2,
total_edges: callCount * 3,
truncated: false,
unknowns: [],
});
expect(edgeShape(completeResult.edges)).toEqual(expectedEdges);
const capped = await traceNativeValues(fanoutAnalysis, {
...request,
max_edges: callCount * 3 - 1,
});
if (!capped.ok) throw capped.error;
const cappedResult = nativeValueTraceSchema.parse(capped.value);
expect(cappedResult).toMatchObject({
total_nodes: callCount + 2,
total_edges: callCount * 3 - 1,
truncated: true,
});
expect(edgeShape(cappedResult.edges)).toEqual(
expectedEdges.slice(0, callCount * 3 - 1),
);
const nodeCapped = await traceNativeValues(fanoutAnalysis, {
...request,
max_nodes: callCount + 1,
});
if (!nodeCapped.ok) throw nodeCapped.error;
const nodeCappedResult = nativeValueTraceSchema.parse(nodeCapped.value);
expect(nodeCappedResult).toMatchObject({
total_nodes: callCount + 1,
total_edges: callCount * 2,
truncated: true,
unknowns: Array.from({ length: callCount }, () => ({
procedure: callerAddress,
address: null,
reason: `Parameter bindings for ${calleeAddress} are unavailable or omitted`,
})),
});
expect(edgeShape(nodeCappedResult.edges)).toEqual(
expectedEdges.filter(({ kind }) => kind !== "argument-binding"),
);
});
});
describe("bounded native value dependency composition", () => { describe("bounded native value dependency composition", () => {
it("reports work truncation and stable pagination without traversing ambiguous calls", async () => { it("reports work truncation and stable pagination without traversing ambiguous calls", async () => {
const bounded = await traceNativeValues(analysis, { const bounded = await traceNativeValues(analysis, {
+18 -9
View File
@@ -305,10 +305,24 @@ export const traceNativeValues = async (
"No seed function could be analyzed", "No seed function could be analyzed",
), ),
); );
const parametersByProcedure = new Map<string, typeof nodes>();
const returnsByProcedure = new Map<string, typeof nodes>();
for (const node of nodes) {
if (node.kind === "parameter") {
const parameters = parametersByProcedure.get(node.procedure) ?? [];
parameters.push(node);
parametersByProcedure.set(node.procedure, parameters);
} else if (
node.operation?.opcode === "RETURN" &&
node.operation.inputs.length > 1
) {
const returns = returnsByProcedure.get(node.procedure) ?? [];
returns.push(node);
returnsByProcedure.set(node.procedure, returns);
}
}
for (const binding of callBindings) { for (const binding of callBindings) {
const parameters = nodes.filter( const parameters = parametersByProcedure.get(binding.callee) ?? [];
(node) => node.procedure === binding.callee && node.kind === "parameter",
);
if (parameters.length === 0) if (parameters.length === 0)
unknowns.push({ unknowns.push({
procedure: binding.caller, procedure: binding.caller,
@@ -339,12 +353,7 @@ export const traceNativeValues = async (
}); });
} }
if (binding.output) if (binding.output)
for (const node of nodes.filter( for (const node of returnsByProcedure.get(binding.callee) ?? []) {
(node) =>
node.procedure === binding.callee &&
node.operation?.opcode === "RETURN" &&
node.operation.inputs.length > 1,
)) {
if (edges.length >= input.max_edges) { if (edges.length >= input.max_edges) {
truncated = true; truncated = true;
break; break;
@@ -1,5 +1,7 @@
import { expect, it } from "vitest"; import { expect, it } from "vitest";
import { err } from "../domain/result.js";
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
import type { ArtifactReader } from "./ArtifactReader.js"; import type { ArtifactReader } from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js"; import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
@@ -66,3 +68,47 @@ it("retains the same failed reader owner and retries before admitting work", asy
await scope.close(); await scope.close();
expect(closeAttempts).toBe(4); expect(closeAttempts).toBe(4);
}); });
it("retains a failed source analyzer until shutdown can verify its release", async () => {
const scope = new ArtifactResourceScope();
let releaseAllowed = false;
const failure = new AnalysisOutputError(
"analyze_javascript_application",
"Owned worker is still running",
);
const owner = {
kind: "javascript-source-analysis" as const,
resource: "fixture source analyzer",
analysis: {
async analyze() {
return err({
error: failure,
javascript: null,
module: null,
projection: null,
});
},
async close() {
if (!releaseAllowed) throw failure;
},
},
};
expect(await scope.release(owner)).toEqual({
kind: "failed",
cause: failure,
});
let admitted = false;
await expect(
scope.run(async () => {
admitted = true;
}),
).rejects.toMatchObject({
cleanup: { resources: ["fixture source analyzer"] },
});
expect(admitted).toBe(false);
await expect(scope.close()).rejects.toMatchObject({
cleanup: { resources: ["fixture source analyzer"] },
});
releaseAllowed = true;
await scope.close();
});
+9
View File
@@ -1,3 +1,4 @@
import type { JavaScriptSourceAnalysisPort } from "../domain/javascript/javascriptSourceAnalysis.js";
import type { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js"; import type { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
import type { OwnedDirectoryHandle } from "../filesystem/OwnedDirectoryHandle.js"; import type { OwnedDirectoryHandle } from "../filesystem/OwnedDirectoryHandle.js";
@@ -7,6 +8,11 @@ import type { SafeOutputTree } from "./SafeOutputTree.js";
/** One artifact resource and its format-specific cleanup owner. */ /** One artifact resource and its format-specific cleanup owner. */
export type ArtifactResourceOwner = export type ArtifactResourceOwner =
| {
readonly kind: "javascript-source-analysis";
readonly resource: string;
readonly analysis: JavaScriptSourceAnalysisPort;
}
| { | {
readonly kind: "directory-handle"; readonly kind: "directory-handle";
readonly resource: string; readonly resource: string;
@@ -134,6 +140,9 @@ export class ArtifactResourceScope {
const cleanup = async (owner: ArtifactResourceOwner): Promise<void> => { const cleanup = async (owner: ArtifactResourceOwner): Promise<void> => {
switch (owner.kind) { switch (owner.kind) {
case "javascript-source-analysis":
await owner.analysis.close();
return;
case "reader": case "reader":
await owner.reader.close(); await owner.reader.close();
return; return;
+133 -67
View File
@@ -54,6 +54,7 @@ export class AsarArtifactReader implements ArtifactReader {
>(); >();
#snapshot: { readonly path: string; readonly sha256: string } | undefined; #snapshot: { readonly path: string; readonly sha256: string } | undefined;
#snapshotRoot: string | undefined; #snapshotRoot: string | undefined;
#snapshotOperation: Promise<void> = Promise.resolve();
constructor( constructor(
private readonly path: string, private readonly path: string,
@@ -234,62 +235,20 @@ export class AsarArtifactReader implements ArtifactReader {
expectedSha256?: string, expectedSha256?: string,
signal?: AbortSignal, signal?: AbortSignal,
): Promise<void> { ): Promise<void> {
abortIfNeeded(signal); await this.#withSnapshotOperation(async () => {
if (this.#snapshot === undefined && this.#snapshotRoot !== undefined) abortIfNeeded(signal);
await this.close(); if (this.#snapshot === undefined && this.#snapshotRoot !== undefined)
if (this.#snapshot === undefined) { await this.#closeSnapshot();
try { if (this.#snapshot === undefined) await this.#captureSnapshot(signal);
this.#snapshotRoot = await mkdtemp( if (
join(tmpdir(), "rea-asar-snapshot-"), expectedSha256 !== undefined &&
this.#snapshot?.sha256 !== expectedSha256
)
throw new ArtifactReaderFailure(
"integrity",
`ASAR container changed before interpretation: ${this.path}`,
); );
const snapshotPath = join(this.#snapshotRoot, "container.asar"); }, signal);
const hash = createHash("sha256");
const handle = await open(
this.path,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
const source = handle.createReadStream();
const digesting = new Transform({
transform(chunk: Buffer, _encoding, callback) {
hash.update(chunk);
callback(null, chunk);
},
});
await pipeline(
source,
digesting,
createWriteStream(snapshotPath, { flags: "wx", mode: 0o600 }),
signal === undefined ? {} : { signal },
);
this.#snapshot = { path: snapshotPath, sha256: hash.digest("hex") };
} catch (cause: unknown) {
const primary =
signal?.aborted === true
? new ArtifactReaderFailure(
"cancelled",
"ASAR operation cancelled",
{ cause },
)
: asarFailure(this.path, "snapshot", cause);
try {
await this.close();
} catch (cleanupCause: unknown) {
throw ArtifactReaderFailure.withCleanup(
primary,
ArtifactReaderFailure.cleanupObservation(cleanupCause, this.path),
);
}
throw primary;
}
}
if (
expectedSha256 !== undefined &&
this.#snapshot.sha256 !== expectedSha256
)
throw new ArtifactReaderFailure(
"integrity",
`ASAR container changed before interpretation: ${this.path}`,
);
} }
/** Read the same captured container that supplies headers and packed members. */ /** Read the same captured container that supplies headers and packed members. */
@@ -302,30 +261,86 @@ export class AsarArtifactReader implements ArtifactReader {
} }
async close(): Promise<void> { async close(): Promise<void> {
let cleanupFailure: ArtifactReaderFailure | undefined; await this.#withSnapshotOperation(async () => {
for (const [handle, { path }] of this.#unpackedHandles) { let cleanupFailure: ArtifactReaderFailure | undefined;
for (const [handle, { path }] of this.#unpackedHandles) {
try {
await this.#closeUnpackedHandle(handle);
} catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup(
cleanupFailure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, path),
);
}
}
try { try {
await this.#closeUnpackedHandle(handle); await this.#closeSnapshot();
} catch (cause: unknown) { } catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup( cleanupFailure = ArtifactReaderFailure.withCleanup(
cleanupFailure ?? cause, cleanupFailure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, path), ArtifactReaderFailure.cleanupObservation(cause, this.path),
); );
} }
if (cleanupFailure !== undefined) throw cleanupFailure;
});
}
async #captureSnapshot(signal?: AbortSignal): Promise<void> {
try {
this.#snapshotRoot = await mkdtemp(join(tmpdir(), "rea-asar-snapshot-"));
const snapshotPath = join(this.#snapshotRoot, "container.asar");
const hash = createHash("sha256");
const handle = await open(
this.path,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
const source = handle.createReadStream();
const digesting = new Transform({
transform(chunk: Buffer, _encoding, callback) {
hash.update(chunk);
callback(null, chunk);
},
});
await pipeline(
source,
digesting,
createWriteStream(snapshotPath, { flags: "wx", mode: 0o600 }),
signal === undefined ? {} : { signal },
);
this.#snapshot = { path: snapshotPath, sha256: hash.digest("hex") };
} catch (cause: unknown) {
const primary =
signal?.aborted === true
? new ArtifactReaderFailure("cancelled", "ASAR operation cancelled", {
cause,
})
: asarFailure(this.path, "snapshot", cause);
try {
await this.#closeSnapshot();
} catch (cleanupCause: unknown) {
throw ArtifactReaderFailure.withCleanup(
primary,
ArtifactReaderFailure.cleanupObservation(
cleanupCause,
this.#snapshotRoot ?? this.path,
),
);
}
throw primary;
} }
}
async #closeSnapshot(): Promise<void> {
const root = this.#snapshotRoot; const root = this.#snapshotRoot;
if (root !== undefined) { if (root !== undefined) {
try { try {
await this.removeSnapshot(root); await this.removeSnapshot(root);
if (this.#snapshotRoot === root) this.#snapshotRoot = undefined; if (this.#snapshotRoot === root) this.#snapshotRoot = undefined;
} catch (cause: unknown) { } catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup( throw ArtifactReaderFailure.withCleanup(cause, {
cleanupFailure ?? cause, reason: `Could not remove ASAR snapshot: ${root}`,
{ resources: [root],
reason: `Could not remove ASAR snapshot: ${root}`, });
resources: [root],
},
);
} }
} }
if (root === undefined || this.#snapshotRoot === undefined) { if (root === undefined || this.#snapshotRoot === undefined) {
@@ -334,7 +349,29 @@ export class AsarArtifactReader implements ArtifactReader {
this.#resetArchiveState(); this.#resetArchiveState();
this.#entries.clear(); this.#entries.clear();
} }
if (cleanupFailure !== undefined) throw cleanupFailure; }
async #withSnapshotOperation<Value>(
operation: () => Promise<Value>,
signal?: AbortSignal,
): Promise<Value> {
const previous = this.#snapshotOperation;
let release!: () => void;
this.#snapshotOperation = new Promise<void>((resolve) => {
release = resolve;
});
try {
await waitForOperation(previous, signal);
} catch (cause: unknown) {
void previous.then(release, release);
throw cause;
}
try {
abortIfNeeded(signal);
return await operation();
} finally {
release();
}
} }
#snapshotPath(): string { #snapshotPath(): string {
@@ -511,6 +548,35 @@ const abortIfNeeded = (signal?: AbortSignal): void => {
throw new ArtifactReaderFailure("cancelled", "ASAR operation cancelled"); throw new ArtifactReaderFailure("cancelled", "ASAR operation cancelled");
}; };
const waitForOperation = async (
operation: Promise<void>,
signal?: AbortSignal,
): Promise<void> => {
if (signal === undefined) return operation;
abortIfNeeded(signal);
await new Promise<void>((resolve, reject) => {
const removeAbortListener = (): void =>
signal.removeEventListener("abort", onAbort);
const onAbort = (): void => {
removeAbortListener();
reject(
new ArtifactReaderFailure("cancelled", "ASAR operation cancelled"),
);
};
signal.addEventListener("abort", onAbort, { once: true });
operation.then(
() => {
removeAbortListener();
resolve();
},
(cause: unknown) => {
removeAbortListener();
reject(cause);
},
);
});
};
const toArtifactPath = (listed: string): string => { const toArtifactPath = (listed: string): string => {
const portable = sep === "\\" ? listed.replaceAll("\\", "/") : listed; const portable = sep === "\\" ? listed.replaceAll("\\", "/") : listed;
return portable.replace(/^\/+|\/+$/gu, ""); return portable.replace(/^\/+|\/+$/gu, "");
+14 -1
View File
@@ -191,7 +191,7 @@ export class SafeOutputTree {
"format", "format",
`Invalid expected extraction size for ${relativePath}`, `Invalid expected extraction size for ${relativePath}`,
); );
const path = normalizeArtifactPath(relativePath); const path = normalizeDestinationPath(relativePath, this.platform);
this.#registry.add(path, "file"); this.#registry.add(path, "file");
const lineage = await this.#prepareParent(path); const lineage = await this.#prepareParent(path);
const parent = lineage.at(-1); const parent = lineage.at(-1);
@@ -569,6 +569,19 @@ const writeAll = async (
} }
}; };
const normalizeDestinationPath = (
relativePath: string,
platform: NodeJS.Platform,
): string => {
const path = normalizeArtifactPath(relativePath);
if (platform === "win32" && path.includes(":"))
throw new ArtifactReaderFailure(
"path",
`Windows destination cannot materialize artifact path as a regular file: ${path}; ':' denotes alternate data stream syntax.`,
);
return path;
};
const abortIfNeeded = (signal?: AbortSignal): void => { const abortIfNeeded = (signal?: AbortSignal): void => {
if (signal?.aborted === true) if (signal?.aborted === true)
throw new ArtifactReaderFailure( throw new ArtifactReaderFailure(
+23 -13
View File
@@ -9,7 +9,7 @@ import {
} from "../../domain/propertyListKeys.js"; } from "../../domain/propertyListKeys.js";
import { decodeNibArchive, type NibArchiveDocument } from "./NibArchive.js"; import { decodeNibArchive, type NibArchiveDocument } from "./NibArchive.js";
import { import {
estimatePropertyListDecodeBytes, preflightPropertyListDecode,
InterfaceBuilderDecodeBudget, InterfaceBuilderDecodeBudget,
InterfaceBuilderDecodeBudgetExceeded, InterfaceBuilderDecodeBudgetExceeded,
} from "./InterfaceBuilderDecodeBudget.js"; } from "./InterfaceBuilderDecodeBudget.js";
@@ -47,7 +47,7 @@ export const analyzeInterfaceBuilderBundle = async (input: {
const omittedArchives: Array<{ path: string; reason: string }> = []; const omittedArchives: Array<{ path: string; reason: string }> = [];
let unreportedOmittedArchives = 0; let unreportedOmittedArchives = 0;
const incompleteHierarchies = new Map<string, number>(); const incompleteHierarchies = new Map<string, number>();
const prototypeKeyOmissions = new Map<string, number>(); const prototypeKeyOmissions = new Map<string, number | null>();
let omitted = 0; let omitted = 0;
let attempted = 0; let attempted = 0;
let aggregateInputBytes = 0; let aggregateInputBytes = 0;
@@ -93,6 +93,7 @@ export const analyzeInterfaceBuilderBundle = async (input: {
bytes.subarray(0, 10).toString("ascii") === "NIBArchive"; bytes.subarray(0, 10).toString("ascii") === "NIBArchive";
let nib: ReturnType<typeof projectNibArchive> | null = null; let nib: ReturnType<typeof projectNibArchive> | null = null;
let xmlText: string | undefined; let xmlText: string | undefined;
let binaryPrototypeKeyObserved = false;
if (isNibArchive) { if (isNibArchive) {
const documentBudget = new InterfaceBuilderDecodeBudget( const documentBudget = new InterfaceBuilderDecodeBudget(
decodeBudget.remainingBytes, decodeBudget.remainingBytes,
@@ -109,23 +110,25 @@ export const analyzeInterfaceBuilderBundle = async (input: {
} else { } else {
if (bytes.subarray(0, 8).toString("ascii") !== "bplist00") if (bytes.subarray(0, 8).toString("ascii") !== "bplist00")
xmlText = decodeXmlPlistText(bytes); xmlText = decodeXmlPlistText(bytes);
const decodedReservation = estimatePropertyListDecodeBytes( const decodedReservation = preflightPropertyListDecode(
bytes, bytes,
decodeBudget.remainingBytes, decodeBudget.remainingBytes,
xmlText, xmlText,
); );
decodeBudget.reserve( decodeBudget.reserve(
decodedReservation, decodedReservation.estimatedBytes,
"plist representation exceeds the aggregate Interface Builder decode budget", "plist representation exceeds the aggregate Interface Builder decode budget",
); );
binaryPrototypeKeyObserved =
decodedReservation.binaryPrototypeKeyObserved;
} }
const { value: raw, omittedPrototypeKeys } = const { value: raw, omittedPrototypeKeys } =
nib === null nib === null
? decodePlist(bytes, xmlText) ? decodePlist(bytes, xmlText, binaryPrototypeKeyObserved)
: { value: nib.raw, omittedPrototypeKeys: 0 }; : { value: nib.raw, omittedPrototypeKeys: 0 };
if (nib !== null && nib.omitted > 0) if (nib !== null && nib.omitted > 0)
incompleteHierarchies.set(entry.path, nib.omitted); incompleteHierarchies.set(entry.path, nib.omitted);
if (omittedPrototypeKeys > 0) if (omittedPrototypeKeys !== 0)
prototypeKeyOmissions.set(entry.path, omittedPrototypeKeys); prototypeKeyOmissions.set(entry.path, omittedPrototypeKeys);
const documentHash = createHash("sha256").update(bytes).digest("hex"); const documentHash = createHash("sha256").update(bytes).digest("hex");
documents.push({ documents.push({
@@ -226,7 +229,7 @@ const finalizeHierarchyCoverage = (
result: ReturnType<typeof buildInterfaceBuilderAnalysis>, result: ReturnType<typeof buildInterfaceBuilderAnalysis>,
counts: { counts: {
incompleteHierarchies: ReadonlyMap<string, number>; incompleteHierarchies: ReadonlyMap<string, number>;
prototypeKeyOmissions: ReadonlyMap<string, number>; prototypeKeyOmissions: ReadonlyMap<string, number | null>;
invalid: readonly string[]; invalid: readonly string[];
omittedArchives: readonly { path: string; reason: string }[]; omittedArchives: readonly { path: string; reason: string }[];
unreportedOmittedArchives: number; unreportedOmittedArchives: number;
@@ -641,11 +644,18 @@ const readEntry = async (
/** Project decoded plist data and dates while retaining omitted-key coverage. */ /** Project decoded plist data and dates while retaining omitted-key coverage. */
const decodePlist = ( const decodePlist = (
bytes: Buffer, bytes: Buffer,
xmlText?: string, xmlText: string | undefined,
): { readonly value: JsonValue; readonly omittedPrototypeKeys: number } => { binaryPrototypeKeyObserved: boolean,
const { value, omittedPrototypeKeys } = ): {
bytes.subarray(0, 8).toString("ascii") === "bplist00" readonly value: JsonValue;
? { value: parseBinary(bytes), omittedPrototypeKeys: 0 } readonly omittedPrototypeKeys: number | null;
: parseXmlPropertyList(xmlText ?? decodeXmlPlistText(bytes)); } => {
const binary = bytes.subarray(0, 8).toString("ascii") === "bplist00";
const { value, omittedPrototypeKeys } = binary
? {
value: parseBinary(bytes),
omittedPrototypeKeys: binaryPrototypeKeyObserved ? null : 0,
}
: parseXmlPropertyList(xmlText ?? decodeXmlPlistText(bytes));
return { value: projectPlistValue(value).value, omittedPrototypeKeys }; return { value: projectPlistValue(value).value, omittedPrototypeKeys };
}; };

Some files were not shown because too many files have changed in this diff Show More