Compare commits

...
Author SHA1 Message Date
Kaoru0822andClaude Opus 5.5 c865e20a0b feat(website): translate Get started into Traditional Chinese (draft)
Pilot translation created with `website.py translate zh-hant get-started`
and translated in place. It stays a noindex draft until review, so it is
not linked from other pages, the sitemap or hreflang.

- locales/zh-hant.json: Taiwan-standard UI strings for the header, footer,
  copy messages, star label and out-of-date notice.
- Prose, headings, captions, Copy labels, the agent prompts, title and
  description are translated; commands and paths come from English.
- Ten OS names are marked data-os-ui for confirmation (PowerShell's
  messages, the Start menu, File Explorer and Copy as path on Windows;
  Terminal, the Pathname command and Show Package Contents on macOS).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:31:29 +00:00
Kaoru0822andClaude Opus 5.5 0fd4021842 feat(website): keep translation tests independent of page text
The translation tests now drop existing translated pages from their site
copy, read expected UI strings from the locale file, and insert their own
paragraphs and code changes instead of editing particular English
sentences, so ordinary page edits and new translations do not break them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:31:28 +00:00
Kaoru0822andClaude Opus 5.5 4db6ff461f feat(website): fit the language menu on narrow screens
The menu's list opened leftward past the screen edge on phones, and at
320px the fourth header button overlapped the brand. Open the list to the
right below 640px, and hide the GitHub star count below 360px when a page
has a language menu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:31:28 +00:00
Kaoru0822andClaude Opus 5.5 ce58aebc0e feat(website): add language infrastructure for translated pages
Prepare translated versions of the home, Get started, Guides and FAQ pages
in zh-Hans, zh-Hant, ja, ko, de and fr. English stays at the root and each
language lives under its own directory. No translated page exists yet, so
English pages only gain an og:locale tag.

- LANGUAGES in website.py owns each language's directory, lang/hreflang tag,
  menu name and og:locale; locales/<language>.json holds the shared UI text
  (non-English locales start as English placeholders).
- sync renders the header, footer and sharing regions in the page's
  language: localized navigation with an EN badge on English-only targets,
  a native-details language menu (same page, else the language's reviewed
  home, else omitted), hreflang alternates with x-default, og:locale
  alternates and the brand link to the language's home.
- Translations record source-sha256 and status (draft or reviewed). Drafts
  are noindex and stay out of the sitemap, menus, hreflang and navigation.
- sync copies every Copy-targeted code block from English by ID, and fills
  the website:translation-notice region when the English text has changed.
  The hash leaves out shared regions and those code blocks.
- The verifier checks records, copied code, ID/Copy/data-msg parity while
  fresh, whitespace between CJK characters in ja/zh, and the page's lang.
  An out-of-date translation, inline-code differences, links to English
  pages with a reviewed translation and English placeholder UI strings
  are warnings, shown as GitHub annotations and in the job summary.
- New commands: `translate <language> <route>` creates a draft and
  `translations` reports status, freshness and OS names to confirm.
- styles.css adds per-language CJK font stacks, CJK heading, emphasis and
  line-breaking rules, German hyphenation, and the menu, badge and notice.
- The publisher runs sync after the checks so notices are current.
- website/translation.md documents the workflow, rules, register, glossary
  (with Korean), OS names to confirm and the review checklist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:23:29 +00:00
Kaoru0822andClaude Opus 5.5 80f6365523 refactor(website): run the website lane for the page-script test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:09:16 +00:00
Kaoru0822andClaude Opus 5.5 dabcd783d5 refactor(website): keep page text when a script message is missing
A page that omits a message attribute (a translation that drops one, or a
page without the shared footer) must not show empty or "undefined" text:

- Copy buttons still copy without #copy-status or its messages, keep their
  own label and leave the status empty.
- The star label is skipped when the header has no label for the count.
- The percent demo, dinosaur game and speed check leave the current text in
  place for a missing message; an unknown {placeholder} stays literal.

scripts/test-website-scripts.mjs runs main.js, home.js and dino-demo.js
against a small document model with node:test. `website.py check` runs it,
so the website CI job and the publisher's preparation job now set up the
repository's Node.js version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:09:16 +00:00
Kaoru0822andClaude Opus 5.5 3d34185171 refactor(website): move UI text out of scripts before translation
Prepare the website for localized pages without changing what English
readers see:

- Add --font-sans and --font-mono tokens and use them for every font stack.
- Read copy feedback from the shared #copy-status element, restore each
  Copy button's own label, and build the GitHub star label from header
  data attributes with Intl.PluralRules and Intl.NumberFormat for the
  page language.
- Move the percent demo, dinosaur game and speed-check messages into
  data-msg-* attributes on the home page and dino lab.
- Render the skip link from the shared header template.
- Fix the lowercase Ghidra macOS/Linux copy-button label.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 07:09:16 +00:00
YsopwnableandGitHub 21650455a9 fix(evidence): normalize unknown revisions and preserve mutation intent (#1733)
Normalize generated revision fields before hashing and retain complete typed
creation/update requests in a shared mutation Evidence producer. Keep imported
revision authentication and canonical persisted identity/digests unchanged.

Fixes #1723
Fixes #1724

Assisted-by: Suika (gensokyo-skills:suika-scatter-gather-planning)
2026-10-11 14:56:46 +08:00
YsopwnableandGitHub 844fa52772 fix(evaluation): authenticate delivered Evidence citations
Authenticate completed successful REA Evidence records for citation accounting and shared fixture grading. Preserve genuine structured and complete text-only delivery, including authenticated inventory substeps.

Closes #1728.
2026-10-11 14:48:46 +08:00
YsopwnableandGitHub e36c82becd fix(artifacts): preserve emitted ZIP entry identity
Reject duplicate ZIP filenames before an emitted entry source lookup can be overwritten. Preserve valid repeated enumeration, case-distinct paths and typed inventory path failures.

Closes #1725. Related: #1480.
2026-10-11 14:25:03 +08:00
YsopwnableandGitHub 092de8f4b3 perf(snapshot): restore large legacy caches within live memory bounds
Stream legacy snapshot cache rows independently, authenticate complete Evidence once, and restore exact immutable payloads without assembling duplicate full graphs. Preserve the existing file format, provenance, resource admission, cancellation and cache export isolation.

Verified through real Linux SDK MCP import of the original 83 MB Ghidra snapshot and full 41 MB bundle hash, actual Windows CLI/MCP PE recovery and the installed Windows large-file decoder boundary, focused tests and static checks.

Fixes #1680. Refs #1065.
2026-10-11 13:59:40 +08:00
morluto 3f4ea32189 ci: reduce macOS runner contention 2026-10-11 13:57:32 +08:00
YsopwnableandGitHub 966dc3acde perf(evidence): reuse authenticated file snapshots during session import (#1721)
Seal file-backed MCP imports at the session boundary while preserving mutable
ordinary file reads. Validate internally constructed ledger candidates directly
with the existing immutable bundle constructor.

Assisted-by: Suika (gensokyo-skills:suika-scatter-gather-planning)
2026-10-11 13:53:01 +08:00
morluto 91f6840c35 ci: parallelize native verification and share test artifacts 2026-10-11 13:43:20 +08:00
YsopwnableandGitHub d9b4ae182d fix(native): retain failed command cleanup owners
Retain failed native command owners through both xcrun resolution and command capture. Retry all owned cleanup failures before new commands and after consumers join their launched work at close; preserve partial output and every resource diagnostic.

Refs #740.
2026-10-11 13:39:42 +08:00
8be7ad496c docs(website): highlight the setup guide on the homepage (#1714)
Show the hero's Setup guide link as a secondary button and add a link
to the step-by-step setup guide under the Set up REA panel.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 22:32:12 -07:00
morluto 01762b10c6 perf(tests): consolidate TypeScript dialect producer checks 2026-10-11 13:29:50 +08:00
YsopwnableandGitHub 29ff887f65 perf(domain): avoid duplicate readiness bundle validation
Avoid reparsing the already validated Evidence bundle in readiness reports; retain semantic ID, graph and manifest authentication. Reuse canonical digest ownership and remove an unused verifier temporary directory.

Refs #1065.
2026-10-11 13:29:11 +08:00
morluto 693ed332b5 perf(tests): remove timed output and synchronize browser fixtures 2026-10-11 13:20:16 +08:00
morluto d3088c3795 perf(javascript): reuse collected callable identities for returns 2026-10-11 13:20:16 +08:00
morluto a82d14e4db perf(cli): defer command-specific modules until dispatch 2026-10-11 13:20:15 +08:00
morluto 5f4c7ff6df perf(tests): reuse stateless workflows and immutable fixtures 2026-10-11 13:20:15 +08:00
YsopwnableandGitHub 13e03e9436 perf(evidence): fuse streamed JSON token handling (#1694)
Fuse upstream JSON parsing, scalar packing, heap admission, and value assembly into synchronous token stages while retaining asynchronous file reads and existing capacity, cancellation, decoding, identity, and typed-error semantics. Declare stream-chain 4.2.6 directly.

Validation: 30 focused tests; check:fast, Knip and formatting; actual Flameox paired full-content oracles; real Linux evidence replay and Windows source-built package/Ghidra snapshot recovery; final-head CI 31/31 success.

Refs #1065, #1680.
2026-10-11 12:36:35 +08:00
4353089c67 style(website): kbd, install-node width and nested collapsible indentation (#1699)
Style <kbd> keys, cap the Get started Node.js help section at the lede
width, and indent open .evidence-details content with a subtle left
border so nested collapsibles step in at any depth.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:32:32 -07:00
morluto 57c5acac2e docs(setup): confirm published 6.4.0 client support
Verify all 17 automatic client IDs through isolated npm package dry runs and confirm the OpenCode configuration directory override. Replace obsolete npm 6.3.0 warnings with the verified release checkpoint.

Fixes #1650.
2026-10-11 12:19:15 +08:00
0ecf3f2438 perf(mcp): keep Playwright and type-only imports off MCP startup (#1691)
* fix(build): emit no runtime imports for type-only bindings

Under verbatimModuleSyntax, an import whose specifiers all carry inline
`type` modifiers compiles to `import {} from "..."`, a side-effect import
that still resolves and evaluates the module. 78 compiled modules carried
one, including `import {} from "playwright-core"` in the browser scenario
session, which loaded Playwright's module graph at MCP startup.

Use top-level `import type` for these bindings and enable oxlint's
typescript/no-import-type-side-effects rule so the pattern cannot return.

Refs #1685

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(mcp): load Playwright when a scenario launches, not at startup

The browser scenario and Electron active providers are constructed before
the stdio server serves, and both bound Playwright's launchers at module
load. Resolve `chromium` and `_electron` from playwright-core when a
capture opens instead, as the web module resolver already does.

MCP startup now resolves 1,312 modules (4,902 resolutions) instead of
1,598 (6,076) before reading the first request, and no longer loads
playwright-core. Provider construction, the tool catalog, and capability
inventory are unchanged.

A boundary test spawns the stdio server, initializes, lists tools, and
asserts playwright-core and isomorphic-git were not resolved; the CLI
probe now exercises the real launch path to prove detection.

Closes #1685

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 21:04:05 -07:00
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>morluto
a56f7bb668 chore(main): release rea-agents 6.4.0 (#1399)
* chore(main): release rea-agents 6.4.0

* docs(release): write contributor-aware 6.4.0 changelog

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 11:57:29 +08:00
morluto ee46323b49 fix(windows): preserve drive-root private runtime paths
Join the admitted parent and generated child name without duplicating a drive-root separator. Reuse that name for handle-relative creation and path provenance.

Add native snapshot, readback, and cleanup coverage through the Ghidra dot-prefixed temp fallback, plus a session-root contract regression.

Fixes #1690
2026-10-11 11:51:39 +08:00
morluto a99857df23 test(release): keep credits attached to their contribution 2026-10-11 11:51:17 +08:00
morluto 6b043dc3de fix(release): avoid duplicated contributor thanks 2026-10-11 11:49:49 +08:00
morluto 4003151ec1 fix(ci): allow browser runtime action transport latency 2026-10-11 11:45:46 +08:00
92dac97102 docs(website): add beginner help to the native analysis guide (#1689)
Add a short glossary, an analysis-tool chooser linked to native setup,
a note that REA does not include DX-Ball, and help for copying a file's
full path.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:43:06 -07:00
YsopwnableandGitHub 89ea2cad19 perf(domain): reuse canonical JSON key encodings
Reuse exact JSON key encodings within the existing 8,192-code-unit digest-buffer quantum and avoid ordinary member value frames. Preserve canonical bytes and cooperative yielding; resume objects when a pending property is omitted.

Flameox paired experiments establish a local digest-workload improvement, with complete-result/EvidenceID oracles unchanged. Full CLI acceleration and billed token reductions are not claimed.

Refs #1065.
2026-10-11 11:40:47 +08:00
ea06264a56 docs(website): list the setup guide first on the Guides page (#1688)
Move the "Set up REA" entry ahead of the native, JavaScript and browser
guides so new readers see setup before the worked examples.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:33:30 -07:00
morluto 59b4a3c319 test(windows): verify native addon in a renamed Node host 2026-10-11 11:32:07 +08:00
xeropresenceandGitHub ea1c164e5e fix(windows): load native controls in Node-API hosts not named node.exe (#1531)
fix(windows): resolve Node-API imports from the host executable

Delay-load the Node-API import library and bind "node.exe" to the running
process, matching node-gyp's win_delay_load_hook. The addon now loads under
any Node-API host (Node.js, Bun) regardless of its executable file name.
2026-10-11 11:30:41 +08:00
dc676b4d85 docs(website): add beginner help to the Get started page (#1684)
Add collapsible help sections for installing Node.js, choosing a coding
agent, opening a terminal, restarting a client, copying a folder's full
path and finding an Electron app's .asar file.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 11:27:12 +08:00
ad3e41d46e fix(browser): resolve XLink and SVG prefixes in XHTML dom_urls (#1683)
* fix(browser): resolve XLink and SVG prefixes in XHTML dom_urls

DOMSnapshot has no namespace URI. HTML documents still use the parser's
local names. In XHTML and other XML documents, resolve in-scope xmlns
declarations so an element is SVG only in the SVG namespace, and report
any XLink href as href when plain href is absent.

Refs #1586

* fix(browser): bound XML namespace resolution and reject cyclic ancestry

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 11:26:26 +08:00
YsopwnableandGitHub 17f704050f feat(evidence): discover retained records without full payloads (#1682)
* feat(evidence): discover retained records without full payloads

* fix(evidence): guide oversized results to retained discovery

* style(tests): format retained discovery recovery assertion
2026-10-11 11:18:26 +08:00
c8820e03db fix(apple): decode binary app Info.plist files without plutil (#1476)
* fix(apple): decode binary app Info.plist files without plutil

* test(apple): use Windows-legal names in the binary plist bundle test

* fix(apple): ignore __proto__ entries in binary app Info.plist files

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 11:14:15 +08:00
morluto ab80ec640a fix(ci): report package verifier stage progress 2026-10-11 11:11:38 +08:00
morluto 9ce6ef895e test: reuse stdio MCP server in acceptance file 2026-10-11 11:11:38 +08:00
morluto b3e977e6d0 fix(browser): bound screenshot comparison memory 2026-10-11 11:11:38 +08:00
morlutoandYou Wish a99b2fd3b5 test: retain focused CommonJS export regressions
Keep uncovered assignment and binding regressions in the shared CLI and MCP runtime oracle, with a POSIX escaped-path fixture.

Co-authored-by: You Wish <29498139+Y0UW1SH@users.noreply.github.com>
2026-10-11 11:07:15 +08:00
Rudy Celekliandmorluto 4e4b15b825 fix(browser): validate PNG chunks and preserve unsupported encodings
Validate chunk checksums, type bytes, ordering and end boundaries before pixel comparison. Distinguish malformed PNGs and corrupt DEFLATE streams from recognized unsupported encodings through CLI and MCP error projection.

Adapted from #1375 with additional decoder and public boundary regressions. Eighteen decoder tests and six compiled CLI/MCP cases passed.

Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 11:07:15 +08:00
morluto 21dbcf7a14 perf(cli): defer binary provider composition 2026-10-11 11:07:15 +08:00
Kaoru0822andmorluto 76296c5a38 fix(native): bind NativeAOT metadata to declared PE section extents
Section headers distinguish the declared virtual extent from the on-disk raw extent, which may include alignment padding. Resource RVA ownership and overlap use VirtualSize, falling back to SizeOfRawData when VirtualSize is zero, and payload reads still require file-backed bytes.

The NativeAOT snapshot reader applies the same range rule to RVA reads, metadata scans, executable targets, and derived overlay bounds. Page-rounded loader mappings may expose tail bytes, but those bytes lie outside the declared section range used by these parsers.
2026-10-11 11:07:15 +08:00
morlutoandYou Wish e38550462e test(native): stabilize resource-limited binary layout controls
Keep the configured launcher alive while its limited Python child runs so ownership checks retain the admitted executable identity. Trigger memory exhaustion with one oversized ELF zero-fill tail rather than racing section traversal against a timeout.

Adapted from #1357. JavaScript syntax, strict C syntax compilation, formatting, and static checks passed. The Linux pwntools verification lane was not run on this macOS host.

Co-authored-by: You Wish <29498139+Y0UW1SH@users.noreply.github.com>
2026-10-11 11:06:10 +08:00
morlutoandYou Wish 80896a1d64 fix(process): settle launcher exit without dropping descendant lineage
Refresh a mismatched launcher snapshot only when a new live process table proves its PID absent. Preserve uncertainty for detached descendants and continue checking ownership of surviving group members before signaling. Index retained lineage membership instead of rescanning it for each refreshed process.

Adapted from #1357. Validation: 71 source tests passed, one Linux-only identity case skipped; 32 real process-boundary tests passed on macOS.

Co-authored-by: You Wish <29498139+Y0UW1SH@users.noreply.github.com>
2026-10-11 11:06:10 +08:00
morluto e5bc4e85ec fix(ci): close speculative browser fixture connections 2026-10-11 11:05:08 +08:00
38221cb12e fix(native): bound PE resource RVAs by section virtual size (#1678)
The PE resource reader treated each section as covering
`max(VirtualSize, SizeOfRawData)` bytes, the mapping #1596 removed from
managed RVA reads. A resource payload in the raw alignment padding past
VirtualSize was reported with a hash of bytes the image loader never maps.
A later section whose virtual range starts inside that padding was refused
as an overlapping mapping.

Containment and overlap now use the loader's virtual extent: VirtualSize,
or SizeOfRawData when VirtualSize is zero. Ranges must still be file-backed.

Refs #1596

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 20:01:45 -07:00
3749ccec47 feat(ghidra): recover signed AArch64 relative jump tables (#1592)
* feat(ghidra): recover signed AArch64 relative jump tables

Generalise the AArch64 relative-table matcher to signed byte/halfword entries, UXTW indexes, byte or LSL #2 offsets, B.CS/B.HS bounds and a CBZ route for zero entries. Cross-check it against Ghidra's own load table when one exists, and add a fixture modelled on the RenderScript blend kernel.

Refs #487

* fix(ghidra): preserve typed labels for normalized switch selectors

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 10:45:35 +08:00
a67dbf6a48 fix: reject directory paths for the Wakaru limiter and mitmdump
`REA_JAVASCRIPT_PRLIMIT_COMMAND` and `REA_MITMDUMP_COMMAND` were admitted
after an access check alone, which a searchable directory passes. Admission
then succeeded, and the failure surfaced later as a launch-time adapter
error instead of the capability refusal that names the configured path.

Require a regular file, as #751 did for the firmware engine and limiter.
Linux-only tests mirror that change; both checks only run on Linux hosts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
morlutoandClaude Opus 5.5 bd3a9bbbaf fix(native): classify PE resource input failures
Report missing and non-file selections as input issues, malformed PE images as format errors, and unstable reads as artifact changes. Keep existing limit, permission, cancellation, and other I/O classifications.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
morlutoandClaude Opus 5.5 74050978be fix(javascript): keep dependency manifests out of application roots
Select the shallowest inventoried nondependency manifest and retain dependency package evidence without assigning Electron entry roles. Keep the artifact root when no application manifest is observed.

Adapted from PR #1392, with compiled CLI and stdio MCP regressions for root selection and dependency-only inputs.

Co-authored-by: Marnix Pluim <marnixpluim@users.noreply.github.com>

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:43:48 +08:00
b212e92f9b ci: select semantic tests and fixture lanes for their guarded inputs (#1675)
Edits to `release.yml` or `real-ghidra-windows.yml` selected only the
workflow lane, so the shards holding releaseWorkflow, releaseCheckpoint
and realGhidraWindowsWorkflow tests were skipped. Actionlint checks
syntax only, so a PR could drop the package-before-publish gate, the
registry canary or the default-branch-only self-hosted runner restriction
without CI noticing.

Conformance fixture sources under tests/ set only the code lane, so the
conformance-fixtures and reconstruction-readiness jobs, the only callers
of `verify:fixtures` and `verify:readiness`, never ran for them. Route
the directories that build-conformance-fixtures.mjs compiles to both
lanes, and readiness applications to the readiness lane.

Both were raised by the Codex review of #1660 after it merged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 19:41:40 -07:00
morluto 990e1ad8c1 fix(browser): clear losing storage verifier timers 2026-10-11 10:35:42 +08:00
morluto 76d8f68cb9 feat(ghidra): inspect NativeAOT PE metadata read-only 2026-10-11 10:31:49 +08:00
fe574c0c9f fix(javascript): recover Electron APIs through default imports and optional calls (#1673)
`import electron from "electron"` bound the module with a leading `default`
export segment, so `electron.ipcMain.handle(...)`, `new
electron.BrowserWindow(...)` and destructuring from that binding matched no
Electron API. Electron's CommonJS exports define no `default` member, so the
segment can only denote the module object; drop it when classifying
Electron bindings.

The binding classifier already recorded optional-call roots, but the IPC,
context-bridge, utility-process and sender-validation visitors accepted only
plain calls. `ipcMain?.handle(...)`, `contextBridge?.exposeInMainWorld(...)`
and the common `event.senderFrame?.url.startsWith(...)` check were dropped
without a limitation.

Refs #1656

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 10:31:08 +08:00
3a7a3074e3 feat(javascript): recognize tRPC procedures and GraphQL operations (#1670)
* feat(javascript): recognize tRPC procedures and GraphQL operations

Record RPC client syntax as network endpoint candidates in both
analyze_javascript_application and analyze_web_bundle, distinguished by
mechanism:

- trpc:procedure:<method> keeps the static member path of
  root.<path>.<method>(...), which tRPC proxies preserve through
  identifier minification. React hooks and TanStack option factories are
  recognized directly. Vanilla query/mutate need a root bound to
  createTRPCClient/createTRPCProxyClient, or a tRPC source marker with a
  router path of at least two segments; subscribe needs the marker and a
  tRPC observer argument. `this` roots, browser globals and RTK Query
  `endpoints` chains are never procedures; cache-key helpers are excluded.
- trpc🔗<link> records the literal url of documented HTTP links.
- graphql:<type> records named operations from string and cooked template
  literals with a valid operation header, and from precompiled
  OperationDefinition objects; a graphql-tag loc.source.body copy is not
  reported twice.

Method, callback and factory names follow @trpc/client and React/TanStack
integration sources and documentation for 11.19. Carrier segments before
the router path, useUtils helpers and minifier-renamed links remain
documented limitations.

Validation: typecheck; domain tests for recognition and false positives
(Apollo, RxJS, stores, RTK Query, browser globals, prose); web bundle
analyzer test; end-to-end analysis of a minified fixture through the
built MCP server on Windows 11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): preserve RPC binding identity and bundled calls

* style(reference-source): fix CI formatting

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 10:28:52 +08:00
morluto 83e52559b8 style(reference-source): format loader binding changes 2026-10-11 10:20:04 +08:00
morluto 824ecf109d feat(release): add contributor-aware release notes workflow 2026-10-11 10:19:16 +08:00
morlutoandLucien 8b4c3de3b3 fix(browser): seed storage by browser partition
Co-authored-by: Lucien <1428823446@qq.com>
2026-10-11 10:17:53 +08:00
morlutoandRudy Celekli f998fcb1cc fix(reference-source): respect require bindings
Reuse lexical ownership to exclude shadowed require calls while retaining native CommonJS and direct createRequire bindings. Preserve candidate edges when JavaScript or TypeScript runtime mode is unknown.

Co-authored-by: Rudy Celekli <rudy@gradiahq.com>
2026-10-11 10:10:59 +08:00
morluto f88c99021b fix(ghidra): normalize Mach-O imported symbol names 2026-10-11 09:59:27 +08:00
You Wishandmorluto 49e6f00ed0 fix(ghidra): recover returning import callers 2026-10-11 09:59:27 +08:00
af88184eee feat(ghidra): configurable import language and analysis seeds (#1667)
* feat(ghidra): configurable import language and analysis seeds

REA_GHIDRA_LANGUAGE_ID and REA_GHIDRA_COMPILER_SPEC_ID set the import
language when the header cannot identify the compiler (for example,
Borland C++ PE images).

REA_GHIDRA_SEED_FILE seeds functions, code and labels before
auto-analysis, for code reached only through computed jumps or data.

Both are validated before launch, recorded in the analysis profile and
verified in the bridge handshake.

* fix(ghidra): parse language XML and preserve seed ordering

* test(ghidra): use explicit platform paths for seed launch checks

* fix(javascript): retain cleanup failure reason in error causes

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 09:57:38 +08:00
c1efa6467d feat(process): bound a finalization interval before forced capture timeout
Add optional finalization_ms to keep observing a captured root after SIGTERM, record the actual finalizer exit and each signal attempt, and escalate once when needed. Bind the initial root identity to captured ownership and preserve observations through cleanup uncertainty.

Fixes #1253.

Co-authored-by: Mykola Dementiev <16836724+nikolay-dementiev@users.noreply.github.com>
Co-authored-by: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com>
2026-10-11 09:55:43 +08:00
morluto 1e51c732bb fix(javascript): preserve CommonJS assignment result and receiver uncertainty
Resolve values returned by simple assignment chains at the export boundary. Suppress property links proven to target a detached receiver; retain unresolved links when reassignment may preserve receiver identity. Identify root assignments from their AST target rather than the displayed export name.

Validation: 25 semantic tests, seven targeted compiled CLI/MCP scenarios with Node runtime controls, build, and check:fast passed. Existing full acceptance suite passed before receiver refinements; the final changes were checked with targeted scenarios rather than repeating its 130-second run.
2026-10-11 09:47:41 +08:00
morluto 7956934965 fix(javascript): preserve getter receiver mutation aliases
Resolve this, private fields and super reads through receiver-aware references. Separate instance fields from prototype descriptors, preserve alternative inheritance branches and allow finite repeated getter paths.

Add domain precision regressions and compiled CLI/MCP coverage for the four reported getter shapes.

Fixes #1665
2026-10-11 09:46:53 +08:00
morluto db4e360846 fix(browser)!: preserve WebSocket connections and comparable payload coverage
Retain source URLs, empty connections, and ordered frame streams. Compare same-URL streams without relying on transient request IDs, and compare selected shapes and byte counts only through common known coverage. Malformed binary frames retain known metadata with an unknown byte count.

BREAKING CHANGE: inspect_web_page network.websocket_events is replaced by network.websocket_connections, whose records contain request_id, url, and events. Frame payload_bytes is nullable when decoding cannot establish its size. No legacy alias is retained.

Validation: 78 domain/browser/MCP focused tests, check:fast, docs:check, and real Chrome 154 CDP capture passed. Real capture retained six connections and twelve frames with source URLs and selected shapes. Full browser verifier was interrupted during an unrelated slow scenario stage; its assertions were exercised by the bounded capture proof.
2026-10-11 09:43:25 +08:00
morluto 7bb4ac219c fix(ghidra): reject conflicting address aliases in batches 2026-10-11 09:35:22 +08:00
morluto 3df67c640a fix(ghidra): isolate optional nativeaot script sources 2026-10-11 09:35:22 +08:00
morluto 044aa34ce6 perf(native): index call-binding parameter and return nodes
Build ordered per-procedure indices from admitted nodes instead of scanning every node twice for each call binding. Preserve edge order, capacity limits, omitted-parameter unknowns, and surviving return bindings.

Validation: five focused service tests and check:fast passed. Four paired Flameox blocks at 4096 call sites passed all eight full-result digest oracles. Median paired reduction was 183.02 ms (71.4%) on this synthetic fan-out workload; real-provider performance is unverified.

Flameox evidence: 5a4c98ed7d224587c10a0f4596c8680d3a0330eb112a629014448ff1ea4634e7.
2026-10-11 09:33:36 +08:00
morluto 31bfff6fdf perf(domain): order JSON shapes through their structural tree
Traverse sorted siblings instead of comparing complete shared path prefixes. Return the typed producer result directly while preserving independent path segments and schema validation at unknown-input boundaries.

Validation: 12 domain/CDP boundary tests, build, and check:fast passed. Six paired Flameox blocks retained identical full-output digests across deep, mixed, Unicode, and wide controls. Depth-512 inference median fell from 10.78 ms to 1.51 ms (120 samples per variant); whole harness median fell from 518.6 ms to 327.5 ms.

Flameox evidence: f4a93424f34e5bfea254ada4ef278dc9bbb6c21047c977e20da130e11a8899e9. Timings describe this bounded workload only.
2026-10-11 09:30:05 +08:00
morluto 026126eafe fix(artifacts): serialize ASAR snapshot ownership
Serialize preparation and close for each reader so concurrent callers share one owned snapshot. Preserve per-call cancellation and expected digest checks, cleanup retry ownership, and reuse after close.

Validation: 34 ASAR boundary tests, one fork-isolated snapshot race regression, test discovery, and check:fast passed.
2026-10-11 09:28:28 +08:00
YsopwnableandGitHub 29cc1b8746 fix(javascript): exclude overwritten Electron API identities (#1664)
* fix(javascript): exclude overwritten Electron API identities

* test(javascript): bind Electron in file-base path fixture

* test(electron): bind BrowserWindow in preload override fixtures

* test(mcp): allow cold catalog validation in runtime probe

* test(electron): bind APIs in option reconstruction fixtures
2026-10-11 09:25:32 +08:00
LuisandGitHub ad41afa2b5 fix(setup): recognize Omarchy as an Arch-based host (#1662) 2026-10-11 09:23:39 +08:00
morluto 69a8b3a954 fix(ghidra): expose terminal-call boundary assumptions
Preserve observed terminal-call sites and callee flags across function views, raw bridge responses, Evidence, and snapshot replay. Version analysis profiles so older unqualified results cannot be reused.

Qualify Ghidra function extents without changing external no-return flags or claiming recovery of excluded code.

Validation: 76 focused tests, check:fast, docs:check, formatting, and the full Ghidra 12.1.4 host lane pass. Real CLI and all five stdio MCP function views retain the terminal-call evidence; returning puts calls remain unflagged. The reported stripped Rust ELF and Windows host remain unverified.

Fixes #1658
2026-10-11 09:08:01 +08:00
morluto 4733fc810a fix(ghidra): qualify defined-string search coverage
Carry defined-data scope through list and search Evidence and bind it into the analysis profile so older snapshots cannot replay unqualified results. Clarify the tool contract and document known-address byte/reference inspection.

Validation: 36 focused tests, check:fast, docs:check, and real Ghidra 12.1.4 CLI/stdio MCP on macOS arm64. The Rust Mach-O fixture contains the phrase and Ghidra defines it; the reported Linux ELF omission was not reproduced.

Fixes #1659
2026-10-11 08:51:02 +08:00
morluto d21132fc97 fix(ci): format Electron IPC verification assertion 2026-10-11 08:34:22 +08:00
morlutoandGitHub 66c344e289 ci: scope provider and native verification by changed inputs (#1660) 2026-10-11 08:31:27 +08:00
morluto 28bd0d990a test(electron): verify aliased APIs against a real IPC round trip
Extend the existing real Electron public-tool fixture with assignment aliases for BrowserWindow, contextBridge, ipcMain, and ipcRenderer. Assert exact static recovery counts and observe the handler invocation after a real click.

Verified the public CLI/stdio MCP fixture, IPC invocation, cancellation, and owned-process cleanup against checksum-verified Electron 43.3.0 on macOS ARM64. Build and check:fast pass. The separate readiness reload/crash scenario still fails on a destroyed execution context; this commit does not claim the full verify:electron lane passes.
2026-10-11 08:30:09 +08:00
morluto 2612e474b6 fix(javascript): recover Electron API identity through aliases
Reuse semantic module provenance for aliases and keep property path segments intact through Electron matching. Reject shadowed names and nested or dotted lookalikes. Reassigned and conditionally initialized module origins remain ambiguous, with their source origins retained.

Add a compiled CLI/stdio MCP regression for windows, preloads, context bridges, IPC operations, pairing, and source ranges. Validate 771 JavaScript domain tests, seven composition tests, the public acceptance workflow, build, and check:fast.

Fixes #1656
2026-10-11 08:27:46 +08:00
morluto a11fbc0df9 fix(ci): format JavaScript getter mutation regressions 2026-10-11 08:20:39 +08:00
morluto 639b73097b fix(javascript): preserve getter results on direct property writes
Replacing or deleting a class getter property does not mutate the object it would return. Apply the existing object-accessor write boundary to instance and static getters.

Validation: source mutation regressions and compiled CLI/stdio MCP acceptance preserve literal 1 for all four assignment/deletion cases; check:fast passes.
2026-10-11 08:15:47 +08:00
morluto bb074d84cb perf(javascript): reuse cycle sets during semantic evaluation
Track active bindings with per-evaluation push/pop instead of copying ancestor sets on every alias. Restore membership in finally so cycles and sibling branches keep their existing semantics.

Validation: 173 domain tests, compiled CLI/stdio MCP returned-reference acceptance, and check:fast pass. A randomized six-block Flameox experiment retained identical complete semantic output and reduced the bounded 112-alias analyzer median from 36.33 ms to 17.00 ms.
2026-10-11 08:14:45 +08:00
morluto 204e2c42a1 fix(ci): register Termux browser verifier as Knip entry point 2026-10-11 08:13:41 +08:00
tyloryangGitHubtyloryang <[邮箱]>morluto
87edf14940 fix(browser): support Termux Android scenarios (#1655)
* fix(browser): support Termux Android scenarios

* fix(build): lock Termux artifact generation

* ci(android): add manual Termux browser smoke verification

---------

Co-authored-by: tyloryang <[邮箱]>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 08:10:06 +08:00
YsopwnableandGitHub 890dfa49f6 fix(javascript): preserve cleanup failures during preparation cancellation (#1654) 2026-10-11 08:05:08 +08:00
morluto bc7846db91 fix(javascript): track references returned by getters
Follow object, instance, and static getter results when invalidating shared values. Respect descriptor replacement, known spread overrides, inheritance alternatives, and unresolved computed keys. Preserve conservative uncertainty for fresh getter results and verify shared mutations through CLI and MCP.

Fixes #1494
2026-10-11 08:03:17 +08:00
morluto 5485483f7d fix(javascript): preserve exact CommonJS export keys
Retain literal property keys as member segments, omit unresolved computed keys, and distinguish nested writes from dotted export names. Preserve empty keys and omit object-literal prototype setters. Verify bundler observations through Node, CLI, and MCP consumers.
2026-10-11 07:59:47 +08:00
morluto c916850852 fix(process): release terminals after observation failures
Keep queued observation failures available through frames while allowing the resource owner to release xterm and its owned addons. Catch write-callback failures so cleanup cannot hang, and retain actual disposal failures for retry.
2026-10-11 07:47:33 +08:00
morluto 26038912f6 fix(javascript): preserve cancellation during worker preparation 2026-10-11 07:36:39 +08:00
morluto aa7c5b7b5e fix(ghidra): reject address names owned by another symbol
Check function and label ownership in the existing namespace before renaming. Keep same-address renames idempotent and reject duplicate batch members inside their individual transactions. Extend real-provider namespace checks for single and batch collisions without conflating function and label fixture names.

Validation: real Ghidra 12.1.4 host-native verification passed on macOS ARM64 with JDK 21.0.12.1, including public CLI/stdIO MCP, qualified-name round trips, rollback, source-byte immutability, and complete cleanup. Windows mutation authority remains unchanged.

Fixes #1649
2026-10-11 07:35:48 +08:00
morluto 095fad9406 Merge remote-tracking branch 'origin/main' 2026-10-11 07:32:50 +08:00
morluto 8b2f8514c5 fix(javascript): ignore bare CommonJS exports rebinding
Assigning a new value to the exports alias does not replace module.exports. Stop inventing a default export link and definite return-shape changes for that assignment.

Validation: 22 semantic structure source tests and 49 compiled CLI/MCP acceptance tests passed. The new regression checks Node require returns an empty export object for both versions and both adapters report missing defaults without definite changes.
2026-10-11 07:30:52 +08:00
morluto c57e759ae6 fix(browser): bypass desktop keyrings in verification fixtures
Use password-store=basic for owned raw Chrome verification processes and document the same option for dedicated caller-owned Linux investigation profiles. Locked keyrings can stall navigation and leave placeholder frame URLs; keep the existing committed-URL authorization boundary.

Validation: real Chrome 154 CLI and stdio MCP browser verifier passed on macOS ARM64, plus docs freshness and check:fast. The locked Linux desktop keyring environment was not reproduced locally.
2026-10-11 07:30:25 +08:00
morluto 7b25a73ac4 fix(browser): bound CDP messages by the shared budget
Apply the 64 MiB protocol message limit when connect callers omit an override. Reject invalid explicit limits, including values that wrap through ws signed 32-bit coercion and disable enforcement. Preserve the typed payload_limit reason through disconnect and later commands.

Validation: 16 transport boundary tests passed, including UTF-8 exact/+1 limits and an oversized frame header without allocating its body. Real Chrome 154 CLI/MCP browser verification and check:fast passed. This caps message bytes, not decoded JSON heap size.
2026-10-11 07:30:14 +08:00
morluto aa257ea4b1 fix(browser): decode source-map response framing
Share the ECMA-426 JSON parse view across fetched maps and local point tracing while retaining exact response text, bytes, and digest. Handle BOM and HTTP newline variants without treating unterminated prefixes as valid JSON.

Validation: 89 source-map tests passed; source/runtime build and check:fast passed. Real Chrome 154 CLI and stdio MCP browser verification passed on macOS ARM64.
2026-10-11 07:29:51 +08:00
YsopwnableandGitHub 4999e4be2c fix(javascript): isolate source analysis and retain bounded partial evidence (#1651)
* refactor(javascript): validate transferable file projections

* test(javascript): consume decoded module and static facts

* fix(javascript): isolate source analysis and retain bounded partial evidence

* test(javascript): verify cleanup retention and repeated source paths
2026-10-11 07:22:50 +08:00
morluto 92cc0edd8e docs(setup): identify Qwen and OpenCode release gaps
Document the npm 6.3.0 boundary beside Qwen registration and OpenCode configuration override workflows. Registry metadata still identifies 6.3.0 as latest; source support requires publication and actual registry-package verification before issue closure.

Refs #1650.
2026-10-11 07:10:46 +08:00
morluto 573598c632 fix(artifacts): reject Windows stream paths before extraction
Validate Windows destination representability at the writer before registering or creating any entry. Preserve literal logical names in inventory and POSIX output. Refuse NTFS alternate stream syntax with typed path evidence and roll back prior extracted files.

Return producer path diagnostics directly, removing the presentation layer English-prefix classifier. Verify stream leaf/parent syntax, input stream cleanup, extraction rollback, inventory fidelity, POSIX materialization, and error projection: 63 passed, one platform-specific skip. Native Windows execution remains unverified here.

Fixes #1574. Adapted from and supersedes #1575.
2026-10-11 07:10:45 +08:00
morluto 8d3cfef753 fix(plist): disclose binary prototype-key omissions
Reuse the bounded binary preflight to observe dictionary keys before the pinned decoder drops them. Keyed archive results disclose the omission and Interface Builder marks archive decoding partial. XML keeps its existing exact counts; binary reporting deliberately leaves the count unknown because overwritten subtrees and decoder projection can change visibility.

Remove the superseded size-only preflight contract. Cover ASCII, UTF-16 and extended lengths, shared references, nested and shadowed dictionaries, and the value-only negative control. Synthetic plist and filesystem fixtures; no plutil or ibtool verification.

Fixes #1594. Supersedes #1593.
2026-10-11 07:02:39 +08:00
morluto 975dff2e25 fix(dotnet): bound RVA reads by section virtual size
Use the shared resolver for virtual containment, overlap checks, and file-backed extent. Raw alignment padding no longer supplies method bytes or shadows a later virtual section. Preserve malformed body coverage and suppress CIL hashes outside the admitted range.

Verify zero, exact-end, truncated-body, padding, and true-overlap cases through compiled CLI and MCP consumers. Synthetic PE fixtures only; no Windows loader or real managed-provider verification.

Fixes #1596. Supersedes #1595.
2026-10-11 07:02:16 +08:00
morluto 1b527f4898 fix(ci): share runtime fixtures and allow modest duplication headroom 2026-10-11 06:53:20 +08:00
YsopwnableandGitHub b09dd333ab fix(tests): separate Copilot model IDs from wire models (#1647)
* fix(tests): separate Copilot model IDs from wire models

* fix(tests): inspect parsed reference import diagnostics
2026-10-11 06:52:45 +08:00
morluto 60fbb2f06e fix(install): share arguments between the plan and npm execution
Include the Linux user prefix in the planned command. Build the install
arguments once and shell-escape the printed arguments so paths with spaces
or metacharacters remain reusable. Cover dry-run admission, exact argument
round-trip and the unprefixed macOS command with native Bash 3.2.

Refs: #1417
2026-10-11 06:41:51 +08:00
morlutoandFazmin 8b4c96e6c6 fix(cli): preserve native option descriptions in help
Keep the optional address description visible to incur and describe the actual
bundle plist default and absolute-path support. Extend the existing command
help sweep instead of traversing the inventory a second time.

Refs: #1418
Co-authored-by: Fazmin <fazmin.nizam@gmail.com>
2026-10-11 06:41:46 +08:00
morluto 95a77b6e25 fix(web)!: retain capture read cleanup through export
BREAKING CHANGE: WebScriptExportError accepts error context and residual paths in an options object.
2026-10-11 06:30:56 +08:00
morluto 67966c69a5 fix(setup): retain client configuration cleanup owners 2026-10-11 06:30:56 +08:00
morluto 279c942fe1 refactor(reference): consolidate Git metadata cleanup 2026-10-11 06:30:56 +08:00
morlutoandGitHub 54e2890b30 fix(ci): check decoded reference paths and restore formatting (#1648)
* style(browser): format cached SVG ancestry check

* fix(tests): assert decoded reference-source error paths
2026-10-11 06:29:26 +08:00
0f605bca09 fix(browser): report SVG xlink:href links in dom_urls (#1602)
* fix(browser): report SVG xlink:href links in dom_urls

DOM snapshots have no namespace URI, so SVG context is an svg ancestor
outside foreignObject. When href is missing, report xlink:href as href
with the same base URL and origin policy.

Fixes #1586

* fix(browser): cache SVG ancestry during DOM capture

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 06:23:07 +08:00
Immanuel TikhonovandGitHub 423b11c8ef fix(javascript): preserve trailing slash in package directory requests (#1615) 2026-10-11 06:23:05 +08:00
FengandGitHub 52331ab637 fix(setup): preserve configured MCP server environment (#1607)
* fix(setup): preserve configured MCP server environment

* fix(setup): validate retained server environment before writes
2026-10-11 06:23:02 +08:00
akushonkamenandGitHub 3e931a7e35 fix(browser): fold loopback spellings into one attach admission key (#1604)
A CDP endpoint serves every loopback stack on one port, but the attach
admission key was built from the endpoint origin string, so scenarios
attaching through http://127.0.0.1:PORT and http://[::1]:PORT produced
two lease keys, passed admission in parallel and the browser rejected
the second client. Collapse literal loopback hostnames to one identity
in the lease key (port and target still partition it) without touching
endpoint validation.

Validation: npx vitest run src/browser/PlaywrightScenarioBrowser.test.ts (9 passed; the two new loopback tests failed before the fix), npx vitest run src/browser src/domain (1745 passed, 2 failed, both failing on pristine main), npm run test:fast (3896 passed, 12 failed, 7 skipped; the failure set is identical on pristine main at 3894 passed), plus typecheck, oxlint, oxfmt and resolved module-boundary verification.
2026-10-11 06:22:59 +08:00
AdolaniumandGitHub 6edd5cf045 fix(browser): keep declared scenario secrets out of redacted evidence (#1590)
A redaction marker such as [REDACTED:password] can contain the secret it replaced, and the byte replacer never scanned the marker it inserted. Cookie names, domains, paths, and storage keys were also stored raw. Choose a marker that does not contain a declared spelling, drop the result when a spelling remains, and redact storage identity fields.
2026-10-11 06:22:57 +08:00
1a6ccec636 fix(browser): skip Windows Electron signals without verified lineage (#1588)
capture_electron_scenario called taskkill on the recorded leader PID before lineage could be verified. On Windows the process table is empty, so that proof never arrives and the signal still ran. Refuse the cleanup without signaling. A verified lineage still terminates the leader and its descendants.

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 06:22:54 +08:00
Luis ArayaandGitHub afc9d819aa test(android): preserve reason in JADX cleanup-failure teardown poll (#1639) 2026-10-11 06:22:51 +08:00
Luis ArayaandGitHub 7b3011b41e fix(tests): bind HOME in native client and deepseek verifiers (#1638) 2026-10-11 06:22:48 +08:00
6849ecf70a docs(installation): clarify that npm 6.3.0 excludes automatic pi and hermes setup (#1637)
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 06:22:37 +08:00
morluto 926c2d6ba7 test(ghidra): preserve metadata in interior-label fixture
Place the imported control label inside a valid multi-byte NOP instead of suppressing Mach-O function-start metadata. Encode the supported ARM64 and x86-64 instructions explicitly and retain the load dependency so the control remains past the function entry. Assert that Ghidra imports a label and no procedure at the control address before checking name rejection and address containment.

Two real arm64 Ghidra 12.1.4 imports confirmed both fixture strategies produce labels; only default linking retains LC_FUNCTION_STARTS. The full native-value CLI/MCP verifier passed with 349 successful and 52 expected rejected boundary calls, including imported aliases, complete exports, and startup cancellation cleanup. Host ARM64 and Apple x86-64 fixture compilation passed; x86 provider execution remains unverified.
2026-10-11 06:10:10 +08:00
morluto c7345512ea perf(native): index admitted value-trace nodes
Replace a graph-wide node scan for each parameter use with membership in the set of admitted node IDs. Preserve node-budget omission, pagination, and existing ID-collision behavior.

Flameox profiles identify the eliminated scan. Six-run medians on a bounded synthetic four-procedure workload improved from 2538.17 ms to 220.32 ms (11.52x); full JSON responses have identical SHA-256 4071fc7ec17be01e6a23d53814a47b3b1bb0ebd80099d21ef8431da487801725. This measures production trace composition with a synthetic provider, not real Ghidra target speed.

External design research through DeepWiki was checked against Ghidra 12.1.4 DataGraph: https://github.com/NationalSecurityAgency/ghidra/blob/8b6bbb857accdfa20dc5b2f5dea471178c2e9fbc/Ghidra/Features/CodeCompare/src/main/java/ghidra/features/codecompare/graphanalysis/DataGraph.java
2026-10-11 06:10:10 +08:00
morluto 53e7fadbd0 fix(reference)!: retain source and metadata cleanup owners
Require a caller-owned artifact resource scope for source traversal and Git metadata reads. Preserve cancellation, filesystem reasons, partial entries and bytes, and cleanup observations through import and the CLI. Transfer exact failed file owners at regular-file admission and close boundaries, including multiple failures encountered by Git reference resolution.

Validation: 169 focused tests passed across 26 files; two Windows-only cases skipped. Build, typecheck, lint, formatting, knip, architecture guards, discovery and docs checks passed.

BREAKING CHANGE: Reference source readers and imports require an ArtifactResourceScope. VCS metadata reads return Result, and CLI import failures include observed details and partial evidence.
2026-10-11 06:10:10 +08:00
morluto 215001a1ce refactor(filesystem): share directory cleanup ownership
Reuse the existing directory close state machine for artifact and reference readers. ArtifactResourceScope can retain the exact directory owner when cleanup fails.
2026-10-11 06:10:10 +08:00
YsopwnableandGitHub 9515d2fcd3 fix(tests): verify Qwen skills through an isolated native home (#1646) 2026-10-11 06:05:57 +08:00
YsopwnableandGitHub ac9677bd6c fix(test): preserve Ghidra interior-label controls on Mach-O (#1640)
* fix(test): preserve Ghidra interior-label controls on Mach-O

* test: preserve selector semantics and accept signaled fixture exits

* test(javascript): exercise deep returns through the public CLI
2026-10-11 05:45:00 +08:00
morluto 69fe5b6010 perf(javascript): index structured observations by path
Build package, JSON-module and source-map indexes once instead of rescanning every observation list per file. Preserve first-observation lookup semantics when paths repeat and share each JSON lookup with the file projection.

Validation: duplicate-observation consumer regression and reconstruction suites. Flameox captured before/after CPU profiles on 3,000 JSON files; the complete CLI output stayed byte-identical. Exploratory samples do not establish an end-to-end timing improvement.
2026-10-11 05:33:21 +08:00
morluto 684199ba0b fix(javascript)!: retain nested artifact readers in caller scopes
Require the caller artifact resource scope when reading JavaScript artifacts. Transfer failed nested ASAR cleanup to that scope so later work retries the same owner before admission. Preserve traversal failures and attempt every remaining iterator during unwind.

Validation: reconstruction, integrity, content identity, cancellation and nested-reader cleanup boundary checks.

BREAKING CHANGE: readJavaScriptArtifactFiles now requires ArtifactResourceScope as its third argument; the optional abort signal is fourth.
2026-10-11 05:33:15 +08:00
morluto a17e257f43 fix(artifacts): preserve failures while unwinding nested traversals
Finish every remaining iterator from inner to outer, retaining cleanup observations without replacing the original traversal failure. A real nested ASAR regression reproduces the previous integrity-error masking.
2026-10-11 05:33:14 +08:00
morluto 5502ca9c97 fix(artifacts): retain reader owners after cleanup failures
Keep directory handles under explicit close ownership and retain ZIP and Mach-O admission owners when initialization or cleanup fails. Preserve cancellation and primary failure details, and avoid false successful release after native handle invalidation.

Validation: filesystem ownership, FIFO, archive and reconstruction boundary checks; static and architecture checks; real macOS CLI/MCP bundle workflows. Windows native behavior was not verified.
2026-10-11 05:33:05 +08:00
morluto be75ed5e4b fix(publication)!: retain cleanup owners across export workflows
Require an ArtifactResourceScope for web script exports and share its lifetime with CLI or MCP shutdown. Retain setup-failed output trees in firmware and Wakaru providers. Preserve completed publications when descriptor cleanup fails, returning the full result after a successful retry or as a partial observation while cleanup remains unresolved.
2026-10-11 05:17:10 +08:00
morluto 3f8c51e9b5 fix(artifacts): retain output ownership through publication failures 2026-10-11 05:17:10 +08:00
morluto e0e49e9dd5 fix(artifacts)!: keep streamed reads within descriptor ownership
Use bounded positional reads for unpacked ASAR members and borrowed Mach-O slices so destroying a stream cannot bypass its descriptor owner. Retain uncertain ASAR closes and preserve borrowed sources after cancellation.

BREAKING CHANGE: Injected ASAR unpacked handles expose fd and read instead of createReadStream.
2026-10-11 05:17:10 +08:00
morluto be6272f261 fix(artifacts)!: preserve descriptor cleanup authority
Keep rejected closes unresolved after descriptor invalidation and share concurrent close attempts. Preserve primary admission failures and transfer the exact failed handle owner into inventory scopes.

BREAKING CHANGE: ArtifactResourceScope file-handle owners require OwnedFileHandle. Root inventory sources retain their single cleanup owner.
2026-10-11 05:17:10 +08:00
clydeandGitHub 32d13c431e test(javascript): retain resource ownership in deep-return fixture (#1644) 2026-10-11 05:16:03 +08:00
morluto 0fa0b04628 fix(electron): check Windows lineage before process cleanup
Return an unverified cleanup failure before invoking numeric process-tree termination when owned lineage is unavailable. Preserve the partial capture and existing cleanup diagnostics.

Exercise the provider capture boundary with stable application-state observations; assert no Windows cleanup call and retained partial Evidence. Source tests pass; native Windows behavior was not verified.
2026-10-11 04:43:02 +08:00
morluto ffe0574e17 fix(artifacts)!: retain cleanup owners through client shutdown
Keep failed reader, root descriptor, and output-tree cleanup with the caller. Drain admitted work before close, retry the same retained owners, and block new resource work while cleanup is incomplete. Preserve primary failures and complete inventory observations.

Thread ownership through JavaScript reconstruction, the CLI's one-shot scope, MCP server shutdown, verifier scripts, and test fixtures. Remove redundant ownership flags and the unused validated test wrapper.

Validate with 850 passing focused tests across 90 suites (two skipped), targeted classification and cleanup retries, static and unused-code checks, and real macOS CLI/MCP directory, ZIP, and DMG workflows. Native Windows and deep-analysis providers were not verified.

BREAKING CHANGE: Direct artifact inventory/extraction and JavaScript analysis APIs require an explicit caller-owned ArtifactResourceScope. CLI and MCP input contracts are unchanged.
2026-10-11 04:43:02 +08:00
morluto 87d5ec93d5 perf(javascript): index native Evidence links by observed digest
Build each side's digest-to-Evidence index once instead of scanning native observations for every comparison item. Retain every exact match and avoid linking subjectless Evidence to nodes lacking content identity.

Comparison and relationship regressions pass. Isolated alternating unprofiled runs at 2,000 nodes and observations per side have conventional medians 379.924 ms before and 246.466 ms after; the complete 4,000-item outputs match byte-for-byte. Timings are bounded workload evidence, not whole-suite performance.

Flameox profile: flameox://evidence/7dca714833dc3d254a75704670a81de656b4b31d4b77a72236eecb82459b3d86
2026-10-11 04:43:02 +08:00
morluto f1c58cc53f fix(reference): reject FIFO replacements without blocking
Use the shared nonblocking, no-follow regular-file opener at the source-read boundary and report a nonregular replacement as changed. Keep descriptor identity and extent checks.

The compiled subprocess regression timed out on the baseline and completes without a FIFO writer after this change. Reference reader failure, capacity, cancellation, and import checks pass.
2026-10-11 04:43:02 +08:00
morluto 6636045c9c fix(reference): match exclusions against observed entry kinds
Inspect each entry without following links before applying directory-only patterns. Retain the winning rule and original entry path while pruning excluded directories before reading their children.

Validate directory exclusions, negation, symlinks, explicit secret rules, and compiled CLI provenance alongside reference reader failure and cancellation coverage.
2026-10-11 04:43:02 +08:00
YsopwnableandGitHub b74d9a1bb2 fix(onboarding): isolate client homes and clarify npm release support (#1627)
Bind HOME and USERPROFILE in disposable native client verifiers, exercise default personal skill discovery, and clarify automatic Pi/Hermes setup availability in the checked npm release.

Validation: integrated build/static/docs checks, repeated native client workflows, real Codex-backed client trials and two independent cold-npx Codex installs. All 19 CI checks passed on a7ba07ba. Other model providers, platforms and deep-analysis engines retain their explicit verification TODOs.
2026-10-11 04:37:31 +08:00
6e0d5d64d3 feat(ghidra): admit set_address_name and set_addresses_names (#1635)
* feat(ghidra): admit set_address_name and set_addresses_names

Ghidra sessions rejected the existing naming tools with "operation is not declared by this provider", so agents had to use annotate_native_function, which returns a complete function dossier per rename and cannot name data addresses.

The Java bridge now names a local function entry (namespace-aware, like annotate_native_function) or creates/renames the primary label at any other mapped address, verifying readback. set_addresses_names applies each address in its own transaction and reports per-address booleans. Both are mutating capabilities: excluded from read-only (Windows P0) handshakes and from snapshot replay.

* fix(ghidra): normalize qualified symbol names and reject empty leaves

Share namespace-aware leaf normalization across functions and primary labels. Invalid qualified leaves now return false for naming operations without marking the provider unavailable; function annotations retain typed invalid-input errors.

Extend real CLI/MCP namespace verification with namespaced globals, readback replay, failed-name rollback, mixed batches, and capability checks.

Validation: 75 focused tests and npm run check pass. Ghidra 12.1.4/JDK 21 namespace verification passes on macOS arm64 with unchanged source bytes and complete cleanup. The full host lane passes this workflow before the existing entry-alias fixture fails because the compiler places its interior label at the function entry.

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-11 04:35:38 +08:00
YsopwnableandGitHub d6f698d8af fix(ghidra): admit carved MIPS ELF with incomplete section tables (#1629)
* fix(ghidra): inspect carved MIPS ABI declarations without missing sections

* test: share filesystem fixtures and cleanup assertions
2026-10-11 04:33:05 +08:00
morluto bbd01923e9 fix(artifacts): own DMG readers before attachment
Make inventory reader construction synchronous and DMG attachment lazy, so the inventory registers its reader before native operations start. Share initialization, reject work after close, wait for an in-flight attach before cleanup, and preserve retryable close state.

Cover failed-attach inventory cleanup, concurrent initialization, close-before-use, and close-during-attach alongside existing ownership, cancellation, and integrity cases. Session-level retention after inventory cleanup failure remains a separate concern.
2026-10-11 04:15:26 +08:00
morluto 5a77a962ec perf(reference): size read buffers from admitted file lengths
Use the descriptor size to bound each allocation, with a one-byte EOF probe, and stop on observed growth. Retain short-read, cancellation, and final identity validation.

A 2,001-file benchmark requests 4,200,307 bytes instead of 266,469,376 with the same captured-byte digest. Flameox shows filesystem waits dominate, so this is an allocation reduction without a CPU speedup claim.
2026-10-11 04:15:26 +08:00
morluto 6585293c56 fix(reference)!: preserve exclusion rule provenance
Record the exact matching pattern and distinguish project, default, caller, and configured-sensitive exclusions without changing ignore ordering. Include that provenance in graph commitments and validate the contract through compiled CLI and MCP consumers.

BREAKING CHANGE: Pattern-based reference graph exclusions require pattern and distinguish project-ignored and default-ignored reasons. Re-import saved graphs that omit matching patterns.
2026-10-11 04:15:26 +08:00
YsopwnableandGitHub be44b5098e test(javascript): share semantic graph fixtures (#1631) 2026-10-11 04:05:35 +08:00
morluto 6db97d5162 perf(javascript): remove duplicate workflow Evidence snapshots
Pass typed workflow results directly to createEvidence, which validates generic JSON and takes the owned snapshot. Remove the extra generic parse and full result clone from all five workflow helpers; domain producers retain their specific schema validation.

Flameox confirmed JSON validation and copying costs. Alternating warm runs creating Evidence for a valid 2,001-node trace reduced median boundary time from 15.94 ms to 11.20 ms over 120 samples per variant, with identical output digests and independent mutable-input snapshots. This does not measure end-to-end workflow latency.

Validation: 39 focused CLI, SDK MCP and filesystem workflow tests and 17 Evidence tests passed; build, static and documentation checks passed. No deep analysis providers were exercised for this change.
2026-10-11 03:56:59 +08:00
morluto 39da0b150b fix(artifacts): bind member reads to emitted metadata
Reject Mach-O and ZIP entries whose metadata no longer matches the reader inventory, preventing a sibling adapter key from selecting bytes under another entry. Reuse the existing ASAR metadata snapshot and equality rules and read Mach-O ranges from owned records.

Move the real system-lipo test to the provider boundary suite, release its reader, and derive slice expectations from the host FAT table.

Validation: 68 focused artifact, ASAR and MCP tests plus ZIP cancellation passed. Real macOS lipo was exercised against /bin/ls; Ghidra, IDA and other deep providers were not run. Static, docs, architecture and test discovery checks passed.
2026-10-11 03:53:50 +08:00
morluto 3967cfa43e fix(reference): distinguish root access failures from invalid input
Share root filesystem failure classification between import preflight and traversal. Missing roots remain invalid selections; permission and I/O errors retain their reason and reach the CLI as execution failures. Preserve cancellation priority and expose unexpected exceptions.

Validation: compiled CLI permission regression, direct reader error details, import/traversal and unreadable-directory boundaries passed. Windows-only cases were skipped on macOS; static, architecture and documentation checks passed.
2026-10-11 03:53:50 +08:00
morluto 2014ee7d68 perf(javascript): compare scalar semantic query seeds directly
Avoid repeatedly canonicalizing scalar seeds for every literal node. Valid literal metadata without a scalar value now remains unmatched instead of throwing.

Flameox identified canonicalization as the hotspot. Alternating warm query benchmarks over 2,002 nodes reduced the 35.5 KB seed median from 36.34 ms to 0.158 ms; a short seed improved from 0.304 ms to 0.128 ms with identical result digests. These measurements cover the query function, not end-to-end analysis.

Validation: 38 domain query/traversal tests and focused compiled CLI/MCP consumers; static, architecture and documentation checks passed.
2026-10-11 03:53:50 +08:00
YsopwnableandGitHub 5100a1a1a5 test(clients): verify Copilot compact schema workflows (#1624) 2026-10-11 03:43:24 +08:00
morluto de0d341524 fix(artifacts): retain unpacked ASAR handles after close failures
Own unpacked FileHandles until closure succeeds, deduplicate in-flight closure, and retain cleanup resources for retry without replacing the original read or identity failure. Attempt handle and snapshot cleanup independently and retain failed snapshots.

Consolidate unpacked admission, identity checking, cancellation and stream setup, removing the redundant same-handle stat. Cover stat and identity failures, post-stat cancellation, combined cleanup failure/retry, and stream completion/cancellation with real file descriptors.

Validation: 32 ASAR boundary tests, artifact extraction and SDK MCP consumers, check:fast. Native Windows descriptor inspection and real-provider workflows were not verified.
2026-10-11 03:36:36 +08:00
morluto d1f04512af refactor(javascript): remove redundant semantic query passes
Use resolved seeds directly, retain one list of relevant unknowns, and stop sorting adjacency used only for complete reachability. Preserve canonical graph-array output order and candidate coverage scans.

Validation: 37 query and traversal tests, public JavaScript reconstruction and invalid-Evidence consumers, check:fast. No real-browser or deep-provider workflow verified.
2026-10-11 03:36:35 +08:00
morluto 514df0918c fix(analysis-view): preserve inline Evidence error paths
Project authenticated inline Evidence failures at source.evidence while retaining their reasons and messages. Supply the matching request root to input issue projection so nested values remain identifiable.

Validation: both service entrypoints and SDK MCP integrity regressions, advertised-schema consumer suite, check:fast and architecture guards. No real-provider verification.
2026-10-11 03:36:35 +08:00
morluto 8fb79c67ee perf(evidence): remove disposable ledger work from standalone import
Count already-validated records and unknown revisions without reimporting, sealing, and exporting a temporary empty ledger. Keep canonical integrity checks at the file boundary and verify counts and malformed bundles through the compiled CLI.

Validation: Evidence file and CLI consumer suites; check:fast; Flameox CPU profiles and alternating 3.8 MB import benchmark (315.8 ms to 102.5 ms warm median, identical result digests). No real-provider workflow changed or verified.
2026-10-11 03:36:35 +08:00
clydeandGitHub 6a41a29963 fix(apple): preserve directory requirements in dylib paths (#1623) 2026-10-11 03:30:27 +08:00
YsopwnableandGitHub c12a532b9c perf(evm): avoid repeated cleanup diagnostics (#1622)
Retain the original owned-command cleanup failure and one captured-output copy while preserving cleanup resources, truncation and retry semantics.

Verify socket release after cancelled CDP cleanup while retaining the independent detach regression.

Validation: 167 tests across 13 files, all eight local gates, real small Chrome/EVM CLI and pinned-SDK MCP controls, and 20/20 exact-head CI checks.
2026-10-11 03:28:16 +08:00
YsopwnableandGitHub b28c0337fc perf(javascript): reduce retained semantic analysis allocations (#1601)
Reduce retained semantic allocations and memoize lexical scope ancestry while preserving semantic identities. Refs #1065 and #1563; the complete-directory memory failure remains open.
2026-10-11 03:20:05 +08:00
morluto f4e437e544 refactor(artifacts): remove unused extraction inventory indexes
Derive needed artifact IDs directly from selected occurrences. Keep the node lookup local to selection, drop the unread occurrence map and collector helpers, and pass only manifest and selected integrity metadata to materialization.

Validation: existing extraction completeness, identity, unsupported-format, packed/unpacked ASAR, and public MCP workflows pass. Nested archive selection and equal-content occurrence coverage remain intact. This is a structural simplification; no throughput improvement is claimed.
2026-10-11 03:19:29 +08:00
morluto 8ba0a887ff refactor(javascript): bind application links only at graph finish
Remove the empty application-node index from incremental projection state. The completed application graph now owns the only binding pass, and indexing returns the map directly instead of a wrapper.

Validation: 76 builder/composition tests and 19 filesystem, CLI-output, and SDK MCP trace boundary tests pass. Identity budgets and streaming/full-projection parity remain covered; no throughput improvement is claimed.
2026-10-11 03:19:29 +08:00
morluto f9ea9a9268 fix(process)!: bound terminal buffers before capture
Validate columns × (rows + scrollback) against a 1,000,000-cell xterm working-buffer budget, including every resize with the selected scrollback. Document the budget in the advertised MCP input and process-capture guide; never clamp caller dimensions.

Validation: default/exact-boundary schema cases, prelaunch MCP rejection, compiled CLI capture, terminal rendering, normalization, cancellation, and lifecycle checks pass on macOS. This is resource admission, not a measured serializer throughput improvement; Linux-only and real deep-analysis providers were not exercised.

BREAKING CHANGE: process scenarios whose initial or resized terminal exceeds 1,000,000 combined visible-plus-scrollback cells are now rejected before launch.
2026-10-11 03:19:29 +08:00
morluto 78778d91a4 fix(artifacts): retain output identity through cleanup retries
Track created directory and file identities before materializing or removing output. Refuse displaced paths and untracked entries, preserve completed cleanup work across permission failures, and prevent publication once rollback starts. Group cleanup files once per attempt instead of rescanning them for every directory.

Validation: root, nested-parent, and file symlink replacement; untracked-entry retention; real POSIX permission failure and retry; extraction integrity and CLI/MCP consumers pass. Path identity revalidation cannot eliminate syscall-boundary pathname races in portable Node. Windows behavior was simulated; native Windows and Linux-only publication providers were not exercised.
2026-10-11 03:19:29 +08:00
morluto d510f3e928 perf(browser): share JSON shape paths across repeated array rows
Accumulate observations once per structural path instead of allocating and serializing a path for every payload value. Remove the CDP body-shape wrapper and retain primitive, empty-array, and literal-property semantics.

Validation: CDP capture/comparison regressions and CLI/MCP boundary checks pass; 1,507 differential cases match the previous implementation. A 2.5 MB repeated-row workload improves from 105.0 ms to 14.6 ms warm median across alternating runs. Real browser providers were not exercised.
2026-10-11 03:19:29 +08:00
YsopwnableandGitHub 043fd2579d fix(firmware): recover after workspace removal failures (#1616) 2026-10-11 03:16:34 +08:00
YsopwnableandGitHub a761981897 test: remove module and timestamp assumptions from boundary fixtures (#1621) 2026-10-11 03:15:44 +08:00
YsopwnableandGitHub 2653b194b6 feat(skills)!: improve Codex onboarding and task recovery (#1613)
Guide task-centered recovery from observed client, registration, PATH and
provider blockers. Install the recovery reference, document exact JavaScript
module selectors, and require clarification before searching for an unnamed
app. Validate public setup in isolated Codex accounts and preserve failed
or unrecognized tool outcomes in the existing evaluator.

Refs: #1587

BREAKING CHANGE: Optional verify:agent reports now use schemaVersion 4.
Consumers must accept expectedFirstTool: null and targetClarificationPassed
for missing-target scenarios. Production CLI/MCP contracts are unchanged.
2026-10-11 03:13:56 +08:00
YsopwnableandGitHub 16d9c5adef fix(ci): preserve cleanup boundaries and bounded failure assertions (#1618)
Interrupt optional CDP domain shutdown on caller cancellation while preserving an independent bounded detach attempt. Retain original overflow and cleanup failures, verify workspace recovery, preserve user skill files, and reject another unverified firmware worker.

Validation: 149 integrated tests; repeated 64 affected boundary tests; full local check. Standalone CI includes real Chrome CLI/stdio MCP and real Web runtime checks.

Refs #1613
2026-10-11 02:54:12 +08:00
morluto b582b6ecb4 fix(managed): enforce module identity before joining token facts
Validate declared artifact and MVID scopes at managed Evidence admission, and reject conflicting known module declarations across graph inputs. Join boundary tokens to members only under matching known MVID scopes; preserve facts and report partial coverage when scope is unknown.

Prefer an observed MVID-bearing declaration with its supplying Evidence. Normalize GUID casing for module node identity while preserving original strings in observations.

Validation: 285 tests across 34 focused suites passed, followed by the final 12-test managed graph suite and managed CLI/MCP regressions. check:fast and architecture guards passed; no real native providers were run.
2026-10-11 02:52:11 +08:00
morluto ba056e08a4 fix(providers): bound launcher diagnostics without losing byte counts
Give Ghidra and Hopper an 8 MiB aggregate retained-output budget while continuing to drain their process pipes. Expose total observed bytes at the supervisor boundary and retained bytes plus explicit truncation in provider diagnostics.

Redact known transport tokens even when the retained prefix ends inside one, preserving adjacent caller path/query evidence. Existing partial-launch and cleanup ownership remain intact.

Validation: 285 tests across 34 focused suites passed; the final cutoff-fixture refinement passed all 31 Ghidra/Hopper diagnostic tests. check:fast and architecture guards passed. Provider fixtures use real child processes; real Ghidra/Hopper applications were not verified.
2026-10-11 02:50:06 +08:00
morluto cb13a27bfa perf(artifacts): reuse the admitted ASAR header size
Use the header size already validated against the owned archive snapshot instead of asking the ASAR SDK to read and allocate its raw header again.

Validation: ASAR freshness, containment, streaming-allocation, and MCP extraction suites passed within the 285-test focused run. check:fast and architecture guards passed.
2026-10-11 02:49:05 +08:00
morluto 3d79903fb1 perf(browser): normalize precise coverage once by script identity
Pass validated coverage into normalization and index retained sources by script ID instead of scanning them for every script. Preserve source revalidation, UTF-16 bounds, nested counts, and unavailable-source uncertainty.

Validation: 285 tests across 34 focused suites, check:fast, and architecture guards passed. Flameox-guided 8,000-script normalization benchmark improved from 101.6 ms to 5.5 ms median with identical full-result digests; this is normalization CPU, not end-to-end browser timing.
2026-10-11 02:48:59 +08:00
morluto dce81ca400 perf(javascript): count candidate relevance without a second adjacency
Scan eligible candidate relations directly for query ambiguity instead of allocating and sorting another traversal index. Preserve direction, relation filters and cycle handling.

Semantic graph and builder suites plus public application workflow CLI tests passed. Full query outputs matched on a generated 20000-node graph; the single repeated-query timing run is indicative only.
2026-10-11 02:32:11 +08:00
morluto bb764d26bd perf(artifacts): stop parsing parent paths after a missing branch
Walk delimiter-terminated parent segments lazily instead of splitting full paths before discovering that no deeper observed parent exists. Preserve canonical paths and observed ancestry.

Verified real ZIP inventory with multiple 30000-component siblings and the full entry-order suite. A 96-member deep ZIP benchmark retained an identical full inventory digest and reduced median scan time from 517.5 to 464.5 ms across five measured inventories.
2026-10-11 02:32:04 +08:00
morluto acd8d86898 perf(android): validate prerequisites once per retained JADX session
Resolve current paths and JVM options and rehash source identities for each request. Probe Java modules and engine capabilities only when creating a new session; remove the superseded full configuration resolver.

Focused configuration, session retirement, cancellation, cleanup and Android MCP boundaries passed. The 34-operation retained session now performs one prerequisite probe. Protocol fixtures do not establish real JADX or JDK performance.
2026-10-11 02:31:57 +08:00
morluto e6d0b3993d perf(process): batch liveness rechecks within ownership phases
Recheck unreadable token and identity observations against one fresh live process snapshot per phase. Preserve token mismatches, fail closed on inspection errors, and retain liveness failure diagnostics.

Validated with focused process ownership, native capture, CLI and MCP boundaries. An eight-member injected-unreadable workload using native ps reduced snapshots from nine to two with equivalent failures; real permission-denied provider behavior was not exercised.
2026-10-11 02:31:07 +08:00
morluto 92a901e0db perf(process): share settlement probes across captured groups 2026-10-11 02:21:50 +08:00
morluto cef6d6d3c4 perf(artifacts): reuse trie membership for case-folded siblings 2026-10-11 02:21:40 +08:00
morluto bd972bf80b perf(javascript): reuse capture completeness during load classification 2026-10-11 02:21:35 +08:00
morluto ac9709a80a perf(reconstruction): index semantic evidence contexts per derivation 2026-10-11 02:21:28 +08:00
morluto d1e6fe2eca test(artifacts): close owned Mach-O readers after each case 2026-10-11 02:11:13 +08:00
morluto a147ecd4b4 perf(artifacts): index observed parents for deep archive paths 2026-10-11 02:10:50 +08:00
morluto 22fdcb8ab7 perf(process): share batched token observation across ownership checks 2026-10-11 02:10:44 +08:00
morluto b22d817073 perf(process): batch ownership tokens during rooted cleanup
Use the host bulk token reader for each cleanup validation while preserving identity checks before and after token reads. Fall back for unsupported readers or omitted rows, retain explicit unavailable observations, and preserve batch and individual failures. Source ownership regressions and real owned-process boundary tests pass; a warm Darwin validation probe reduced median time from 90 ms to 52 ms.
2026-10-11 01:59:30 +08:00
morluto 3d876c5c1a perf(javascript): validate semantic factory inputs once
Construct typed semantic records directly from cloned, normalized input and derived commitments instead of reparsing the same fields. Keep full graph integrity validation and verify caller mutation isolation. Four alternating cold runs preserved complete result digests while median projection time fell about 8 percent and 17 percent for the two 20,000-node capacity workloads.
2026-10-11 01:54:46 +08:00
morluto cf346368c4 perf(javascript): reuse graph buckets and query indexes
Append overlapping observations to node buckets instead of recopying groups, and share one node index across semantic seed admission and traversal. Preserve merged provenance and absent-seed behavior.
2026-10-11 01:54:36 +08:00
morluto e52c0a72c5 fix(browser): retain incomplete WebMCP replay coverage
Mark malformed WebMCP event envelopes, tool lists and entries as invalid protocol values while preserving valid siblings and out-of-scope classifications. Verify toolsAdded and toolsRemoved replay uncertainty through CDP discovery and capture comparison.
2026-10-11 01:54:26 +08:00
morluto 999e3e2098 fix(android): cancel admitted snapshot copies with their operation
Forward caller and timeout cancellation through engine and APK snapshots, release stream handles before removing partial output, and retain the existing root cleanup authority. Verify cancellation after destination bytes appear with a sparse filesystem fixture.
2026-10-11 01:54:20 +08:00
dakjdakdandYsopwnable c1217e711b fix(javascript): retain runtime scope in bundled module relationships 2026-10-11 01:36:12 +08:00
morluto 0d89978333 fix(android): release snapshot streams before closing file handles
Retain both non-auto-closing copy streams and destroy them after verification, before awaiting their FileHandle closure. This releases stream-held descriptor references on success and failure instead of deadlocking Android snapshot cleanup.

Validation: all 33 JADX subprocess integration tests passed on macOS with Node 24.18.0; check:fast and formatting passed. No real JADX provider execution or Linux/Windows verification was performed locally.
2026-10-11 01:12:48 +08:00
morluto 11e3fae697 fix(ci): repair boundary regressions and evidence assertions
Detach cancelled CDP sessions within the cleanup budget and retain event-finalization failures across retries. Honor Git filesystem encodings and exclude the promise-API probe from source-read failures.

Align existing integration regressions with retained process resources, filesystem-order-independent truncation, owned skill files, explicit native access facts and streamed JSON errors.

Validation: 132 acceptance tests passed (2 skipped), 83 affected boundary tests passed, Git/JSON/CLI boundary regressions passed, check:ci, docs:check and test discovery passed. Local verification used Node 24.18.0 on macOS; cross-platform CI remains pending.
2026-10-11 01:00:30 +08:00
morluto 603d04ee3a fix(ci): repair format gate and pin maintenance skill destinations
format:check failed on ApktoolProvider; existingMaintenanceScope now
returns skillDestinations (consumed by maintenance planning), so pin
the field in the exact-shape scope assertions.
2026-10-11 00:33:36 +08:00
morluto 472f313232 refactor(native): match UI controls by explicit object identity 2026-10-11 00:28:39 +08:00
morluto 1f7e279aa2 fix(adb): retain rollback ownership and cancelled acquisition facts 2026-10-11 00:28:39 +08:00
morluto db8ccaa251 fix(android): bind JADX snapshots and hashes to admitted sources 2026-10-11 00:28:25 +08:00
morluto 5daff4d67e fix(artifacts): retain Mach-O source identity through inventory 2026-10-11 00:28:25 +08:00
morluto a9fd1aecae fix(application): verify configuration and reference input reads 2026-10-11 00:28:24 +08:00
morluto abb8e81d3a fix(filesystem): bound and verify admitted file and JSON reads 2026-10-11 00:28:24 +08:00
YsopwnableandGitHub 6d8f95e38d fix(clients): preserve portable MCP schemas in native workflows (#1580)
* test(clients): verify native Command Code workflows

* docs: synchronize Italian provider guide links

* fix(contracts): make ADB identifier patterns portable

* fix(contracts): preserve portable ADB patterns after formatting
2026-10-11 00:09:43 +08:00
morluto c1b5ccf0a4 refactor(server): admit complete evidence workflows and share handlers
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:24 +08:00
morluto 5f30ea6ff2 fix(engines): serialize JEB acquisition and respect IDA cancellation
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:23 +08:00
morluto d438675677 fix(process): retain capture resources and bound filesystem snapshots
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:23 +08:00
morluto a84dea5500 fix(apktool): join active work and retain cleanup authority
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:22 +08:00
morluto 2a66ec9094 fix(browser): retain startup owners and coordinate retryable close
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:22 +08:00
morluto d1235fadc9 fix(providers): retain worker cleanup owners through shutdown
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:21 +08:00
morluto a01148553e fix(setup): preserve recovery backups and retain installer cleanup
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:21 +08:00
morluto a6f76f5da9 fix(graphs): separate source identities from display representations
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:20 +08:00
morluto 2ca3ad2386 fix(reference): preserve exact paths and unify symlink lookup
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:20 +08:00
morluto 5d0a883605 fix(native): bound metadata projection and compact Mach-O bind runs
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:19 +08:00
morluto 0f0d6e4d43 fix(managed): budget decoding before representation expansion
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:19 +08:00
morluto 6d4cefc4f7 fix(artifacts): admit bounded reads and preserve source identity
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:18 +08:00
morluto 2a0b58e440 refactor(cleanup): share failure composition and retain native UI owners
Validation: integrated changes passed formatting, typecheck, lint, resolved module-boundary verification and git diff --check. Tests and real-provider workflows were not run.
2026-10-11 00:03:18 +08:00
867 changed files with 70434 additions and 9393 deletions
@@ -0,0 +1,134 @@
---
name: rea-changelog-update
description: Prepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.
---
# REA changelog update
Use this for release-note preparation and contributor attribution. Read
[docs/releasing.md](../../../docs/releasing.md) from the repository root for
publication authority and the existing checkpoint checks. This skill prepares
notes; it does not authorize merging, tagging, or publishing.
## Inventory before prose
Fetch complete history and tags. Select the published baseline tag from
`.release-please-manifest.json` on the release PR's **base branch**, and pin that
branch's exact SHA as the editorial target. REA tags are `rea-agents-VERSION`.
Use the proposed version, including its prerelease suffix; do not change the
`always-bump-minor` strategy to imitate OpenClaw's calendar versions.
```bash
node scripts/release-notes.mjs inventory \
--repository morluto/rea --base rea-agents-PREVIOUS \
--target FULL_BASE_BRANCH_SHA --version VERSION \
--manifest /tmp/rea-VERSION.contributions.json
node scripts/release-notes.mjs record \
--manifest /tmp/rea-VERSION.contributions.json \
--summary \
--output /tmp/rea-VERSION.record.md
```
Read the inventory, not just the existing bot draft. It includes side-branch
commits, GitHub-associated PRs whose merge commits are in the range, commit
authors/co-authors, and confirmed closing issues with their reporters. GitHub
connections are paginated. Network errors stop generation; rerun into a new
output file after recovery. Existing output files are never overwritten.
Unresolved commit identities are retained as names/emails with `user: null`;
never guess a handle or thank the merger as a substitute. The report preserves
unknowns. This is a reviewed metadata snapshot, not independent proof that a
human authored a patch. For unattributed identities, unusual merge strategies,
backports, reversions, or references that GitHub did not associate, inspect
the source PRs and diffs and report the limitation. The current collector uses
confirmed PR-closing relationships for issue-reporter credit; unrelated issue
mentions are not automatically credited. Do not claim that merely inventoried
work still ships after a revert.
## Write for users and retain credit
Rewrite only the newest `## [VERSION]` section in `CHANGELOG.md`, preserving its
version/date/link heading. Use single-line Markdown list items under:
- `### Highlights`: a few release-defining user outcomes; no quota for small releases.
- `### Changes`: capabilities and meaningful behavior changes.
- `### Fixes`: reliability, correctness, compatibility, and safety fixes.
- `### ⚠ BREAKING CHANGES`: retain when applicable, with migrations and original
PR/commit references required by the existing checkpoint validator.
Group related work by analyst workflow or user impact. Explain the resulting
behavior in short, concrete entries. Highlights summarize the release; avoid
repeating their full descriptions in Changes. Internal cleanup, tests, CI, and
routine documentation generally belong only in the provenance file. Read ambiguous PRs/diffs before making
claims. Distinguish simulated-provider checks from real-provider verification.
Credit each represented contribution once in the public notes with its link
and `Thanks @...` for its verified human author, known co-authors, and confirmed
closing-issue reporters. Grouping preserves all those credits in one line.
When Highlights, detailed entries or migrations reference the same work,
put its thanks in one of those entries, not all of them. Include issue links.
Direct-commit bullets carry the commit link and known human credit.
Exclude GitHub bots and known agent accounts. REA's observed Anthropic and
Cursor co-author trailers resolve to `@claude` and `@cursoragent` with GitHub
type `User`; the credit policy excludes them while retaining their metadata.
At the REA maintainer's request, omit `@morluto` self-thanks while preserving
authorship in the inventory. Other human contributors remain eligible.
Use explicit GitHub links for cross-repository references to avoid ambiguous
`#NNN` numbers. Unknown handles stay unknown; they are not fabricated credit.
Append the generated `record --summary` provenance link verbatim at the end
of this release. The linked inventory accounts for every discovered in-range
PR and direct contribution, including internal work omitted from the prose.
Do not paste the full `record` output into public notes: it duplicates thanks
and obscures the user-facing changes. Full records remain available for review.
Do not hand-edit generated credits to make validation pass. Refresh the inventory
when product history changes; a prose-only correction can reuse the snapshot.
## Finalize with Release Please
Before changing the live release PR, have authorization for that PR update.
Local preparation and validation can proceed independently.
1. Add `rea:release-notes-finalized` to the selected open release PR **before**
editing its branch/body. Main proposal pushes respect this label. This is
our workflow guard, not Release Please's `autorelease: snooze` mechanism.
A proposal already running must finish before editing. Do not dispatch
manual preparation over finalized notes.
2. On that release branch, save the reviewed inventory as
`docs/releases/VERSION.contributions.json`. This is durable public release
provenance, not a temporary planning file. Do not add credentials or local
environment dumps. Keep the existing version artifacts aligned.
3. Run the existing checkpoint validator and the new checker:
```bash
node scripts/release-notes.mjs check \
--manifest docs/releases/VERSION.contributions.json
node scripts/release-notes.mjs render \
--manifest docs/releases/VERSION.contributions.json \
--output /tmp/rea-VERSION.notes.md
```
The checker supports inline PR bodies, including their surrounding metadata,
up to GitHub's 65,536-character limit. It fails on larger notes rather than
truncating credit; Release Please's overflow-link bodies are not integrated.
4. Update the release PR body with the rendered section while preserving its
Release Please header/footer and component/version structure. Use a body
file for `gh pr edit --body-file`; changing only `CHANGELOG.md` does not
change Release Please's release body. Push only authorized changes.
5. Freshly fetch the PR body to a file and run `check --pr-body FILE`. Review
the complete diff and wait for current CI. Publication checks the body
again before release creation. Avoid further body edits while publishing.
The finalization label freezes bot refreshes, not main. New product changes
after the inventory target require refreshing the notes before publication.
The workflow checks target ancestry and allows only release metadata paths
after that target. It also requires the inventory target to equal the reviewed
merge's first parent, so intervening main commits require a fresh inventory.
Review diffs within the allowed release metadata paths too: the path guard does
not prove that package dependency/script edits are version-only changes.
To resume automatic proposals, remove the finalization label and let the bot
regenerate; expect editorial changes to be replaced.
Existing releases without a contribution inventory retain their current
publication path. Never retag or republish a package to correct release prose.
@@ -2,7 +2,7 @@
name: reverse-engineer-anything
description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.
metadata:
version: "36"
version: "37"
---
# REA
@@ -25,57 +25,18 @@ a skill installed from repository main may describe capabilities absent from
an older npm release. Keep complete inline Evidence when that server does not
advertise retained references.
When tools are absent or registration is stale:
If tools are absent, a call is blocked by the client, or a required provider is
unavailable, read [connection and recovery](references/connection-and-recovery.md).
Diagnose the observed blocker, continue useful supported work, and resume the
original task after repair. Client approval, shell PATH, registration and engine
availability have different remedies; avoid repeating setup for all of them.
1. Diagnose without changing files. For Codex, run
`npx -y rea-agents@latest doctor --client codex --json`. Substitute the current
supported client: `claude_code`, `claude_desktop`, `codex`, `cursor`,
`gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`, `copilot_cli`,
`commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`, `hermes`, or `grok_bot`. `grok_bot` has no local
file registration: doctor reports the Grok Bot chat step, and setup does not
write `mcp.json`. If the client is unknown, use `doctor --json` and
inspect its registration results before choosing a setup scope.
2. Distinguish the reason. Missing, malformed, or stale registration needs a
scoped configuration repair. An aligned registration with no tools in the
active session needs a restart/reconnection; doctor cannot prove that the
current agent has connected. A missing provider affects only tasks requiring
that provider: static JavaScript inspection needs neither Hopper nor Ghidra,
and Android inspection has separate bring-your-own JADX/Java prerequisites.
3. For a configuration repair, prepare the read-only plan:
`npx -y rea-agents@latest setup --client codex --dry-run --json`.
Use the current client's ID, show its exact proposed paths, backups, and
changes, and obtain approval before setup writes configuration or installs
Hopper. Setup normally installs the matching bundled skill too; include that
replacement in the reviewed plan. After approval, apply the same scope with
`npx -y rea-agents@latest setup --client codex --yes`. Add `--install-hopper`
only if that separate installation was needed and explicitly approved.
4. Restart/reconnect the affected agent. Verify that REA tools actually appear
in the session, then resume the original investigation. If they remain
absent, inspect the client's MCP launch error rather than repeating setup.
REA setup never installs or upgrades Node.js, npm, Homebrew, Java, Ghidra, IDA,
JADX, Binwalk, or Unblob. Use existing prerequisites; do not install unrelated
software to repair
MCP registration. For an unsupported client, use manual stdio registration
with a version-pinned `rea-agents` package or continue through the CLI.
A concrete CLI fallback for an operator-supplied JavaScript tree or ASAR is:
```bash
npx -y rea-agents@latest analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
npx -y rea-agents@latest inspect-analysis-view app-view.json
```
No MCP registration or native engine is required. The CLI returns the complete
Evidence record directly, and a real application can produce hundreds of
megabytes, so save it to a file rather than reading it into context. Start from
the summary view, then request a module page
(`{"kind": "page", "collection": "modules", "offset": 0, "limit": 32}`) or one
module the same way, or pass the saved Evidence inline to
`trace-application-feature`. Read graph, limitations, and unknowns with the
same care as an MCP result. This fallback does not establish that MCP is
configured. Native CLI tasks still require their selected engine.
Prefer an existing `rea` CLI. When none is installed and REA package acquisition
is within the authorized scope, use `npx -y rea-agents@latest` in its place.
Configuration repair starts with a read-only setup plan; show paths, backups and
skill replacement before applying authorized changes. Preserve existing approval
and target-execution boundaries. REA setup never installs or upgrades unrelated
software, including Node.js, npm, Homebrew, Java, Ghidra, IDA, JADX, Binwalk or Unblob.
## Route the target first
@@ -140,9 +101,11 @@ after opening; IDA does not supply `binary_overview`. Consult session availabili
when composing broader workflows. Modern direct callers and unsupported dossier
facets remain unknown; live IDA results are not replayed from snapshots.
If the app is missing, ask which app to inspect. Resolve a human-readable app
name to one clear installed artifact when possible; ask only when matches are
ambiguous. Never choose an example app on the user's behalf.
If the user says only "my app" without a name, path, or target already selected
for this task, ask which app before searching for artifacts. Paths in unrelated
saved reports and nearby test fixtures do not select the user's target. Resolve
an actual human-readable app name to one clear installed artifact when possible;
ask when matches are ambiguous. Never choose an example app on the user's behalf.
In a target-free session, use `open_binary` to bind any archive/package or
native target whose analysis tool operates on the active target. Do not call an
@@ -157,7 +120,11 @@ For a JavaScript application or ELF layout whose size is unknown or large,
call `analyze_javascript_application` or `inspect_binary_layout` with
`"detail": "summary"`, then inspect the retained `parent_evidence_id` with
`inspect_analysis_view`.
Do not repeat an identical tool call. Make a focused follow-up only when the
Page module identities and select the returned `node_id` before requesting an
item; a single file path can represent several graph roles. Do not repeat an
identical analysis or rejected call without a changed input or resolved blocker.
Recheck session state when a lifecycle change makes that status useful.
Make a focused follow-up only when the
returned result leaves a specific question unanswered.
If MCP reports `resource_constraint` with `details.resource: "transport"`, call
@@ -168,6 +135,24 @@ session to a selected path. The analysis remains complete in the ledger; avoid
repeating it merely to request the same oversized response. Complete CLI JSON
output also streams without a single MCP frame.
When the retained parent ID is unknown, use `get_evidence_bundle` with
`detail: "summary"` and exact filters for operation, target SHA-256,
analysis-profile digest or canonical procedure address. Select its returned
Evidence ID for the next view instead of requesting the complete bundle.
`retention: "complete-record"` means the whole record is stored; native
value-flow truncation and unknown metadata remain separate. For an exported
portable bundle, use `rea inspect-evidence-bundle` with JSON containing `path`,
`detail: "summary"` and the same `filters`. Reading that file still requires
validating its full contents within the local memory budget.
A client can also truncate a successful tool result without a REA transport
error. Treat that preview as incomplete; use the retained parent ID for smaller
views or export when the full record is needed. Do not parse the preview as
canonical JSON or repeat the producer. For JavaScript, read the
[module-view and trace recipes](references/javascript-applications.md) before
drilling into a large graph; a broad bidirectional trace can expand unrelated
containment relationships even after summary analysis.
Every conclusion must distinguish observations, inferences, and unknowns. Cite
Evidence IDs, preserve limitations and incomplete coverage, and never imply
that static analysis observed execution. Runtime requests execute the declared
@@ -175,7 +160,9 @@ target and lifecycle; do not broaden the target or action beyond those fields.
Within the user's requested investigation, call available analysis tools
directly. REA does not require permission grants or per-call approval flags.
Follow the declared request scope and the host's actual access requirements.
The agent client can independently require approval. Follow the declared request
scope and the host's actual access requirements; use the recovery guide for an
observed client-policy rejection.
## Plan broader investigations
@@ -0,0 +1,127 @@
# Connection and task recovery
Read this guide only for an observed connection, command, approval or provider
blocker. Keep the user's target and question in view. Explain what prevented the
operation and the next useful step; after repair, continue the investigation.
Commands below use an installed `rea`. If none is available, use the package
invocation in `SKILL.md` when acquiring REA is authorized. Follow the connected
server's schemas and the installed CLI's help rather than assuming repository
main and an older release expose the same arguments.
## Choose the remedy from the observed failure
| Observation | Meaning and next step |
| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No REA tools in the active session | Inspect registration with `rea doctor --client codex --json`. A skill installation alone does not register MCP. |
| Missing, malformed or stale registration | Prepare a scoped setup plan; apply only authorized changes with backups. |
| Aligned registration, but no connected tools | Restart/reconnect the client. Doctor checks registration; it cannot establish the current agent's connection. Inspect the client's launch error if reconnection fails. |
| `MCP tool call requires approval, but approval policy is never` | Codex rejected the call before REA executed it. Do not install an engine or repeat setup to solve this client-policy decision. See the approval section below. |
| REA returns `provider_unavailable` or an unavailable tool entry | Read that operation's provider diagnostics or `binary_session.result.tool_availability`. Resolve only the prerequisite needed for the task; a complete tool catalog does not imply every operation is available. |
| `rea` is missing in a child shell, or Node is incompatible | Inspect the actual shell PATH and launcher/runtime. A prefix installation may be absent from a login shell's PATH; this does not establish a broken MCP registration. |
If the client cannot launch its shell/sandbox helper or read the installed skill,
inspect its launch diagnostic. For example, Codex can refuse helper aliases when
CODEX_HOME is under `/tmp`; use an authorized profile outside the OS temporary
directory. REA setup cannot repair the client's executor. Preserve the sandbox
instead of silently disabling it to make an analysis command work.
Static JavaScript and ordinary ZIP inventory need neither Hopper nor Ghidra.
HAR inspection is offline and does not require a live browser. Native
decompilation, Android analysis and firmware extraction have their own
bring-your-own prerequisites. Use supported inventory/static evidence when it
answers part of the question, and name the unavailable evidence separately.
Do not substitute another target. Label observations from ordinary shell tools
as such; only actual REA output supplies REA Evidence.
## Repair registration when needed
For Codex:
```bash
rea doctor --client codex --json
rea setup --client codex --dry-run --json
```
Use the current supported client's ID: `claude_code`, `claude_desktop`, `codex`,
`cursor`, `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`,
`copilot_cli`, `commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`,
`hermes`, or `grok_bot`. If unknown, inspect `doctor --json` before choosing a
scope. `grok_bot` uses a chat registration step rather than local `mcp.json`.
For unsupported clients, use version-pinned manual stdio registration or CLI.
Show the plan's exact paths, backups, changes and bundled-skill replacement.
Obtain approval for writes when that concrete scope is not already authorized,
then apply the same scope with `rea setup --client codex --yes`. Add
`--install-hopper` only when separately needed and explicitly approved. Do not
install or upgrade unrelated prerequisites. Restart/reconnect, verify tools
actually appear, and resume the original task. If they remain absent, inspect
the launch failure rather than looping through setup.
## Codex tool approval is separate from REA readiness
REA tools have no permission-grant handshake. Codex can still require approval
because an operation creates owned resources, launches a helper or has cleanup
effects. An advertised tool or successful doctor result does not override that
client policy. `workspace-write` alone does not resolve an MCP approval
rejection, and `approval_policy = "never"` does not mean every MCP tool is
preapproved.
When interactive approval is available, use the client's approval flow for the
requested operation. In an unattended run, continue through an authorized CLI
workflow if the execution environment permits it, or report the blocked
operation and the approval needed to proceed. Do not retry the same rejected
call, silently broaden approvals, or rewrite MCP effects as read-only.
For an operator who explicitly wants to preapprove named operations in a
disposable Codex profile, current Codex supports per-tool configuration. For
example, opening and closing an analysis session can be scoped independently:
```toml
[mcp_servers.rea.tools.open_binary]
approval_mode = "approve"
[mcp_servers.rea.tools.close_binary]
approval_mode = "approve"
```
Use the actual server ID and only the operations whose effects were approved;
this example does not approve inspection, extraction or runtime execution.
Show the proposed config edit and preserve unrelated entries and backups before
writing. Managed policy can constrain client settings. Consult the
[Codex configuration reference](https://learn.chatgpt.com/docs/config-file/config-reference)
for the installed client's supported settings. This is an optional recovery
choice, not a prerequisite for every REA task.
## Continue through an installed CLI
Check the command environment only if the launcher fails:
```bash
command -v rea
command -v node
node --version
rea --version
```
Use the public launcher at its known installation path with a supported existing
Node on PATH when the child shell loses a prefix installation. Avoid internal
`dist/cli.js` entrypoints or global toolchain changes. MCP may have its own valid
launcher even when a shell cannot resolve `rea`.
For a user-supplied JavaScript tree or ASAR, select output files in the authorized
workspace and save complete Evidence before reading a view:
```bash
rea analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
rea inspect-analysis-view app-view.json
```
These commands do not register MCP or install an analysis engine. CLI JavaScript
JSON is complete and can be very large; redirect it rather than loading it into
context. Use an existing JSON processor if `jq` is absent, without installing
one just for this recipe. Inspect a module page or item using the same inline
source and the [JavaScript recipes](javascript-applications.md). Preserve graph
coverage, limitations and unknowns. Native CLI analysis still requires its
selected engine, and all CLI operations retain the host's execution permissions.
@@ -38,3 +38,11 @@ or absence:
Process captures are opt-in behavioral evidence, not a security sandbox. V3
captures cannot be upgraded to V4; rerun the original scenario. Distinguish root
exit from descendant settlement and require freshness when the task needs it.
A timed-out target that must write a final report needs `finalization_ms`: REA
sends SIGTERM to the captured root, keeps observing, and sends SIGKILL once
only if the target has not exited by the end of the interval.
`exit.finalization.signals` lists each attempt with its delivery result; the
observed exit signal stays in `exit.signal`, `exit.code` is null for every
deadline exit, and the code of a finalizer that exited on its own is in
`exit.finalization.exit_code` (null on a signal exit or when no exit was
observed).
@@ -1,20 +1,106 @@
# JavaScript and Electron application artifacts
Use `analyze_javascript_application` directly on the operator-supplied ASAR or
extracted tree. The complete result, graph, and Evidence context are returned
inline when they fit the MCP frame. Source-map contents are part of the static analysis. Start from its
findings, coverage, unknowns, and graph context, then make a focused follow-up
only when a specific question remains unanswered.
extracted tree. When size is unknown or large and the connected schema advertises
`detail`, request `detail: "summary"`. This returns an `inspect_analysis_view`
Evidence record with a distinct ID and a `normalized_result.parent_evidence_id`
for the complete retained application analysis. Source-map contents are part of
that static analysis. Read findings, coverage, limitations and unknowns, then
follow a specific unanswered question. Older servers without `detail` return
the complete application Evidence inline when it fits.
When the connected schema advertises `max_heap_mb` and `analysis_timeout_ms`,
these select the isolated source analyzer's old-space heap and per-source
deadline. Defaults are 1024 MiB and 300000 ms. They do not change the MCP
process heap. Resource-limited analyses preserve completed static facts and
report partial semantic coverage with specific unknown scopes; this does not
mean the source is malformed. Inspect those limits before retrying a source.
When expanded relationships exceed the parent's available memory, inspect the
file's modules item view for retained `static_analysis` and
`semantic_module_analysis`. Their facts remain available even when graph links
are unknown. Cumulative result exhaustion can stop analysis before every file
is examined; the inventory and remaining unknown scopes distinguish that case.
Timeout or cancellation after source analysis begins retains partial Evidence
in the same session. A returned `error.details.partial_observation` reference
can be supplied directly to `inspect_analysis_view`. If the SDK cancels locally, retrieve its recorded ID from
the Evidence bundle before inspecting the partial analysis.
## Read a module without repeating analysis
Use the actual returned parent ID, on the same connection, as `source.evidence_id`.
The following IDs are placeholders to replace with returned values. When the
module's node ID is not already known, page identities before selecting an item.
Pages return only node IDs, paths and roles; choose a useful page size and follow
`coverage.next_offset` when another page matters:
```json
{
"source": {
"kind": "retained-evidence",
"evidence_id": "ev_ACTUAL_PARENT_ID"
},
"view": { "kind": "page", "collection": "modules", "offset": 0, "limit": 16 }
}
```
Select the relevant page entry by its exact `node_id` and recorded role:
```json
{
"source": {
"kind": "retained-evidence",
"evidence_id": "ev_ACTUAL_PARENT_ID"
},
"view": {
"kind": "item",
"collection": "modules",
"selector": { "node_id": "jag_node_ACTUAL_NODE_ID" }
}
}
```
Module pages include asset, bundled and source module roles, so one path can
identify several entries. Use `selector.path` only when uniqueness is established;
the returned node ID avoids needing a complete inventory to prove uniqueness. On ambiguity,
select the matching node ID from the page or the error's candidates. Follow
`next_offset` when another page matters. An item exposes that module node's
recorded observations, identity, hashes, exports and source ranges; it does not
include every child syntax observation, related edge or original source file.
An absent item fact is not evidence of absence from the whole application.
## Trace only the remaining question
Use `trace_application_feature` on the parent application Evidence for one
literal node ID, route, string, API, IPC channel, module, or native export.
Select `incoming` for relationships leading to the seed, `outgoing` for those
leaving it, and `both` when the question needs both. Trace traversal includes
containment relationships, so even a narrow seed can expand many nodes; a
bidirectional trace can reach unrelated sibling modules. A module item or page
is preferable when its observations already answer the question. Do not treat
all reached nodes as participants in the feature.
Reuse retained parent Evidence when the server advertises it, or supply complete
Evidence inline. Include Hopper or Ghidra Evidence only when its artifact digest
matches exactly.
## Recover an oversized display
An oversized response reports `resource_constraint` and
`details.resource: "transport"`. Reuse
`details.reported_limits.evidence_reference` with `inspect_analysis_view` for a
summary, one module, or a stable page of module identities. Use
`trace_application_feature` once a module, route, or string seed is known.
summary, one module, or a stable page of module identities.
`export_evidence_bundle` writes the complete canonical session to a
caller-selected file. These workflows keep the original Evidence and coverage;
a broad follow-up can also exceed framing.
A client can truncate a successful result inside its text envelope without a
REA `resource_constraint`. Treat a truncation notice or incomplete JSON as a
display limitation; do not claim the full record was examined. Reuse the known
parent ID for smaller views, or export complete Evidence to a selected file
while the session remains open. If the parent ID was lost from the preview,
use the prior analysis summary or explicit ledger entries to recover it; do
not invent an ID or rerun analysis just to reproduce the same display.
BrowserWindow preferences, preload and contextBridge surfaces, IPC
registrations, utility processes, and native binding requests are static syntax
observations. Only a unique exact literal IPC channel match supports an inferred
@@ -22,13 +108,6 @@ pairing. Dynamic or ambiguous channels remain unresolved. A requested `.node`
member is not a verified native export. Never claim runtime reachability,
registration, defaults, or policy enforcement from static analysis.
Use `trace_application_feature` on existing application Evidence for one
literal node ID, route, string, API, IPC channel, module, or native export.
Choose a direction and reuse the returned application Evidence ID when the
connected server advertises retained references, or supply complete Evidence
inline. Include
Hopper or Ghidra Evidence only when its artifact digest matches exactly.
For version comparison, analyze each version once, then call
`compare_application_versions`. Accept only its digest, source-map, structural
fingerprint, or non-module semantic matches. Module ordinals and minified names
@@ -152,7 +152,7 @@ context. PSP/stack/device state remains unmodeled. See [DOS guide](https://githu
### Function annotations in Ghidra
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations.
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier.
## Offline PE resource inventory
+6
View File
@@ -0,0 +1,6 @@
self-hosted-runner:
labels:
- hopper
- ghidra-12-1-2
# Hosted label introduced after the pinned actionlint release.
- macos-15-intel
+188 -81
View File
@@ -5,6 +5,10 @@ on:
branches: [main]
pull_request:
branches: [main, "release/*"]
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ci-${{ github.event.pull_request.number || github.ref }}
@@ -36,37 +40,39 @@ jobs:
name: Classify changes
runs-on: ubuntu-latest
outputs:
code: ${{ steps.package.outputs.required }}
package: ${{ steps.package.outputs.required }}
code: ${{ steps.plan.outputs.code }}
runtime: ${{ steps.plan.outputs.runtime }}
package: ${{ steps.plan.outputs.package }}
apple: ${{ steps.plan.outputs.apple }}
fixtures: ${{ steps.plan.outputs.fixtures }}
readiness: ${{ steps.plan.outputs.readiness }}
docs: ${{ steps.plan.outputs.docs }}
workflows: ${{ steps.plan.outputs.workflows }}
browser: ${{ steps.plan.outputs.browser }}
binary_layout: ${{ steps.plan.outputs.binary_layout }}
evm: ${{ steps.plan.outputs.evm }}
javascript_recovery: ${{ steps.plan.outputs.javascript_recovery }}
recorded_crash: ${{ steps.plan.outputs.recorded_crash }}
web_network: ${{ steps.plan.outputs.web_network }}
web_runtime: ${{ steps.plan.outputs.web_runtime }}
web_source_map: ${{ steps.plan.outputs.web_source_map }}
nativeaot: ${{ steps.plan.outputs.nativeaot }}
website: ${{ steps.plan.outputs.website }}
matrix: ${{ steps.plan.outputs.matrix }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Select package and Windows lanes
id: package
shell: bash
- uses: actions/setup-node@v4
with:
node-version-file: .nvmrc
- name: Select verification lanes
id: plan
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
if [[ "${GITHUB_EVENT_NAME}" != "pull_request" ]]; then
echo "required=true" >> "${GITHUB_OUTPUT}"
# Compare the PR's changes, excluding unrelated updates on its base.
elif git diff --quiet "${BASE_SHA}...${HEAD_SHA}" -- \
. \
':(exclude)docs/**' \
':(top,exclude,glob)README*.md' \
':(exclude)AGENTS.md' \
':(exclude)CONTRIBUTING.md'; then
echo "required=false" >> "${GITHUB_OUTPUT}"
else
diff_status=$?
# A failed comparison must fail classification, not report a scope.
if [[ "${diff_status}" -ne 1 ]]; then
exit "${diff_status}"
fi
echo "required=true" >> "${GITHUB_OUTPUT}"
fi
CI_FULL: ${{ contains(github.event.pull_request.labels.*.name, 'ci:full') }}
run: node scripts/ci/plan.mjs
static:
name: Static checks
@@ -84,17 +90,17 @@ jobs:
node-version-file: .nvmrc
cache: npm
- run: npm ci
- if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
- if: needs.changes.outputs.code == 'true'
run: npm run check:ci
- if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
- if: needs.changes.outputs.code == 'true'
run: npm run verify:test-discovery
- if: github.event_name == 'pull_request' && needs.changes.outputs.code != 'true'
- if: needs.changes.outputs.code != 'true'
run: npm run format:check
test-shard:
name: Test shard (${{ matrix.shard }})
needs: [changes, build]
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-latest
strategy:
fail-fast: false
@@ -109,14 +115,8 @@ jobs:
- run: npm ci
- uses: actions/download-artifact@v4
with:
name: rea-dist
path: dist
- name: Generate test artifacts from the downloaded runtime
run: |
node scripts/generate-mcp-tool-catalog.mjs
node scripts/generate-skill-metadata.mjs
node scripts/generate-product-catalog.mjs
node scripts/generate-completion-ledger.mjs
name: rea-test-runtime
path: .
- run: npm run test:ci:shard:run -- --shard=${{ matrix.shard }}
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
@@ -131,7 +131,7 @@ jobs:
build:
name: Build package
needs: changes
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -140,11 +140,17 @@ jobs:
node-version-file: .nvmrc
cache: npm
- run: npm ci
- run: npm run build:cached
- run: npm run test:prepare
- uses: actions/upload-artifact@v4
with:
name: rea-dist
path: dist
name: rea-test-runtime
path: |
dist/
skills/
.cache/mcp-tool-catalog.json
docs/public/product-catalog.json
docs/verification/managed-conformance-*.json
include-hidden-files: true
if-no-files-found: error
retention-days: 7
@@ -224,7 +230,7 @@ jobs:
conformance-fixtures:
name: Source-built conformance fixtures
needs: changes
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
if: needs.changes.outputs.fixtures == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
@@ -235,7 +241,7 @@ jobs:
windows-native-build:
name: Build bundled Windows x64 native controls
needs: changes
if: github.event_name == 'push' || needs.changes.outputs.package == 'true'
if: needs.changes.outputs.runtime == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
@@ -265,24 +271,14 @@ jobs:
package-e2e:
name: Native package baseline (${{ matrix.platform }} ${{ matrix.architecture }})
needs: [changes, windows-native-build]
if: github.event_name == 'push' || needs.changes.outputs.package == 'true'
if: needs.changes.outputs.runtime == 'true'
strategy:
fail-fast: false
max-parallel: 2
matrix:
include:
- os: ubuntu-latest
platform: linux
architecture: x64
- os: ubuntu-24.04-arm
platform: linux
architecture: arm64
- os: macos-15
platform: darwin
architecture: arm64
- os: macos-15-intel
platform: darwin
architecture: x64
# The four platform jobs use independent hosted runners; keep each
# runner's existing heap and threadpool limits while avoiding a queued
# second platform wave.
max-parallel: 4
matrix: ${{ fromJSON(needs.changes.outputs.matrix) }}
runs-on: ${{ matrix.os }}
timeout-minutes: 20
env:
@@ -317,7 +313,7 @@ jobs:
apple-artifacts:
name: Real Apple artifact CLI/MCP E2E
needs: changes
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
if: needs.changes.outputs.apple == 'true'
runs-on: macos-15
timeout-minutes: 15
steps:
@@ -327,30 +323,39 @@ jobs:
node-version-file: .nvmrc
cache: npm
- run: npm ci
- run: npm run verify:keyed-archive
- run: npm run verify:asset-catalog
- run: npm run verify:macos-bundle
- name: Legacy simulator platform verification
run: npm run verify:legacy-simulator
- name: Native installer and compiled NIB hierarchy regressions
run: npm run test:focused -- tests/boundary/filesystem/installScript.test.ts tests/boundary/filesystem/nibNativeHierarchy.test.ts tests/boundary/filesystem/interfaceBuilderBundle.test.ts
- name: Real binary metadata and Interface Builder integration
run: npm run verify:apple-dispatch && npm run verify:interface-builder
- name: Native plist CLI selection regressions
run: npm run test:focused -- tests/boundary/cli/nativePlistCli.test.ts tests/boundary/cli/nativePlistPrototypeKeyCli.test.ts tests/boundary/cli/nativePlistSelectionFailureCli.test.ts
- name: Native Swift demangling CLI and MCP regressions
run: npm run test:focused -- tests/boundary/cli/nativeDemangleCli.test.ts tests/boundary/mcp/nativeDemangleMcp.test.ts
- name: Native signature target binding and CLI/MCP recovery regressions
run: npm run test:focused -- tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts tests/boundary/cli/nativeSignatureCli.test.ts tests/boundary/mcp/nativeSignatureTargetBindingMcp.test.ts
- name: Process filesystem snapshot descriptor regressions
run: npm run test:focused -- src/process/capture/FilesystemSnapshot.test.ts
- name: Native Darwin ownership and process cleanup regressions
run: npm run test:focused -- tests/boundary/process/darwinProcessRunTokenReader.test.ts tests/boundary/process/directAnalysisSignals.test.ts tests/boundary/process/processCaptureLifecycle.test.ts tests/boundary/process/processCli.test.ts tests/boundary/mcp/sessionMcp.test.ts
- name: Verify Apple artifact formats and metadata
run: |
npm run verify:keyed-archive
npm run verify:asset-catalog
npm run verify:macos-bundle
npm run verify:legacy-simulator
npm run verify:apple-dispatch
npm run verify:interface-builder
- name: Run native Apple and process boundary regressions
run: >-
npm run test:focused --
tests/boundary/filesystem/installScript.test.ts
tests/boundary/filesystem/nibNativeHierarchy.test.ts
tests/boundary/filesystem/interfaceBuilderBundle.test.ts
tests/boundary/cli/nativePlistCli.test.ts
tests/boundary/cli/nativePlistPrototypeKeyCli.test.ts
tests/boundary/cli/nativePlistSelectionFailureCli.test.ts
tests/boundary/cli/nativeDemangleCli.test.ts
tests/boundary/mcp/nativeDemangleMcp.test.ts
tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts
tests/boundary/cli/nativeSignatureCli.test.ts
tests/boundary/mcp/nativeSignatureTargetBindingMcp.test.ts
src/process/capture/FilesystemSnapshot.test.ts
tests/boundary/process/darwinProcessRunTokenReader.test.ts
tests/boundary/process/directAnalysisSignals.test.ts
tests/boundary/process/processCaptureLifecycle.test.ts
tests/boundary/process/processCli.test.ts
tests/boundary/mcp/sessionMcp.test.ts
reconstruction-readiness:
name: Reconstruction readiness
needs: changes
if: github.event_name == 'push' || needs.changes.outputs.code == 'true'
if: needs.changes.outputs.readiness == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
@@ -379,7 +384,7 @@ jobs:
windows-curated:
name: Windows capability and package lane
needs: [changes, windows-native-build]
if: github.event_name == 'push' || needs.changes.outputs.package == 'true'
if: needs.changes.outputs.package == 'true'
runs-on: windows-latest
timeout-minutes: 25
steps:
@@ -476,7 +481,8 @@ jobs:
docs:
name: Docs Generation
if: github.event_name == 'pull_request'
needs: changes
if: needs.changes.outputs.docs == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
@@ -496,3 +502,104 @@ jobs:
docs/verification/managed-conformance-*.json
skills/reverse-engineer-anything
if-no-files-found: error
workflows:
name: Validate workflows
needs: changes
if: needs.changes.outputs.workflows == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Validate workflow syntax and expressions
run: go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 -shellcheck= -pyflakes=
browser:
needs: changes
if: needs.changes.outputs.browser == 'true'
uses: ./.github/workflows/real-browser.yml
binary_layout:
needs: changes
if: needs.changes.outputs.binary_layout == 'true'
uses: ./.github/workflows/real-binary-layout.yml
evm:
needs: changes
if: needs.changes.outputs.evm == 'true'
uses: ./.github/workflows/real-evm-interface.yml
javascript_recovery:
needs: changes
if: needs.changes.outputs.javascript_recovery == 'true'
uses: ./.github/workflows/real-javascript-recovery.yml
recorded_crash:
needs: changes
if: needs.changes.outputs.recorded_crash == 'true'
uses: ./.github/workflows/real-recorded-crash.yml
web_network:
needs: changes
if: needs.changes.outputs.web_network == 'true'
uses: ./.github/workflows/real-web-network-captures.yml
web_runtime:
needs: changes
if: needs.changes.outputs.web_runtime == 'true'
uses: ./.github/workflows/real-web-runtime.yml
web_source_map:
needs: changes
if: needs.changes.outputs.web_source_map == 'true'
uses: ./.github/workflows/real-web-source-map.yml
nativeaot:
needs: changes
if: needs.changes.outputs.nativeaot == 'true'
uses: ./.github/workflows/nativeaot-fixtures.yml
website:
needs: changes
if: needs.changes.outputs.website == 'true'
uses: ./.github/workflows/website-check.yml
required:
name: CI required
if: ${{ always() }}
needs:
[
changes,
pr-title,
static,
build,
test-shard,
test,
conformance-fixtures,
windows-native-build,
package-e2e,
apple-artifacts,
reconstruction-readiness,
windows-curated,
docs,
workflows,
browser,
binary_layout,
evm,
javascript_recovery,
recorded_crash,
web_network,
web_runtime,
web_source_map,
nativeaot,
website,
]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version-file: .nvmrc
- name: Require every selected lane to pass
env:
CI_JOBS: ${{ toJSON(needs) }}
run: node scripts/ci/required.mjs
+1 -6
View File
@@ -2,12 +2,7 @@ name: Source-built NativeAOT fixtures
on:
workflow_dispatch:
pull_request:
paths:
- "bridge/ghidra/extensions/nativeaot/**"
- "scripts/generate-nativeaot-fixtures.mjs"
- "tests/fixtures/nativeaot/**"
- ".github/workflows/nativeaot-fixtures.yml"
workflow_call:
permissions:
contents: read
+1 -30
View File
@@ -7,36 +7,7 @@ on:
description: "Also verify the complete 65,281-section result over MCP (slow)"
type: boolean
default: false
pull_request:
paths:
- ".github/workflows/real-binary-layout.yml"
- "bridge/pwntools/**"
- "src/native/pwntools/**"
- "src/domain/native/binaryLayout*"
- "src/application/binaryDiagnostics/**"
- "src/contracts/native/binaryDiagnosticsToolContracts.ts"
- "src/composition/binaryDiagnostics.ts"
- "src/server/registerBinaryDiagnosticsTools.ts"
- "src/server/createServer.ts"
- "src/cli/binaryDiagnosticsCommands.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/process/OwnedCommand.ts"
- "src/process/ProviderProcess.ts"
- "src/process/ProviderDeadline.ts"
- "src/process/ProcessOwnership*.ts"
- "src/process/PrivateRuntimeRoot.ts"
- "src/artifacts/readStableArtifact*.ts"
- "src/artifacts/ArtifactReader.ts"
- "src/domain/analysisError*.ts"
- "scripts/verify-binary-layout.mjs"
- "scripts/lib/elf-section-name-fixtures.mjs"
- "scripts/lib/large-result-mcp.mjs"
- "scripts/fixtures/binary-layout*"
- "package.json"
- "package-lock.json"
workflow_call:
permissions:
contents: read
+1 -34
View File
@@ -1,40 +1,7 @@
name: Real browser verification
on:
pull_request:
paths:
- ".github/workflows/real-browser.yml"
- "scripts/verify-real-browser.mjs"
- "scripts/verify-browser-network.mjs"
- "scripts/verify-browser-scripts.mjs"
- "scripts/verify-browser-modules.mjs"
- "scripts/verify-browser-dom-destinations.mjs"
- "scripts/lib/browser-*.mjs"
- "scripts/fixtures/browser-network-site.mjs"
- "scripts/fixtures/browser-script-site.mjs"
- "scripts/fixtures/browser-module-site.mjs"
- "scripts/fixtures/browser-verifier-site.mjs"
- "scripts/fixtures/page-cdp-proxy.mjs"
- "src/browser/**"
- "src/domain/browser*.ts"
- "src/domain/webScript*.ts"
- "src/domain/webModule*.ts"
- "src/application/WebModule*.ts"
- "src/composition/webModules.ts"
- "src/artifacts/readStableArtifact.ts"
- "src/artifacts/ArtifactPaths.ts"
- "src/application/WebScriptExportService.ts"
- "src/contracts/webScriptToolContracts.ts"
- "src/server/registerWebScriptTool.ts"
- "src/server/registerWebModuleTool.ts"
- "src/server/createServer.ts"
- "src/cli/webModuleCommands.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolEffects.ts"
- "src/cli/webScriptCommands.ts"
- "package.json"
- "package-lock.json"
workflow_call:
workflow_dispatch:
concurrency:
+1 -27
View File
@@ -2,33 +2,7 @@ name: Real offline EVM interface
on:
workflow_dispatch:
pull_request:
paths:
- ".github/workflows/real-evm-interface.yml"
- "src/evm/**"
- "src/domain/evm/**"
- "src/application/evm/**"
- "src/contracts/evm/**"
- "src/composition/evm.ts"
- "src/cli/evmCommands.ts"
- "src/server/registerEvmTools.ts"
- "src/server/createServer.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/process/OwnedCommand.ts"
- "src/process/ProviderProcess.ts"
- "src/process/ProcessOwnership*.ts"
- "src/process/ProviderDeadline.ts"
- "src/process/PrivateRuntimeRoot.ts"
- "src/artifacts/readStableArtifact*.ts"
- "src/artifacts/ArtifactReader.ts"
- "scripts/verify-evm-interface.mjs"
- "scripts/fixtures/evm-interface.sol"
- "third_party/evmole/**"
- "package.json"
- "package-lock.json"
workflow_call:
permissions:
contents: read
+1 -26
View File
@@ -1,32 +1,7 @@
name: Real JavaScript recovery
on:
pull_request:
paths:
- ".github/workflows/real-javascript-recovery.yml"
- "src/javascript/recovery/**"
- "src/domain/javascript/**"
- "src/application/javascript/JavaScriptRecovery*.ts"
- "src/composition/javascriptRecovery.ts"
- "src/contracts/javascript/javascriptRecoveryToolContracts.ts"
- "src/cli/javascriptRecoveryCommands.ts"
- "src/cli/javascriptApplicationAnalysis.ts"
- "src/server/registerJavaScriptRecoveryTool.ts"
- "src/server/createServer.ts"
- "src/server/sessionAvailabilityPolicy.ts"
- "src/application/CapabilityInventory.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolContracts.ts"
- "src/contracts/toolEffects.ts"
- "scripts/rea.mjs"
- "src/process/**"
- "src/artifacts/SafeOutputTree.ts"
- "scripts/verify/javascript/recovery.mjs"
- "scripts/fixtures/javascript-recovery/**"
- "third_party/wakaru"
- "package.json"
- "package-lock.json"
workflow_call:
workflow_dispatch:
permissions:
+1 -28
View File
@@ -2,34 +2,7 @@ name: Real recorded crash evidence
on:
workflow_dispatch:
pull_request:
paths:
- ".github/workflows/real-recorded-crash.yml"
- "bridge/pwntools/**"
- "bridge/pwndbg/**"
- "src/native/pwntools/**"
- "src/native/pwndbg/**"
- "src/domain/native/recordedCrash*"
- "src/application/binaryDiagnostics/**"
- "src/contracts/native/binaryDiagnosticsToolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/contracts/toolOutputSchemaGroups.ts"
- "src/composition/binaryDiagnostics.ts"
- "src/server/registerRecordedCrashTools.ts"
- "src/server/createServer.ts"
- "src/server/sessionAvailabilityPolicy.ts"
- "src/cli/binaryDiagnosticsCommands.ts"
- "src/cliCommandNames.ts"
- "src/process/**"
- "src/artifacts/readStableArtifact*.ts"
- "src/domain/analysisError*.ts"
- "scripts/verify-recorded-crash.mjs"
- "scripts/verify/recorded-crash/**"
- "scripts/lib/recorded-core-fixtures.py"
- "scripts/fixtures/recorded-crash.c"
- "third_party/pwndbg/**"
- "package.json"
- "package-lock.json"
workflow_call:
permissions:
contents: read
+57
View File
@@ -0,0 +1,57 @@
name: Real Termux browser verification
on:
workflow_dispatch:
concurrency:
group: real-termux-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
env:
HUSKY: "0"
jobs:
real-termux:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- name: Download the pinned Termux app and verify its release checksum
shell: bash
run: |
mkdir -p "$RUNNER_TEMP/rea-termux"
cd "$RUNNER_TEMP/rea-termux"
release=https://github.com/termux/termux-app/releases/download/v0.118.3
apk=termux-app_v0.118.3+github-debug_x86_64.apk
sums=termux-app_v0.118.3+github-debug_sha256sums
curl --fail --location --retry 3 --output "$apk" "$release/$apk"
curl --fail --location --retry 3 --output "$sums" "$release/$sums"
awk -v apk="$apk" '$2 == apk || $2 == "*" apk' "$sums" > selected.sha256
test "$(wc -l < selected.sha256)" -eq 1
sha256sum --check selected.sha256
- name: Enable emulator acceleration
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm
- name: Build and capture a page inside Termux
uses: reactivecircus/android-emulator-runner@4c44018e59b437e86cdfc41da381398f93ed8808 # v2
with:
api-level: 30
arch: x86_64
target: google_apis
ram-size: 4096M
heap-size: 512M
disk-size: 8192M
disable-animations: true
script: bash scripts/verify/termux/emulator.sh
- name: Upload Termux and Android diagnostics
if: always()
uses: actions/upload-artifact@v4
with:
name: termux-browser-diagnostics
path: ${{ runner.temp }}/rea-termux/diagnostics
if-no-files-found: warn
@@ -2,38 +2,7 @@ name: Real historical web network captures
on:
workflow_dispatch:
pull_request:
paths:
- ".github/workflows/real-web-network-captures.yml"
- "bridge/mitmproxy/**"
- "src/browser/history/**"
- "src/domain/webNetworkCapture*.ts"
- "src/domain/analysisError*.ts"
- "src/contracts/errorSchemas.ts"
- "src/domain/explicitSensitive*.ts"
- "src/application/WebNetworkCapture*"
- "src/composition/webNetworkCaptures.ts"
- "src/cli/webNetworkCaptureCommands.ts"
- "src/server/registerWebNetworkCaptureTool.ts"
- "src/server/createServer.ts"
- "src/server/toolRegistrationOptions.ts"
- "src/cli.ts"
- "src/cliCommandNames.ts"
- "src/contracts/toolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/process/OwnedCommand.ts"
- "src/process/ProviderProcess.ts"
- "src/process/ProviderDeadline.ts"
- "src/process/ProcessOwnership*.ts"
- "src/process/PrivateRuntimeRoot.ts"
- "src/process/capture/ProcessCaptureLifecycle.ts"
- "src/artifacts/readStableArtifact*.ts"
- "src/artifacts/ArtifactReader.ts"
- "src/contracts/webNetworkCaptureToolContracts.ts"
- "scripts/verify-web-network-captures.mjs"
- "scripts/fixtures/generate-mitmproxy-capture.py"
- "package.json"
- "package-lock.json"
workflow_call:
permissions:
contents: read
+1 -29
View File
@@ -1,34 +1,6 @@
name: Real website runtime attribution
on:
pull_request:
paths:
- ".github/workflows/real-web-runtime.yml"
- "scripts/verify-browser-runtime.mjs"
- "scripts/lib/browser-runtime-*.mjs"
- "src/process/ProviderProcess.ts"
- "src/process/ProcessOwnership*.ts"
- "scripts/fixtures/browser-runtime-site.mjs"
- "src/browser/execution/**"
- "src/browser/CdpConnection.ts"
- "src/browser/CdpEndpoint.ts"
- "src/browser/CdpTarget*.ts"
- "src/browser/CdpCaptureValues.ts"
- "src/browser/CdpAuthorized*.ts"
- "src/domain/webRuntime.ts"
- "src/domain/webExecution.ts"
- "src/domain/webEventListeners.ts"
- "src/domain/browserObservationError*.ts"
- "src/application/WebRuntime*.ts"
- "src/composition/webRuntime.ts"
- "src/contracts/webRuntimeToolContracts.ts"
- "src/contracts/toolEffects.ts"
- "src/server/registerWebRuntimeTools.ts"
- "src/server/createServer.ts"
- "src/cli/webRuntimeCommands.ts"
- "src/cli/commandCancellation.ts"
- "src/cli.ts"
- "package.json"
- "package-lock.json"
workflow_call:
workflow_dispatch:
permissions:
contents: read
+1 -22
View File
@@ -1,27 +1,6 @@
name: Real website source-map verification
on:
pull_request:
paths:
- ".github/workflows/real-web-source-map.yml"
- "scripts/verify-browser-source-maps.mjs"
- "scripts/lib/browser-source-map-e2e.mjs"
- "scripts/fixtures/browser-source-map-site.mjs"
- "src/javascript/sourceMaps/**"
- "src/browser/assets/**"
- "src/domain/webSource*.ts"
- "src/domain/webScript*.ts"
- "src/application/WebSourceLocation*.ts"
- "src/application/ValidateSelectedWebScript.ts"
- "src/composition/webSourceLocations.ts"
- "src/server/registerWebSourceLocationTool.ts"
- "src/server/createServer.ts"
- "src/contracts/webScriptToolContracts.ts"
- "src/process/**"
- "src/cli/webSourceLocationCommands.ts"
- "src/cli.ts"
- "src/artifacts/readStableArtifact.ts"
- "package.json"
- "package-lock.json"
workflow_call:
workflow_dispatch:
permissions:
contents: read
+34
View File
@@ -35,7 +35,21 @@ jobs:
contents: write
pull-requests: write
steps:
- name: Respect finalized release notes
id: editorial
env:
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
run: |
finalized=$(gh api --paginate "repos/${GH_REPO}/pulls?state=open&base=main&per_page=100" \
--jq '.[] | select(.head.repo.full_name == .base.repo.full_name) | select(.head.ref | startswith("release-please--branches--main--")) | select(any(.labels[]; .name == "rea:release-notes-finalized")) | .number')
if [[ -n "${finalized}" ]]; then
echo "Finalized release notes on PR ${finalized}; automatic proposal refresh is paused." >> "$GITHUB_STEP_SUMMARY"
echo "frozen=true" >> "$GITHUB_OUTPUT"
fi
- uses: googleapis/release-please-action@v4
if: steps.editorial.outputs.frozen != 'true'
with:
token: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
target-branch: main
@@ -80,6 +94,16 @@ jobs:
node scripts/verify-release-checkpoint.mjs \
--phase publish --release-branch main --source-sha "$SOURCE_SHA"
- name: Validate finalized notes before release creation
env:
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
RELEASE_PR: ${{ github.event.pull_request.number }}
SOURCE_SHA: ${{ github.event.pull_request.merge_commit_sha }}
run: |
node scripts/release-notes.mjs publication --repository "$GH_REPO" \
--source "$SOURCE_SHA" --pr "$RELEASE_PR"
- uses: googleapis/release-please-action@v4
id: release
with:
@@ -192,6 +216,16 @@ jobs:
--phase "$RELEASE_PHASE" --release-branch "$RELEASE_BRANCH"
--source-sha "$SOURCE_SHA"
- name: Validate manual finalized notes before release creation
if: inputs.phase == 'publish'
env:
GH_TOKEN: ${{ secrets.RELEASE_PLEASE_TOKEN || secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
SOURCE_SHA: ${{ steps.branch.outputs.sha }}
run: |
node scripts/release-notes.mjs publication --repository "$GH_REPO" \
--source "$SOURCE_SHA"
- uses: googleapis/release-please-action@v4
id: release
with:
+5 -6
View File
@@ -1,12 +1,7 @@
name: Website checks
on:
pull_request:
branches: [main, "release/*"]
paths:
- "website/**"
- "scripts/*website*.py"
- ".github/workflows/**"
workflow_call:
workflow_dispatch:
permissions:
@@ -35,5 +30,9 @@ jobs:
sudo apt-get update
sudo apt-get install --no-install-recommends -y libcairo2 fonts-dejavu-core
python3 -m pip install -r website/requirements.txt
- name: Set up Node.js for website script tests
uses: actions/setup-node@v4
with:
node-version-file: .nvmrc
- name: Check shared HTML, links, assets and contributor workflows
run: python3 scripts/website.py check
+7
View File
@@ -59,8 +59,15 @@ jobs:
sudo apt-get update
sudo apt-get install --no-install-recommends -y libcairo2 fonts-dejavu-core
python3 -m pip install -r website/requirements.txt
- name: Set up Node.js for website script tests
uses: actions/setup-node@v4
with:
node-version-file: .nvmrc
- name: Check shared HTML, links, assets and contributor workflows
run: python3 scripts/website.py check
# An out-of-date translation only warns in checks; publish it with its notice.
- name: Refresh translation notices
run: python3 scripts/website.py sync
- name: Record the published commit
run: |
python3 - <<'PY'
+2 -2
View File
@@ -1,6 +1,6 @@
{
"_comment": "Test files are intentionally NOT excluded: the gate used to report zero clones because it skipped **/*.test.ts, hiding all test-tree duplication. Current actual is ~2.4% lines across src+ and tests/, down from 3.4% when tests were included. Lower this as duplication is removed.",
"threshold": 2.5,
"_comment": "Test files are intentionally NOT excluded: the gate used to report zero clones because it skipped **/*.test.ts, hiding all test-tree duplication. Current actual is ~2.47% lines across src+ and tests/, down from 3.4% when tests were included. The 2.6% limit leaves modest headroom for ordinary changes; lower it as duplication is removed.",
"threshold": 2.6,
"reporters": ["consoleFull"],
"ignore": [
"node_modules/**",
+2 -1
View File
@@ -22,7 +22,8 @@
],
"max-params": ["warn", 4],
"no-var": "warn",
"no-var-requires": "error"
"no-var-requires": "error",
"typescript/no-import-type-side-effects": "error"
},
"overrides": [
{
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "6.3.0"
".": "6.4.0"
}
+64
View File
@@ -1,5 +1,69 @@
# Changelog
## [6.4.0](https://github.com/morluto/rea/compare/rea-agents-6.3.0...rea-agents-6.4.0) (2026-10-11)
### Highlights
- Android: added device inspection, split-APK acquisition, logs, screenshots, file transfers and explicit app lifecycle actions through ADB, plus manifest and localized-string decoding with Apktool. ([#1552](https://github.com/morluto/rea/pull/1552), [#1556](https://github.com/morluto/rea/pull/1556)) Thanks @3QL.
- JEB: browse project units and decompile selected items through a caller-started local JEB MCP server. ([#1536](https://github.com/morluto/rea/pull/1536)) Thanks @3QL.
- Large analyses: discover retained Evidence without loading every payload, and inspect JavaScript graphs, binary layouts and function dossiers through summaries and selected views. ([#1682](https://github.com/morluto/rea/pull/1682), [#1568](https://github.com/morluto/rea/pull/1568), [#1570](https://github.com/morluto/rea/pull/1570), [#1345](https://github.com/morluto/rea/pull/1345), [#1507](https://github.com/morluto/rea/pull/1507), [#1247](https://github.com/morluto/rea/issues/1247)) Thanks @jhuang-tw, @kaoru0822-kitauji, @N0zoM1z0.
- JavaScript: isolate source analysis in an owned process with configurable heap and timeout limits; retain completed facts and explicit unknowns when analysis runs out of resources. ([#1651](https://github.com/morluto/rea/pull/1651), [#1563](https://github.com/morluto/rea/issues/1563), [#1565](https://github.com/morluto/rea/issues/1565)) Thanks @N0zoM1z0, @peter-research.
- Ghidra: configure import language/compiler specifications and seed functions, code and labels before auto-analysis. ([#1667](https://github.com/morluto/rea/pull/1667)) Thanks @sqmagellan.
- RPC discovery: recognize supported tRPC calls, HTTP-link URLs and named GraphQL operations in JavaScript applications and web bundles. ([#1670](https://github.com/morluto/rea/pull/1670)) Thanks @pompiliodonatoapp.
### Changes
- Added Ghidra support for Windows PE DLLs and bounded MIPS ELF analysis, including carved MIPS inputs with incomplete section tables. Windows P0 remains read-only. ([#1303](https://github.com/morluto/rea/pull/1303), [#1362](https://github.com/morluto/rea/pull/1362), [#1629](https://github.com/morluto/rea/pull/1629), [#1290](https://github.com/morluto/rea/issues/1290), [#718](https://github.com/morluto/rea/issues/718), [#1619](https://github.com/morluto/rea/issues/1619)) Thanks @bqw371, @jaelliot, @kvalls, @N0zoM1z0.
- Enabled `set_address_name` and `set_addresses_names` for Ghidra sessions with mutation authority, including namespace-aware data labels and per-address batch results. ([#1635](https://github.com/morluto/rea/pull/1635), [#1634](https://github.com/morluto/rea/issues/1634)) Thanks @giri256.
- Recovered additional signed AArch64 relative jump-table patterns. ([#1592](https://github.com/morluto/rea/pull/1592)) Thanks @RishTheFish21.
- Added PE resource inspection and read-only NativeAOT metadata parsing for supported x64 PE32+/.NET 8 RTR 9.1 inputs. The NativeAOT parser’s Windows-host Ghidra workflow remains unverified on a real Windows host. ([#1482](https://github.com/morluto/rea/pull/1482), [76d8f68](https://github.com/morluto/rea/commit/76d8f68cb918dfb43869f2b649a0b6c0c1cf266d), [#1254](https://github.com/morluto/rea/issues/1254)) Thanks @N0zoM1z0, @thangldw.
- Added Pi, Hermes and Qwen Code setup, including Hermes active-profile support. ([#1376](https://github.com/morluto/rea/pull/1376), [#1403](https://github.com/morluto/rea/pull/1403), [#1415](https://github.com/morluto/rea/pull/1415), [#1409](https://github.com/morluto/rea/issues/1409)) Thanks @rasatpetabit, @yiliang114, @zaydiscold.
- Added `REA_MCP_INPUT_SCHEMA_PROFILE=compact` for size-capped MCP clients such as Kimi/Moonshot, while retaining canonical runtime validation. ([#1514](https://github.com/morluto/rea/pull/1514), [#1580](https://github.com/morluto/rea/pull/1580), [#1484](https://github.com/morluto/rea/issues/1484), [#1581](https://github.com/morluto/rea/issues/1581)) Thanks @haythemsellami, @N0zoM1z0.
- Added live process-capture progress and optional `finalization_ms` to capture shutdown output before timeout escalation; cancellation remains immediate. ([#1483](https://github.com/morluto/rea/pull/1483), [#1583](https://github.com/morluto/rea/pull/1583), [#1252](https://github.com/morluto/rea/issues/1252), [#1253](https://github.com/morluto/rea/issues/1253), [modelcontextprotocol/typescript-sdk#2997](https://github.com/modelcontextprotocol/typescript-sdk/issues/2997)) Thanks @N0zoM1z0, @nikolay-dementiev, @sathariels.
- Improved Codex onboarding and recovery from observed client, PATH and provider blockers. ([#1613](https://github.com/morluto/rea/pull/1613)) Thanks @N0zoM1z0.
- Reduced redirected JSON and terminal output size, and deferred heavy imports until their commands need them. ([#1413](https://github.com/morluto/rea/pull/1413), [#1450](https://github.com/morluto/rea/pull/1450), [#1439](https://github.com/morluto/rea/pull/1439), [21dbcf7](https://github.com/morluto/rea/commit/21dbcf7a1458219989ad291e44ce7d017378b727)) Thanks @kaoru0822-kitauji.
### Fixes
- Electron analysis: follow renamed and default-imported APIs and optional calls, ignore overwritten identities and media preload attributes, and resolve packaged `app.asar` pages and unpacked companions. ([#1391](https://github.com/morluto/rea/pull/1391), [#1393](https://github.com/morluto/rea/pull/1393), [#1400](https://github.com/morluto/rea/pull/1400), [#1664](https://github.com/morluto/rea/pull/1664), [#1673](https://github.com/morluto/rea/pull/1673)) Thanks @kaoru0822-kitauji, @marnixpluim, @N0zoM1z0, @suninrain086.
- JavaScript data flow: preserve returned/yielded reference uncertainty, invalidate property-stored aliases correctly, and retain bundled-module runtime scope. ([#1437](https://github.com/morluto/rea/pull/1437), [#1445](https://github.com/morluto/rea/pull/1445), [#1471](https://github.com/morluto/rea/pull/1471), [#1577](https://github.com/morluto/rea/pull/1577), [#1428](https://github.com/morluto/rea/issues/1428), [#1442](https://github.com/morluto/rea/issues/1442), [#1449](https://github.com/morluto/rea/issues/1449), [#1576](https://github.com/morluto/rea/issues/1576)) Thanks @CassieFox23, @dakjdakd, @N0zoM1z0.
- JavaScript packages: resolve exact exports subpaths, preserve trailing-slash directory requests and exact CommonJS keys, and respect shadowed `require` bindings. ([#1424](https://github.com/morluto/rea/pull/1424), [#1615](https://github.com/morluto/rea/pull/1615), [5485483](https://github.com/morluto/rea/commit/5485483f7da2f6cb66441bb784aaf2a1a5f8731a), [8b2f851](https://github.com/morluto/rea/commit/8b2f8514c50e469c3add8ce9ba7ebebef888b14f), [f998fcb](https://github.com/morluto/rea/commit/f998fcb1cca9994c3589fb70d6e0127fc215f76f), [1e51c73](https://github.com/morluto/rea/commit/1e51c732bbf08191688d6526bf58ab2650b601e3), [#1365](https://github.com/morluto/rea/issues/1365)) Thanks @akshitkhokhani, @AWUQAQ, @immanuwell, @rudycelekli.
- Large JavaScript inputs: handle oversized inline source maps, deep returns and reassignment loops; avoid argument expansion limits and yield during large analysis passes. ([#1386](https://github.com/morluto/rea/pull/1386), [#1368](https://github.com/morluto/rea/pull/1368), [#1390](https://github.com/morluto/rea/pull/1390), [#1539](https://github.com/morluto/rea/pull/1539), [#1525](https://github.com/morluto/rea/pull/1525), [#1542](https://github.com/morluto/rea/pull/1542), [#1545](https://github.com/morluto/rea/pull/1545), [#1557](https://github.com/morluto/rea/pull/1557), [#1380](https://github.com/morluto/rea/issues/1380), [#1401](https://github.com/morluto/rea/issues/1401), [#1538](https://github.com/morluto/rea/issues/1538), [#1462](https://github.com/morluto/rea/issues/1462)) Thanks @cold-atom, @KadirJeager, @kaoru0822-kitauji, @marnixpluim, @N0zoM1z0, @rudycelekli.
- JavaScript receivers: expand escaped methods once, index returned-object targets, and avoid replaying earlier writes or duplicating retained graph allocations. ([#1517](https://github.com/morluto/rea/pull/1517), [#1537](https://github.com/morluto/rea/pull/1537), [#1444](https://github.com/morluto/rea/pull/1444), [#1601](https://github.com/morluto/rea/pull/1601), [#1495](https://github.com/morluto/rea/issues/1495)) Thanks @CassieFox23, @geekwolf, @kaoru0822-kitauji, @N0zoM1z0, @zeynelacikgoez.
- Browser captures: retain WebSocket URLs, empty connections and frame order; compare only common known payload coverage and preserve incomplete WebMCP replay coverage. ([db4e360](https://github.com/morluto/rea/commit/db4e3608465486466591b594cf77037ac150c988), [aa257ea](https://github.com/morluto/rea/commit/aa257ea4b16f2d2a35c12df63359d650b48b1154), [e52c0a7](https://github.com/morluto/rea/commit/e52c0a72c59c740315e89149881a27ce4d89a11c))
- Browser scenarios: seed storage in the correct partition, normalize loopback attach identities, retain declared-secret redaction and require verified lineage before Windows Electron signals. ([#1604](https://github.com/morluto/rea/pull/1604), [#1590](https://github.com/morluto/rea/pull/1590), [#1588](https://github.com/morluto/rea/pull/1588), [8b4c3de](https://github.com/morluto/rea/commit/8b4c3de3b3a3b580adec266f94f29ccedf1da228), [#1497](https://github.com/morluto/rea/issues/1497), [#1591](https://github.com/morluto/rea/issues/1591), [#1589](https://github.com/morluto/rea/issues/1589)) Thanks @Adolanium, @akushonkamen, @dakjdakd.
- Added Termux Android browser-scenario support. ([#1655](https://github.com/morluto/rea/pull/1655)) Thanks @tyloryang.
- Web inputs: accept BOM-prefixed HAR, preserve the first duplicate DOM attribute and resolve SVG/XLink URLs in HTML and XHTML. ([#1509](https://github.com/morluto/rea/pull/1509), [#1516](https://github.com/morluto/rea/pull/1516), [#1602](https://github.com/morluto/rea/pull/1602), [#1683](https://github.com/morluto/rea/pull/1683), [#1496](https://github.com/morluto/rea/issues/1496), [#1586](https://github.com/morluto/rea/issues/1586)) Thanks @CassieFox23, @kaoru0822-kitauji, @sathariels, @taljeri.
- Screenshot comparisons: validate PNG chunks, distinguish corruption from unsupported encodings and bound comparison memory. ([4e4b15b](https://github.com/morluto/rea/commit/4e4b15b8250b29e176d4191c9610a3931b693474), [b3e977e](https://github.com/morluto/rea/commit/b3e977e6d073ab1783cece8581279f87fc360411)) Thanks @rudycelekli.
- Archives: preserve case-distinct names, report collisions at their diverging segment, detect nested formats from bytes and reject Windows alternate-stream extraction paths. ([#1480](https://github.com/morluto/rea/pull/1480), [#1505](https://github.com/morluto/rea/pull/1505), [#1553](https://github.com/morluto/rea/pull/1553), [573598c](https://github.com/morluto/rea/commit/573598c6324698cc48ceafd24ed3864670e08d00), [#1470](https://github.com/morluto/rea/issues/1470), [#1491](https://github.com/morluto/rea/issues/1491)) Thanks @CassieFox23, @kaoru0822-kitauji.
- Cleanup: retain failed readers, ASAR handles, output trees and provider workers for retry without discarding completed output or partial observations. ([ffe0574](https://github.com/morluto/rea/commit/ffe0574e17b9d973860371083cb1a351be912131), [de0d341](https://github.com/morluto/rea/commit/de0d341524dc68250a45b41eb7e49fcecc1e32a9), [3f8c51e](https://github.com/morluto/rea/commit/3f8c51e9b506d88fc41aded7b1a22b9a09037c26), [d1235fa](https://github.com/morluto/rea/commit/d1235fadc901e1fb9ab3149d53063306c8545414), [026126e](https://github.com/morluto/rea/commit/026126eafe741e5a8b857433587fd615cdd44c19))
- Providers: canonicalize IDA workspace roots before identity checks, respect IDA cancellation, serialize JEB acquisition and correct Ghidra no-return/imported-symbol recovery. ([#1481](https://github.com/morluto/rea/pull/1481), [#1419](https://github.com/morluto/rea/pull/1419), [5f30ea6](https://github.com/morluto/rea/commit/5f30ea6ff2dd89aab7af6da6e8dfa5e508ed459b), [f88c990](https://github.com/morluto/rea/commit/f88c99021b88ce67e513025901f57478d8107d0c), [49e6f00](https://github.com/morluto/rea/commit/49e6f00ed08b225abf9ddb7c762553607fdab115)) Thanks @AS9929, @JonahSimon, @Y0UW1SH.
- Binary metadata: respect PE/.NET section extents, bound binary-plist expansion, scope keyed-archive traversal and decode app `Info.plist` files without `plutil`. ([#1678](https://github.com/morluto/rea/pull/1678), [#1515](https://github.com/morluto/rea/pull/1515), [#1378](https://github.com/morluto/rea/pull/1378), [#1476](https://github.com/morluto/rea/pull/1476), [975dff2](https://github.com/morluto/rea/commit/975dff2e25b14b634e14dd5e0db1537d0ca9815b), [76296c5](https://github.com/morluto/rea/commit/76296c5a3868ba3b54be6ed3896719283d6de5f8)) Thanks @jayhemnani9910, @kaoru0822-kitauji, @rudycelekli.
- Windows runtimes: preserve drive-root paths when creating private provider directories. ([ee46323](https://github.com/morluto/rea/commit/ee46323b493994902b04824a7dc8aae2a8117903))
- Compatibility: load Windows native controls under Node-API hosts with renamed executables, preserve dylib directory requirements, support Wakaru 1.x and reject directory-valued Hopper launchers. ([#1531](https://github.com/morluto/rea/pull/1531), [#1623](https://github.com/morluto/rea/pull/1623), [#1540](https://github.com/morluto/rea/pull/1540), [#1426](https://github.com/morluto/rea/pull/1426)) Thanks @evenafterallisaid, @Fazmin, @pionxzh, @xeropresence.
- Reference imports: normalize internal symlink lookup consistently, preserve exact paths and exclusion provenance, report root access failures and reject FIFO replacements without blocking. ([#1535](https://github.com/morluto/rea/pull/1535), [2ca3ad2](https://github.com/morluto/rea/commit/2ca3ad238608adfca80a7ff1aaadea6e59a58225), [6636045](https://github.com/morluto/rea/commit/6636045c9c5b472bba51afee9d434394861e93a4), [3967cfa](https://github.com/morluto/rea/commit/3967cfa43e786c65a57e89c265343983d19ef9d0), [f1c58cc](https://github.com/morluto/rea/commit/f1c58cc53fbd83b968e11b3e2eccebca51299c54)) Thanks @anujsingh-cse.
- Setup: honor active configuration and skill roots, preserve configured MCP environments, validate Hermes/OMP profiles before writing, recognize Omarchy and retain recovery backups. ([#1504](https://github.com/morluto/rea/pull/1504), [#1607](https://github.com/morluto/rea/pull/1607), [#1528](https://github.com/morluto/rea/pull/1528), [#1562](https://github.com/morluto/rea/pull/1562), [#1662](https://github.com/morluto/rea/pull/1662), [a011485](https://github.com/morluto/rea/commit/a01148553e2a37c72ae78405418ab648876a96a6), [#1606](https://github.com/morluto/rea/issues/1606), [#1521](https://github.com/morluto/rea/issues/1521), [#1561](https://github.com/morluto/rea/issues/1561)) Thanks @Adkid-Zephyr, @CassieFox23, @LuisTaffelli, @N0zoM1z0, @peter-research.
- Process capture: retain descendant lineage after launcher exit, preserve identity uncertainty before signaling and release terminals after observation failures. ([80896a1](https://github.com/morluto/rea/commit/80896a1d64b216ef893907249cd4252f86215d49), [c916850](https://github.com/morluto/rea/commit/c9168508527302ac57bc86385ccf0d824697b5dc)) Thanks @Y0UW1SH.
- CLI/MCP errors: accept JSON files for web comparisons, preserve native option help and union correction details, remove invalid aliases and validate doctor against its discovered output contract. ([#1397](https://github.com/morluto/rea/pull/1397), [#1395](https://github.com/morluto/rea/pull/1395), [#1425](https://github.com/morluto/rea/pull/1425), [#1461](https://github.com/morluto/rea/pull/1461), [8b4c96e](https://github.com/morluto/rea/commit/8b4c96e6c6b3ed6806e4cd72b556b5db6114304a), [#1421](https://github.com/morluto/rea/issues/1421), [#1454](https://github.com/morluto/rea/issues/1454)) Thanks @Fazmin, @kaoru0822-kitauji, @N0zoM1z0.
### ⚠ BREAKING CHANGES
- WebSocket results: use `network.websocket_connections` instead of `network.websocket_events`. Connections contain `request_id`, `url` and `events`; allow `payload_bytes: null` when size is unknown. No legacy alias remains. ([db4e360](https://github.com/morluto/rea/commit/db4e3608465486466591b594cf77037ac150c988))
- Direct resource APIs: pass a caller-owned `ArtifactResourceScope` to inventory/extraction, JavaScript analysis, reference imports and web exports; retain it through cleanup/shutdown. For `readJavaScriptArtifactFiles`, scope is argument three and abort signal is argument four. Keep each root inventory source’s single cleanup owner. Use `OwnedFileHandle` owners, injected ASAR `fd`/`read` handles instead of `createReadStream`, VCS `Result` values, and `WebScriptExportError`’s options object. CLI/MCP inputs for the scoped operations are unchanged. ([ffe0574](https://github.com/morluto/rea/commit/ffe0574e17b9d973860371083cb1a351be912131), [53e7fad](https://github.com/morluto/rea/commit/53e7fadbd078422ba3d55b5499a70d237c5f4908), [684199b](https://github.com/morluto/rea/commit/684199ba0be7822266d57ddb8a40e95f087fdc42), [be75ed5](https://github.com/morluto/rea/commit/be75ed5e4b4bfb146af497f8fa5d172a1ea19462), [be6272f](https://github.com/morluto/rea/commit/be6272f26115bd8e94f2ab0df4cf092004587a18), [e0e49e9](https://github.com/morluto/rea/commit/e0e49e9dd53d9d1f1d2330b23f7f880dc1c81774), [95a77b6](https://github.com/morluto/rea/commit/95a77b6e25420b3879bd0e32bdbb14d452b38776))
- Saved reference graphs: exclusions require their matching `pattern` and distinguish `project-ignored` from `default-ignored`. Re-import graphs missing that provenance. ([6585293](https://github.com/morluto/rea/commit/6585293c562853b95c3a70df02c12ad8cc17afc4))
- Terminal captures: keep initial and resized buffers within 1,000,000 cells using `columns × (rows + scrollback)`; oversized scenarios are rejected before launch. ([f9ea9a9](https://github.com/morluto/rea/commit/f9ea9a9268c854d31048da9f8a50cfa37703fe5a))
- Optional agent-verification reports: accept schema version 4, nullable `expectedFirstTool` and `targetClarificationPassed`. Production CLI/MCP contracts are unchanged. ([#1613](https://github.com/morluto/rea/pull/1613))
- Capture comparisons: recapture both sides with matching `literal-replacements-v1` semantics when custom normalization patterns are used. Legacy unversioned pattern captures remain readable but cannot be compared with new pattern captures; captures without patterns are unaffected. ([#1456](https://github.com/morluto/rea/pull/1456), [2b6f14f](https://github.com/morluto/rea/commit/2b6f14f528297cf12c776c960af0859f5cfbefbf)) Thanks @Madhavan006, @N0zoM1z0.
- Artifact graph edges: read child artifact IDs and paths from the occurrence selected by `occurrence_id`, not edge fields `child_artifact_id`/`logical_path`. Earlier inventory Evidence no longer validates against the new commitment. ([#1493](https://github.com/morluto/rea/pull/1493)) Thanks @kaoru0822-kitauji.
- Semantic graph schema: nodes and relations no longer include `identifier_strategy`, and relation IDs no longer hash it. Earlier Evidence no longer validates against this schema. ([#1474](https://github.com/morluto/rea/pull/1474)) Thanks @kaoru0822-kitauji.
- Semantic identities: select IDs from the parent graph; read exact event names from `event_name`, endpoints from `endpoint` and literal values from `properties.value`. New analyses use digest-based IDs for long/reserved event and endpoint names and `value-sha256` literal role keys, with shorter labels; string literal labels now read `string literal`. Previously stored graphs retain their original IDs and remain valid workflow inputs. ([#1427](https://github.com/morluto/rea/pull/1427), [#1423](https://github.com/morluto/rea/pull/1423), [#1407](https://github.com/morluto/rea/pull/1407)) Thanks @N0zoM1z0.
### Contributors
- Thanks also to @BaaRai, @commit-regrets, @coygeek, @Franky100-pig, @inii-sayaa, @jopemachine, @JosephDemarest, @keemsisi, @Totopo27 for contributions included in this release.
Full contribution record: [336 commits and 163 merged PRs](https://github.com/morluto/rea/blob/rea-agents-6.4.0/docs/releases/6.4.0.contributions.json).
## [6.3.0](https://github.com/morluto/rea/compare/rea-agents-6.2.0...rea-agents-6.3.0) (2026-10-09)
+5
View File
@@ -148,10 +148,15 @@ Generated fixtures and manifests remain ignored.
Use the [release guide](docs/releasing.md). Main pushes automatically refresh a
Release Please PR with the next version and changelog. Review its migration
notes, wait for the current CI checks to pass, and merge it into main to publish.
The Release workflow validates and tags that merge; both publishers build its
exact SHA. Ordinary main pushes cannot publish packages. An optional manual
`release/VERSION` checkpoint retains explicit preparation and publication.
For grouped release notes with verified contributor thanks, use the repository's
`rea-changelog-update` skill in `.agents/skills/rea-changelog-update/`. Its inventory,
credit checks, and finalization process are described in the release guide.
Keep new implementation commits on main for the next release. The workflow
owns packaged-artifact verification, npm publication, the published CLI/MCP
canary, and MCP Registry publication. Sync manual checkpoint metadata back to
+151
View File
@@ -0,0 +1,151 @@
import ghidra.app.script.GhidraScript;
import ghidra.program.model.address.Address;
import ghidra.program.model.address.AddressSpace;
import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.Program;
import ghidra.program.model.mem.MemoryBlock;
import ghidra.program.model.symbol.SourceType;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
/**
* Apply caller-declared analysis seeds before default auto-analysis.
*
* The single argument is REA's canonical seed snapshot: one
* {@code 0x<hex>\t(function|code|label)[\tname]} line per seed. Code seeds are
* applied after functions (highest address first), then labels. The outcome is stored as a
* program property that the bridge reports during its handshake.
*/
public final class ReaGhidraApplySeeds extends GhidraScript {
static final String REPORT_OPTION = "REA analysis seeds";
private record Seed(Address address, String kind, String name) {}
@Override
public void run() throws Exception {
String[] args = getScriptArgs();
if (args.length != 1) {
throw new IllegalArgumentException("REA seed script requires exactly one seed file");
}
Path path = Path.of(args[0]);
if (!Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS)) {
throw new IllegalArgumentException("REA seed file is not a regular file");
}
byte[] bytes = Files.readAllBytes(path);
String sha256 = HexFormat.of().formatHex(
MessageDigest.getInstance("SHA-256").digest(bytes)
);
AddressSpace space = currentProgram.getAddressFactory().getDefaultAddressSpace();
List<Seed> seeds = new ArrayList<>();
for (String line : new String(bytes, StandardCharsets.UTF_8).split("\n")) {
if (line.isEmpty()) continue;
String[] fields = line.split("\t", -1);
if (fields.length < 2 || fields.length > 3 || !fields[0].startsWith("0x")) {
throw new IllegalArgumentException("REA seed snapshot is malformed");
}
Address address;
try {
address = space.getAddress(Long.parseUnsignedLong(fields[0].substring(2), 16));
} catch (ghidra.program.model.address.AddressOutOfBoundsException e) {
address = null;
}
seeds.add(new Seed(address, fields[1], fields.length == 3 ? fields[2] : null));
}
// Highest address first: a function body computed before analysis stops at
// an existing function entry, so later creations cannot swallow earlier ones.
seeds.sort((left, right) -> {
if (left.address() == null) return right.address() == null ? 0 : 1;
if (right.address() == null) return -1;
return right.address().compareTo(left.address());
});
int unmapped = 0;
int logged = 0;
int codeDecoded = 0, codeFailed = 0;
int functionCreated = 0, functionExisting = 0, functionFailed = 0;
int labelApplied = 0, labelFailed = 0;
for (String pass : new String[] { "function", "code", "label" }) {
for (Seed seed : seeds) {
if (!seed.kind().equals(pass)) continue;
monitor.checkCancelled();
MemoryBlock block = seed.address() == null
? null
: currentProgram.getMemory().getBlock(seed.address());
if (block == null || (!pass.equals("label") && !block.isInitialized())) {
unmapped++;
continue;
}
switch (pass) {
case "code" -> {
if (decoded(seed.address())) codeDecoded++;
else {
codeFailed++;
if (logged++ < 20) println("REA seed failed: code " + seed.address() + " " + why(seed.address()));
}
}
case "function" -> {
Function existing = getFunctionAt(seed.address());
if (existing != null) {
functionExisting++;
if (seed.name() != null) {
existing.setName(seed.name(), SourceType.USER_DEFINED);
}
} else if (decoded(seed.address()) &&
createFunction(seed.address(), seed.name()) != null) {
functionCreated++;
} else {
functionFailed++;
if (logged++ < 20) println("REA seed failed: function " + seed.address() + " " + why(seed.address()));
}
}
default -> {
try {
createLabel(seed.address(), seed.name(), true, SourceType.USER_DEFINED);
labelApplied++;
} catch (Exception e) {
labelFailed++;
}
}
}
}
}
String report = "{\"format\":\"rea-ghidra-seeds-v1\",\"sha256\":\"" + sha256 + "\"" +
",\"entries\":" + seeds.size() +
",\"function_created\":" + functionCreated +
",\"function_existing\":" + functionExisting +
",\"function_failed\":" + functionFailed +
",\"code_decoded\":" + codeDecoded +
",\"code_failed\":" + codeFailed +
",\"label_applied\":" + labelApplied +
",\"label_failed\":" + labelFailed +
",\"unmapped\":" + unmapped + "}";
currentProgram.getOptions(Program.PROGRAM_INFO).setString(REPORT_OPTION, report);
println("REA seeds applied: " + report);
}
private String why(Address address) {
if (getDataAt(address) != null) return "defined data " + getDataAt(address).getDataType().getName();
var containing = getInstructionContaining(address);
if (containing != null && !containing.getAddress().equals(address))
return "inside instruction at " + containing.getAddress();
Function owner = getFunctionContaining(address);
if (owner != null) return "inside function " + owner.getEntryPoint();
return getInstructionAt(address) == null ? "undecodable" : "function creation refused";
}
/** Decode one instruction flow at an address, keeping an existing decode. */
private boolean decoded(Address address) {
if (getInstructionAt(address) != null) return true;
if (getDataAt(address) != null || getInstructionContaining(address) != null) {
return false;
}
disassemble(address);
return getInstructionAt(address) != null;
}
}
File diff suppressed because it is too large Load Diff
+74 -40
View File
@@ -3,20 +3,25 @@ import ghidra.app.script.GhidraScript;
import ghidra.program.model.address.Address;
import ghidra.program.model.listing.*;
import ghidra.program.model.symbol.FlowType;
import ghidra.program.model.symbol.Reference;
import ghidra.program.model.pcode.PcodeOp;
import java.util.ArrayList;
import java.util.ArrayDeque;
import java.util.Deque;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.TreeSet;
/**
* Undo wrong no-return flags from auto-analysis. A function flagged no-return that contains a
* real RET is an ordinary function (stream readers, cxa_guard_acquire, ...). Callers treated the
* call as a terminator and the decompiler truncated them. Clear the flag, then re-disassemble only
* real RET is an ordinary function (stream readers, cxa_guard_acquire, ...). Known returning
* libc/TLS imports have no body to inspect, so clear their flags by exact external ABI name.
* Callers treated the call as a terminator and the decompiler truncated them. Clear the flag, then re-disassemble only
* the call sites of the functions that were cleared and refit their owning functions in place.
*/
public final class ReaGhidraNoReturnFix extends GhidraScript {
private static final Set<String> RETURNING_IMPORTS = Set.of(
"malloc", "__tls_get_addr", "memcpy", "memset", "strlen", "__errno_location");
@Override
public void run() throws Exception {
// Windows P0 does not admit analysis-database mutation.
@@ -24,66 +29,95 @@ public final class ReaGhidraNoReturnFix extends GhidraScript {
FunctionManager fm = currentProgram.getFunctionManager();
Listing li = currentProgram.getListing();
Set<Address> cleared = new HashSet<>();
for (Function f : fm.getFunctions(true)) {
Deque<Address> pending = new ArrayDeque<>();
for (Function f : fm.getExternalFunctions()) {
monitor.checkCancelled();
if (!f.hasNoReturn() || f.isThunk()) continue;
for (Instruction ins : li.getInstructions(f.getBody(), true)) {
FlowType ft = ins.getFlowType();
boolean decodedReturn = false;
for (PcodeOp operation : ins.getPcode()) {
if (operation.getOpcode() == PcodeOp.RETURN) decodedReturn = true;
}
if (decodedReturn && !ft.isJump() && !ft.isCall()) {
f.setNoReturn(false);
cleared.add(f.getEntryPoint());
break;
}
}
// Do not apply import-name assumptions to local functions or to arbitrary externals.
if (!f.hasNoReturn() || !RETURNING_IMPORTS.contains(externalAbiName(f))) continue;
f.setNoReturn(false);
enqueueReturning(f, cleared, pending);
}
// A thunk inherits the flag of the function it forwards to.
for (Function f : fm.getFunctions(true)) {
monitor.checkCancelled();
if (!f.isThunk()) continue;
Function target = f.getThunkedFunction(true);
if (target != null && cleared.contains(target.getEntryPoint())) {
cleared.add(f.getEntryPoint());
if (clearDecodedReturn(f, li)) {
enqueueReturning(f, cleared, pending);
}
}
int redone = 0;
TreeSet<Address> owners = new TreeSet<>();
if (!cleared.isEmpty()) {
List<Address> sites = new ArrayList<>();
for (Instruction ins : li.getInstructions(true)) {
while (!pending.isEmpty()) {
Address target = pending.removeFirst();
TreeSet<Address> sites = new TreeSet<>();
// References include computed import calls after Ghidra resolves their destination.
// Collect before redisassembly mutates the reference table.
for (Reference reference : currentProgram.getReferenceManager().getReferencesTo(target)) {
monitor.checkCancelled();
FlowType ft = ins.getFlowType();
if (!(ft.isCall() && ft.isTerminal())) continue;
Address[] flows = ins.getFlows();
if (flows.length == 0 || !cleared.contains(flows[0])) continue;
// Do not redisassemble into an independently identified function.
Address next = ins.getMaxAddress().add(1);
if (fm.getFunctionAt(next) != null || !currentProgram.getMemory().contains(next)) continue;
sites.add(ins.getAddress());
sites.add(reference.getFromAddress());
}
for (Address s : sites) {
monitor.checkCancelled();
Instruction ins = li.getInstructionAt(s);
if (ins == null) continue;
FlowType ft = ins.getFlowType();
if (!(ft.isCall() && ft.isTerminal())) continue;
Address[] flows = ins.getFlows();
boolean callsTarget = false;
for (Address flow : flows) {
if (flow.equals(target)) callsTarget = true;
}
if (!callsTarget) continue;
// Do not redisassemble into an independently identified function.
Address next = ins.getMaxAddress().add(1);
if (fm.getFunctionAt(next) != null || !currentProgram.getMemory().contains(next)) continue;
Function owner = fm.getFunctionContaining(s);
if (owner != null && !owner.isThunk()) owners.add(owner.getEntryPoint());
clearListing(ins.getMinAddress(), ins.getMaxAddress());
disassemble(s);
redone++;
}
for (Address e : owners) {
monitor.checkCancelled();
Function f = fm.getFunctionAt(e);
// Refit the body in place so the name, signature and calling convention survive.
if (f != null) CreateFunctionCmd.fixupFunctionBody(currentProgram, f, monitor);
if (owner != null && !owner.isThunk()) {
CreateFunctionCmd.fixupFunctionBody(currentProgram, owner, monitor);
// Auto-analysis may have propagated the false flag to this truncated caller.
// Its RET becomes visible only now; repair its callers in the same pass.
if (clearDecodedReturn(owner, li)) enqueueReturning(owner, cleared, pending);
}
}
// No analyzeChanges here: on libpl2.so it adds about 70 s, which pushes startup past the 330 s deadline.
}
// No analyzeChanges here: on libpl2.so it adds about 70 s, which pushes startup past the 330 s deadline.
println(
"REA no-return fix: cleared=" + cleared.size() + " sites=" + redone + " refit=" + owners.size());
}
private void enqueueReturning(Function function, Set<Address> cleared, Deque<Address> pending) {
if (cleared.add(function.getEntryPoint())) pending.addLast(function.getEntryPoint());
// A thunk inherits the flag of the function it forwards to, including through other thunks.
Address[] thunks = function.getFunctionThunkAddresses(true);
if (thunks == null) return;
for (Address thunk : thunks) {
if (cleared.add(thunk)) pending.addLast(thunk);
}
}
private String externalAbiName(Function function) {
String name = function.getName();
// Mach-O stores C-linkage symbols with one object-format underscore prefix.
if (currentProgram.getExecutableFormat().contains("Mach-O") && name.startsWith("_"))
return name.substring(1);
return name;
}
private boolean clearDecodedReturn(Function function, Listing listing) throws Exception {
if (!function.hasNoReturn() || function.isThunk()) return false;
for (Instruction instruction : listing.getInstructions(function.getBody(), true)) {
monitor.checkCancelled();
FlowType flow = instruction.getFlowType();
if (flow.isJump() || flow.isCall()) continue;
for (PcodeOp operation : instruction.getPcode()) {
if (operation.getOpcode() != PcodeOp.RETURN) continue;
function.setNoReturn(false);
return true;
}
}
return false;
}
}
+6
View File
@@ -136,6 +136,12 @@ existing analysis families directly, for example
- Pull and capture outputs are additive: the destination must not already
exist. Re-running a completed pull refuses the existing directory instead
of overwriting evidence.
- Cancellation during a package pull preserves caller-selected output. The
error includes partial observations identifying completed artifacts and the
interrupted local path; remaining APKs were not attempted.
- Failed package acquisition rollback retains its directory identity and retries
cleanup before later work and during shutdown. Cleanup refuses a replacement
directory. Terminal shutdown stops admission and drains active operations.
- The provider holds no long-lived processes; every invocation is one
bounded adb run. The adb server belongs to the caller.
+34
View File
@@ -24,6 +24,29 @@ REA can attach to a user-owned Chrome-family browser through the Chrome DevTools
Target discovery returns the complete in-scope target array in one result.
## Screenshot comparison memory budget
`compare_web_screenshots` admits both PNGs against a 256 MiB estimated
working-memory budget before inflating either image. This is a processing
buffer budget for this operation, not a PNG dimension limit, CDP transport
limit, or process RSS guarantee. Header-valid dimensions that exceed the
budget return `resource_constraint` with `details.resource: "memory"` and the
estimated and maximum working-set bytes. Invalid PNG structure remains
malformed input; encodings outside supported pixel-comparison coverage remain
unsupported-target errors.
The conservative estimate includes the two decoder-owned decoded input
buffers, the current image's concatenated IDAT buffer, Node zlib's default
16 KiB output chunk backing rounded up for retained chunks, the concatenated
inflate result while its output chunks are retained, one possible extra output
chunk before zlib checks `maxOutputLength`, unfiltered scanline storage,
RGB-to-RGBA expansion, and the first retained RGBA image while the second is
decoded. RGBA output aliases its unfiltered buffer and is counted once.
Admission uses checked integer arithmetic before image-sized allocation.
Caller-owned JSON/base64 strings, input parsing and validation, transport
buffers, garbage-collector behavior, and other runtime memory are outside this
estimate; the budget does not promise a bound on total process memory.
Electron `file://` pages use a separate provider and target boundary; see [electron-observation.md](electron-observation.md).
## Provider authorities
@@ -53,8 +76,19 @@ google-chrome \
REA does not launch, own, or terminate this browser. Use a dedicated profile and stop it yourself when the investigation is complete. Do not expose the debugging port on a non-loopback interface.
On Linux, a reachable but locked desktop keyring can stall Chrome before its
first navigation. For a dedicated investigation profile, add
`--password-store=basic` to bypass that keyring. Empty or placeholder frame URLs
then describe an unloaded page; REA continues to require an allowed, committed
URL. See [the startup investigation](https://github.com/morluto/rea/issues/1384#issuecomment-6093781686).
## Request boundary
CDP connections enforce the shared 64 MiB limit per WebSocket message. A larger
reply, including a valid reply from Chrome, fails with `payload_limit`. The limit
counts message bytes before JSON parsing; it does not bound the memory used by
decoded strings or JSON objects.
Discovery supplies a literal loopback CDP endpoint and returns all eligible
page targets unless an exact HTTP(S) origin filter narrows the result. Follow-up
inspection requests select one target; their default scope is its current origin. `localhost`, private-LAN
+12
View File
@@ -156,6 +156,18 @@ Storage cookies are scoped to the page URL at each capture. Unsupported action
tags, unknown fields, duplicate step IDs, and undeclared secret references fail
validation.
Cookies are installed before the initial navigation. Local storage entries are
seeded once per browser storage partition; session storage entries are seeded
once per origin in each top-level browsing context. Seeding occurs before the
first document scripts for that area. An origin first reached by a later action
receives its seed on that first visit. Application updates, deletions, and
empty values then persist across later navigations in the same area; a new
scenario starts with its own seeds. Partitioned frames can receive separate
local storage seeds even when they share an origin. The pre-script mechanism
uses CDP debugger instrumentation. The observer resumes only pauses whose top
frame belongs to its private seed script; page breakpoints and `debugger`
statements remain under the attached debugger client's control.
The result is Evidence with an initial state followed by one record per
declared action. Each step reports action status, elapsed time, sanitized URLs,
event bounds, and independently typed capture states for screenshot, DOM,
+10 -3
View File
@@ -166,9 +166,16 @@ rea import-reference-source /absolute/path/to/source
```
The import records hashes and metadata for the supplied files, separately
from observations of the current app. File names do not automatically exclude
files. Set `REA_REFERENCE_SECRET_PATTERNS_JSON` to a JSON array of ignore
patterns when you want to exclude selected paths.
from observations of the current app. It honors the source tree's `.gitignore`
and applies default exclusions for common generated and dependency paths such
as `node_modules/`, `dist/`, and `*.log`. Pattern-based exclusions record the
matched pattern and whether it came from project, default, caller, or sensitive
path policy. Set `REA_REFERENCE_SECRET_PATTERNS_JSON` to a JSON array of ignore
patterns when you want to mark selected paths as sensitive; those patterns are
reported separately as configured-secret exclusions.
Re-import saved graphs whose pattern-based exclusions lack `pattern` before
using them in source-to-bundle comparisons.
JavaScript and TypeScript import parsing requires valid UTF-8. Malformed source
bytes retain their original hashes and sizes with a decoding diagnostic; REA
+6
View File
@@ -25,6 +25,12 @@ extraction have independent prerequisites; one does not require the other.
Both require util-linux `prlimit`, normally `/usr/bin/prlimit`; override its
absolute path with `REA_FIRMWARE_PRLIMIT_COMMAND` if needed.
REA retains at most 8 MiB of combined stdout and stderr per command. Exceeding
that diagnostic budget stops analysis with an explicit truncated-output error;
child process limits do not bound parent-side pipe retention. Completed commands
drain their output streams before recording diagnostics. Failed worker cleanup
retains the worker and workspace for retry before another request or at shutdown.
Unblob needs the external extractor for the selected format on its inherited
`PATH`: for example, `debugfs` for ext2/ext3/ext4. Its default Landlock sandbox
and skip rules stay enabled. Sandbox startup failures remain actionable errors;
+12
View File
@@ -29,6 +29,13 @@ section names or the record being inside the initial 4 KiB header probe.
Program and section declarations must agree; duplicate, conflicting, truncated
or out-of-file records are rejected. Stripped section tables and section-only
records are supported. Extended table numbering is explicitly unsupported.
For carved ELFs whose section table extends beyond the captured file, REA
examines every complete captured section header and records the unavailable
headers as a limitation. A valid program-header ABI record can still establish
the supported declaration. Available section declarations must agree with it;
their duplicate, conflicting or unreadable records remain errors. A missing
program-header record does not make ABI declarations in uncaptured sections
absent or supply a default ABI.
The field layout follows
[`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h).
@@ -92,6 +99,11 @@ observations and does not relabel the upstream result as verified.
CLI/MCP discovery and decompilation, direct calls, target/profile/Evidence
identity, snapshot round-trip and final process cleanup remain covered.
For both byte orders, the lane also carves away the section table while retaining
all program-header payloads. It checks the actual Ghidra language, entry-address
decompilation, distinct artifact digest and missing-section limitations through
CLI Evidence and an MCP-produced snapshot. Caller-selected Ghidra heap limits
are passed to these child workflows.
Pseudocode presence is a liveness check, not proof of semantic equivalence.
Instruction decoding and reader agreement likewise do not establish runtime,
delay-slot execution or complete floating-point semantics. Assertion regressions
+64 -35
View File
@@ -1,16 +1,25 @@
# Optional NativeAOT metadata recovery
REA integrates [Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot)
through an optional headless adapter. The original source is an opt-in submodule,
pinned to `effeb734fc570c32650f88b159608979dc7b423e`; its MIT license and existing
source notices are retained. No upstream source or algorithms are rewritten.
REA has a built-in read-only snapshot parser for x64 PE targets with .NET 8 RTR
9.1 metadata. It parses the exact admitted Ghidra session snapshot and uses an
in-memory overlay for rehydrated metadata; it does not define Ghidra DataTypes,
labels, functions, or memory bytes. The hydration semantics are adapted from
[Washi1337/ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot) at
`effeb734fc570c32650f88b159608979dc7b423e` (MIT), cross-checked against
.NET's `MethodTable.h` at runtime commit
`a3b2d40328be0be3f8dd43f0e5cc925b8827b044` (MIT). The required notice ships in
[`licenses/nativeaot-parser-MIT.txt`](../licenses/nativeaot-parser-MIT.txt).
The initial tested layout is .NET runtime **8.0.22**, NativeAOT RTR **9.1**,
x86-64 ELF and native Windows PE **targets analyzed on Linux x64**, using
Ghidra **12.1.4** and JDK **21**. The build accepts any Ghidra 12.1.x release and
javac 21 or newer, and it compiles with `--release 21`. Real verification of
this layout remains the Ghidra 12.1.4 and JDK 21 pair. Other NativeAOT layouts,
architectures, and hosts are unsupported.
The optional headless Ghidra adapter remains available for database annotations
and broader recovery. It uses the same pinned upstream commit as an opt-in
submodule and is a separate, more capable workflow.
The parser currently handles x64 PE32+ and RTR 9.1, including bounded
DEHYDRATED_DATA processing, MethodTable relationships, virtual slots, and
frozen strings. Other RTR versions and PE architectures are reported as
unsupported or partial. The matching .NET 8.0.22 fixture is source-built and
the parser's Windows-host Ghidra workflow has not yet had a real Windows-host
run. This does not establish Windows Ghidra import or decompiler behavior.
Windows Ghidra P0 has no database mutation authority. A PE/CLI or ReadyToRun
assembly should use `inspect_managed_artifact`; it is not NativeAOT.
@@ -43,42 +52,62 @@ arbitrary caller-supplied bytes. The bounded loader accepts regular JARs up to
## Use existing tools
Open the executable with provider `ghidra`. `inspect_native_load_image` returns
optional `observations.metadata_recovery` inline, including the detected header,
layout, recovery status, type category paths/addresses, diagnostics, annotation
coverage and derived-memory SHA. The enclosing `unsupported` load-image status
for PE/ELF concerns independent DOS loader verification; metadata recovery has
its own status. It is not a failed import.
Open the executable with provider `ghidra`. For an x64 PE, `inspect_native_load_image`
returns `observations.metadata_recovery` inline from the captured source snapshot,
including the RTR header, MethodTable addresses, frozen strings, derived overlay
identity, and examined/recovered counts. Its enclosing `unsupported` load-image
status means independent PE load-image verification was not performed; it does
not mean the metadata parser failed or the executable failed to load.
Pass one returned type path or its typed address to `inspect_native_data_type`.
Its optional `metadata_recovery` includes related/base type, implemented
interfaces and virtual slots with targets and known procedure names. Follow
these with `analyze_function`, string searches and ordinary cross-references.
For CLI use:
Pass a returned MethodTable address to `inspect_native_data_type`. A direct PE
query also derives the snapshot metadata from the authenticated Ghidra session
handshake, so it does not require a preceding load-image call. In a persistent
MCP session, load-image followed by type inspection reuses that same derivation.
The database result remains `unavailable` when Ghidra has no DataType at that
address; its `metadata_recovery.source` is `read-only-derived-overlay` and
carries related type, interface, slot, and target observations. Generated
address identities do not claim original managed names. The parser does not
merge frozen literals into Ghidra's database string inventory. Follow slot
targets with `analyze_function` and ordinary cross-references.
The built-in CLI path uses a MethodTable address returned by the parser or
another observation:
```sh
rea inspect-native-load-image ./NativeAotFixture --provider ghidra --json
rea inspect-native-data-type ./NativeAotFixture --type /NativeAOT/MethodTables/Class_ADDRESS_MT --provider ghidra --json
rea inspect-native-data-type ./NativeAotFixture --address 0xMETHODTABLE_ADDRESS --provider ghidra --json
rea function ./NativeAotFixture 0xADDRESS --provider ghidra --json
```
No new MCP workflow or approval flag is required. Configuration opts into the
adapter; without it ordinary Ghidra profiles/results remain unchanged.
The generated `/NativeAOT/...` category path belongs to the optional headless
adapter. Use `--type /NativeAOT/MethodTables/Class_ADDRESS_MT` only when that
adapter is configured and has created the corresponding Ghidra DataType.
Built-in PE analysis includes its parser contract and resource policy in the
Ghidra analysis profile; non-PE profiles do not include that contract.
## Evidence boundaries
Discovery prefers an imported RTR symbol, otherwise the upstream initialized,
non-executable-memory signature heuristic. REA rejects ambiguous candidates,
unknown layout versions, malformed directory rows, unsupported flags and
unmapped ranges before recovery. It requires the verified dehydrated layout.
The built-in parser uses an initialized non-executable PE signature scan and
requires a unique RTR 9.1 header and exactly one DEHYDRATED_DATA row. Rehydrated
bytes live only in a parser-owned overlay, identified by virtual address, size,
and SHA-256; the caller's source Buffer remains unchanged. Limits derive from
captured section extents and format field bounds. Separate resource policies
admit up to 128 MiB of captured PE bytes, 64 MiB-equivalent parser work, 64 MiB
of exact parser-owned overlay/index Buffers, and 4 MiB of serialized NativeAOT
report JSON. These are local parser limits, independent of MCP delivery; the
buffer counter does not claim to cap V8 heap or total process memory. Rows that
do not fit the report budget are omitted with explicit partial status and
coverage. Coverage counts describe the pointer scan, recovered MethodTables,
and frozen-string candidates examined, not every possible runtime type or
object. Cancellation is honored during snapshot I/O and checked immediately
before and after parsing; the synchronous parser call itself is bounded but
cannot be interrupted mid-call. Parser output never claims that a Ghidra
database type was defined.
The adapter performs rehydration, method-table recovery and frozen-object
annotation in one ephemeral Program transaction. Fatal failure/cancellation
rolls it back. Recovery may change analysis-memory bytes, never the original
executable file. Derived ranges have a digest and **no original file offset**;
they are not captured runtime state. Candidate/committed instance annotation
counts distinguish partial frozen-object coverage. Coverage concerns the
recovered candidate set, not every possible object or every runtime type.
The optional adapter performs its annotations in an ephemeral Program
transaction. That workflow may change analysis-memory bytes, never the original
executable file. It is unavailable on Windows P0, which has no database
mutation authority.
REA preserves pre-recovery calling conventions and uses the loaded compiler
specification default for newly created methods. The upstream universal
+34 -4
View File
@@ -8,9 +8,12 @@ Executable bytes remain unchanged.
Use [installation](installation.md#ghidra) for supported hosts, prerequisites,
configuration, startup deadlines, resource controls, and cleanup recovery.
[Windows P0](windows-ghidra-p0.md) has its own admission and lifecycle boundary;
[NativeAOT recovery](ghidra-nativeaot.md) is an optional, separately verified
extension. Check the [release boundary](installation.md#released-package-and-main)
when using npm.
[NativeAOT observations](ghidra-nativeaot.md) include a built-in read-only PE
snapshot parser and a separate optional annotation extension. Check the
[release boundary](installation.md#released-package-and-main) when using npm.
See [import language and analysis seeds](installation.md#ghidra-import-language-and-analysis-seeds)
to override the header-detected language or seed analysis.
Provider selection and profile identity follow
[MCP contracts](mcp-contracts.md#identity-and-discovery). One deep
@@ -29,9 +32,15 @@ not equivalence with another engine's pseudocode or analysis database.
| Strings | Only Ghidra-defined string `Data` is observed. Items report charset, byte length, and whether a required null terminator is missing. The API cannot distinguish a present terminator from fixed/Pascal layouts when no terminator is missing, so that state is named `present_or_not_required`. |
| Memory | Memory-block end addresses are exclusive. Read/write/execute, initialization, overlay, address space, and image base are direct Ghidra observations. |
| Inventory | Procedure, symbol, and string listings plus searches return the complete matching collection in one response; callers do not provide offsets or result-count limits. |
| Search | Literal and Java-regex searches scan the complete immutable inventory and return all matching entries inline. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. |
| Search | Literal and Java-regex searches return every match in the immutable provider-defined inventory; string search does not cover every text sequence in artifact bytes. Search waits for a reply or caller cancellation; there are no caller-supplied offsets or result-count limits. |
| Analysis state | The socket is exposed only after default auto-analysis completes. Established operations wait for their reply or caller cancellation; there is no fixed response-size ceiling. |
String-search completeness applies only to Ghidra-defined string `Data`. Packed
or undefined literals, including Rust strings, may be absent; an empty result
does not prove text is absent. Both list and search Evidence carry this
limitation. When an address is known, use `read_bytes` and `xrefs` to inspect
bytes and references.
## Admitted function-analysis semantics
| Concern | Ghidra contract |
@@ -46,6 +55,27 @@ not equivalence with another engine's pseudocode or analysis database.
| P-code value flow | Function dossiers include up to 3,000 high-p-code operations, at most 64 inputs per operation, and 12,000 def-use edges. Results identify memory reads/writes, branches, and calls by p-code opcode, and mark decompiler-dead operations. Large results report truncation, an omitted-operation lower bound, and known omitted-input/edge counts. The dossier is an intra-function graph; `trace_native_values` composes derived parameter/argument and return/output dependencies across resolved calls. It does not resolve memory aliasing, call side effects, runtime behavior, persistent state, or business meaning. |
| Result extent | Function instruction scans and native API boundary observations remain complete. P-code flow is the explicit bounded exception; its count fields distinguish exact retained counts from lower-bound or known-omitted counts. Caller cancellation and provider failures remain distinct. |
Function extents and instruction completeness refer to Ghidra's analyzed body.
An inferred no-return call can terminate that body before subsequent bytes.
Pseudocode, assembly, procedure info, instruction windows, and dossiers report
observed terminal call sites in Evidence limitations, including resolved callee
names, addresses, and FunctionManager flags when available. Instruction windows
and dossiers also retain these warnings in their result limitations. An
unresolved callee or unestablished terminal-flow basis remains explicit. These
observations qualify the analysis; they do not prove that a no-return flag is
incorrect or recover excluded code. Analysis profiles version this observation
contract so older snapshots cannot replay unqualified results.
On non-Windows hosts, the ephemeral import repair clears no-return flags on the
known returning external ABI names `malloc`, `__tls_get_addr`, `memcpy`, `memset`,
`strlen`, and `__errno_location`. This relies on their expected ABI behavior;
external implementations are not inspected. The repair re-disassembles affected
terminal calls and refits their owning functions, preserving independent function
entries. If a recovered caller reveals a decoded return, its propagated no-return
flag and affected callers are repaired in the same pass. Unknown imports and
genuine no-return imports keep their flags. Analysis profiles version this repair
so snapshots produced before it are not reused. Windows P0 does not run it.
`npm run verify:ghidra` compiles the versioned C oracle into debug and stripped
host-native targets (x86-64/AArch64 ELF on Linux or Mach-O on macOS), plus a native
DWARF 4 type-layout object. It proves the admitted operations, external functions, resolved thunks, exports, stripped-name
+61 -11
View File
@@ -68,13 +68,15 @@ selects the npm release, and persistent MCP registrations are pinned to the
version that performed setup. Installing newer instructions does not update a
running server or its registration.
The release checked on **2026-10-07** was **5.0.0** (133 MCP tools), published
from the fixed checkpoint
[`b33236ec`](https://github.com/morluto/rea/releases/tag/rea-agents-5.0.0).
The public CLI, MCP catalog and target-free session, and isolated update from
4.1.0 to 5.0.0 were verified through npm. The artifact includes Windows native
controls, Android/JADX and firmware tools, Ghidra function annotations, and
retained application-Evidence references.
The npm release checked on **2026-10-11** was **6.4.0**, published from
[`a56f7bb6`](https://github.com/morluto/rea/commit/a56f7bb668ab23fec8d33f7af353ac7ed2fe1895).
The actual registry package accepted dry-run setup plans for all 17 client IDs
listed below, including Qwen Code, Pi and Hermes. OpenCode's plan selected
`OPENCODE_CONFIG_DIR` even when that directory did not yet exist. These checks
used an isolated home on macOS with Node 24; they verify planning and argument
admission, without writing client configuration or establishing live integration
with every client. This resolves the npm 6.3.0 setup gaps tracked in
[#1650](https://github.com/morluto/rea/issues/1650).
Main's catalog describes the current code. A source build or a subsequent
release containing changes after this checkpoint is required for newer
@@ -142,6 +144,25 @@ listed after the table because its connector is not one of these files:
| Pi | `pi` |
| Hermes | `hermes` |
For GitHub Copilot CLI 1.0.95, `gpt-4.1` model metadata can block ordinary
chat with REA's full input-schema profile before sending a model request
(`compaction_static_context_blocked`; [#1554](https://github.com/morluto/rea/issues/1554)).
For the native offline OpenAI-compatible BYOK adapter, full-profile chat and
analysis also passed with a distinct `COPILOT_PROVIDER_MODEL_ID` such as
`rea-gpt-4.1`, `COPILOT_PROVIDER_WIRE_MODEL=gpt-4.1`, and that distinct ID selected
through `copilot --model rea-gpt-4.1`. The local fixture requested
`COPILOT_PROVIDER_MAX_PROMPT_TOKENS=1000000`; configure the limit supported by
your backend. This keeps every full schema and the actual wire model intact.
It does not establish the effective client budget or fix the built-in-ID path.
The existing compact profile passed native chat and analysis with the complete
tool inventory on Linux using a loopback model fixture. After setup, add
`"REA_MCP_INPUT_SCHEMA_PROFILE": "compact"` to the `env` object of the
`mcpServers.rea` entry in Copilot's `mcp-config.json`, preserving its other
settings, then restart the client. Compact advertisements can reduce nested
schema constraints; REA still validates complete canonical inputs. Full-profile
`gpt-5.4` metadata is also verified locally. Live model API acceptance and
Windows remain unverified; see [native client verification](testing.md).
For [Pi](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/mcp.md),
setup writes a `type: "stdio"` entry under `mcpServers.rea` in the user-level
`~/.pi/agent/mcp.json` and installs the bundled skill under `~/.agents/skills`.
@@ -334,6 +355,12 @@ Hopper installation or any file write. Declining or pressing Ctrl-C makes no
changes. Agent configuration writes preserve unrelated entries and comments,
create backups, use atomic replacement, and verify their result. Setup and
uninstall retain an existing `.rea.backup` rather than replacing the first snapshot.
Setup also retains valid environment overrides in REA's existing server entry,
such as `REA_MCP_INPUT_SCHEMA_PROFILE` and `REA_MCP_MAX_RESPONSE_BYTES`.
Detected provider settings replace only matching environment keys.
Setup and doctor report a malformed REA environment map or invalid runtime
setting before setup writes files. Repair the named setting in the client
configuration; setup does not silently discard the rest of the map.
After setup, REA reports which agents, analysis tools, and workflow files passed
its final checks. Restart any agent named in the completion message, then begin
@@ -416,7 +443,7 @@ vendor-defined limits, and a paid license is optional. REA reuses any detected
installation and preserves Hopper during uninstall.
The supported native host baseline is macOS 12+, Ubuntu 24.04+, Fedora 41+,
Nobara 44+, 64-bit Arch Linux, or CachyOS. Ghidra and IDA have their own provider-specific
Nobara 44+, 64-bit Arch Linux, CachyOS, or Omarchy. Ghidra and IDA have their own provider-specific
host requirements; Windows Ghidra uses the [experimental P0 boundary](windows-ghidra-p0.md).
On macOS, approved setup downloads the official DMG, checks its published size
@@ -598,8 +625,10 @@ dossiers, instructions, recovered data types, measured load mappings, loaded
memory bytes, and observed file offsets. Independent load-image attestation
supports DOS MZ and explicitly selected COM; PE returns its measurements with that limitation.
On Linux and macOS, `annotate_native_function` also edits a function name and/or
entry comments atomically and returns refreshed analysis. These session metadata
edits leave executable bytes unchanged and are discarded on close. GUI controls
entry comments atomically and returns refreshed analysis, and `set_address_name`
and `set_addresses_names` name a function entry or any other mapped address
(creating or renaming its primary label) with per-address success results. These
session metadata edits leave executable bytes unchanged and are discarded on close. GUI controls
require Hopper; Windows P0 remains read-only.
Windows P0 admits native x86 and x86-64 PE applications and DLLs on fixed local NTFS volumes.
@@ -684,6 +713,27 @@ is no fixed per-operation or response-size ceiling. Unresolved computed calls
remain unknown, reference-kind provenance is preserved, and provider-specific
pseudocode is never treated as original source or Hopper-equivalent text.
### Ghidra import language and analysis seeds
Ghidra detects the language from the executable header. To set it (for example, Borland C++ PE images):
```bash
export REA_GHIDRA_LANGUAGE_ID="x86:LE:32:default"
export REA_GHIDRA_COMPILER_SPEC_ID="borlandcpp" # optional; requires a language
```
REA passes `-processor`/`-cspec`, checks both against the installation's `.ldefs`, and records them in the analysis profile. DOS imports reject an override.
To seed code that auto-analysis cannot reach (computed jumps, pointers in data):
```bash
export REA_GHIDRA_SEED_FILE=/absolute/path/to/seeds.tsv
```
Each line is `address<TAB>kind[<TAB>name]`, where kind is `function`, `code` or `label` (a label needs a name). Addresses are `0x` hex in the default space; `#` lines are comments. A pre-script applies seeds before auto-analysis: functions from the highest address down, then code, then labels. The seed digest is part of the analysis profile, and the session fails if the applied seeds differ. Result limitations report the outcome counts; a seed fails when it overlaps an instruction Ghidra already decoded.
Label seeds can name mapped uninitialized storage such as `.bss`; function and code seeds require initialized bytes.
### Ghidra heap and CPU controls
Set resource controls in the environment that launches the CLI or MCP server,
@@ -810,7 +860,7 @@ For a client that requires manual configuration, use:
"mcpServers": {
"rea": {
"command": "npx",
"args": ["-y", "rea-agents@6.3.0", "mcp"]
"args": ["-y", "rea-agents@6.4.0", "mcp"]
}
}
}
+97 -3
View File
@@ -60,6 +60,12 @@ manifest and graph commitments, JavaScript Application Graph, static
Electron summary, reconstruction statistics, and explicit limitations. It does
not require a live Hopper, Ghidra, browser, or Electron process.
The application graph selects the shallowest `package.json` outside
`node_modules` as its package root, breaking equal-depth ties by Unicode
code-point order. Dependency manifests remain package observations but do not
declare Electron main or renderer entries. If only dependency manifests are
present, the inventory artifact remains the graph root.
## ASAR integrity
ASAR inventory checks Electron integrity metadata for embedded archive entries
@@ -89,9 +95,9 @@ queries use per-file range indexes, while call sites use exact-range lookups.
Existing node budgets and coverage reporting remain in effect.
Progress identifies the source file being parsed and projected. Cancellation is
checked between files, with event-loop yields that allow notifications and
completed file-local allocations to be released. A single file's synchronous
parser and semantic extraction still need to finish before that boundary.
checked between files and throughout result transfer. The owned analyzer can
be stopped while its synchronous parser or semantic extractor is active;
the CLI/MCP process remains available for control requests.
Graph and Evidence identifiers hash canonical JSON incrementally, without
assembling a single string for the whole graph. The canonical bytes and existing
@@ -113,6 +119,46 @@ smaller view, for example `--format json --filter-output
evidence_id,normalized_result.statistics`. Streaming output does not bound the
memory needed to construct the analysis graph itself.
JavaScript source parsing, static analysis and semantic projection run in one
owned analysis process, reused sequentially across files. Target code is never
executed. The analyzer defaults to 1024 MiB of V8 old-space and a 300000 ms
deadline per source, including startup and fact transfer. Select
`max_heap_mb` and `analysis_timeout_ms` through MCP, or `--max-heap-mb` and
`--analysis-timeout-ms` on the dedicated CLI command. These choices control the
analyzer; the caller's `NODE_OPTIONS` still controls the CLI/MCP process heap.
V8 old-space limits do not impose a process RSS limit.
A source that exhausts the analyzer heap does not terminate the CLI/MCP
process. Completed static facts are retained, other files can still be analyzed,
and unexamined semantic relationships remain explicit unknowns. The parent
also checks transfer expansion and accumulated results against its remaining
heap before accepting semantic facts. A resource-limited result has partial
coverage, the observed limits and an `analyze-javascript-source` unknown scope
with the specific failure, including captured diagnostics when available.
Resource failures do not increment syntax parse failures. There is no fixed
source-length cutoff.
Static observations also need memory when expanded into nodes, relationships
and Evidence. If that expansion does not fit, the file node retains
`static_analysis` and any admitted `semantic_module_analysis` observations,
including their exact values and source locations. Use the public modules item
view to read them. The graph marks those unexpanded relationships as unknown.
If cumulative result capacity is exhausted, analysis stops; inventoried but
unexamined files remain present with explicit unknown scopes.
A timeout stops the remaining analysis and returns typed failure details with
completed partial Evidence. MCP failure details return
`partial_observation: { kind: "retained-evidence", evidence_id: "..." }`
after recording succeeds, so `inspect_analysis_view` can open that analysis
directly without serializing the entire partial graph into the error response.
CLI failure output retains the complete partial Evidence.
Cancellation after source analysis begins likewise
retains completed facts; an SDK client may reject its cancelled request before
receiving that reply. On the same MCP connection, inspect the Evidence bundle
and use the retained analysis ID with `inspect_analysis_view`. Owned analyzer
cleanup is verified before its private files are released. Cleanup uncertainty
is reported separately and retains ownership for retry.
CLI workflows parse larger JSON input files incrementally from one verified
regular-file handle, without constructing a document-sized string. Smaller
files retain native JSON parsing. Both paths require strict JSON and valid
@@ -207,6 +253,46 @@ while locally shadowed receivers can still contribute network endpoint
candidates. A template recovered with a parser error and no cooked value stays
dynamic; its raw spelling is not treated as a valid JavaScript string.
RPC client syntax also contributes network endpoint candidates, in both
`analyze_javascript_application` and `analyze_web_bundle`, distinguished by
`mechanism`:
- `trpc:procedure:<method>` records the static member path between the root
expression and the method, such as `invoice.list` in
`root.invoice.list.useQuery(...)`. tRPC proxies keep these property names
through identifier minification. React hooks and TanStack option factories
(`useQuery`, `useMutation`, `useInfiniteQuery`, `useSubscription`, suspense
and prefetch variants, `queryOptions`, `infiniteQueryOptions`,
`mutationOptions`, `subscriptionOptions`) are recognized directly. The
vanilla `query` and `mutate` methods are recognized on a root bound to
`createTRPCClient` or `createTRPCProxyClient` in the same lexical binding, or on a
path of at least two segments in a source containing `TRPCClientError`,
`trpc-accept`, or `/trpc`. Factory bindings must have one unconditional
initializer and no binding assignments; shadowed locals do not inherit client
identity. `subscribe` additionally requires a tRPC observer
argument (`onData`, `onError`, `onStarted`, `onStopped`, `onComplete`, or
`onConnectionStateChange`). `this` roots, browser globals, and any path
containing an RTK Query `endpoints` segment are never procedures; cache-key
helpers such as `queryKey` issue no request and are excluded.
- `trpc:link:<link>` records the literal `url` option of `httpLink`,
`httpBatchLink`, `httpBatchStreamLink`, and `httpSubscriptionLink` when the
link's callee name survives bundling, including `(0, module.httpBatchLink)`
import-call wrappers.
- `graphql:query`, `graphql:mutation`, and `graphql:subscription` record a
named operation found at the start of a string or cooked template literal —
the name must be followed by variable definitions (`($`), a directive, or a
selection set starting with a field name — or in a precompiled
`OperationDefinition` document object. A graphql-tag `loc.source.body` copy is
not reported twice. Anonymous operations are not reported.
A procedure path is the server-side procedure name, not a resolved URL; the
transport URL depends on the link configuration and batching. When the proxy is
reached through a member expression, such as a webpack module binding
(`l.S.invoice.list`) or a property holding the proxy (`o.api.post.byId`), the
recorded path keeps those leading carrier segments; the procedure path is its
suffix. Utility calls such as `useUtils().invoice.list.fetch()`, `useQueries`,
and links whose callee was renamed by a minifier are not recognized.
Each recovered bundle module retains the exact factory-source digest. A complete
bounded AST also receives a `babel-ast-v1` structural fingerprint that ignores
ordinary identifier names while retaining syntax, literals, operators, object
@@ -230,6 +316,14 @@ substitute version-dependent Electron defaults for omitted `webPreferences`.
Dynamic option objects, bridge keys, API objects, and IPC channel expressions
remain unknown and make coverage partial.
Electron API identity follows proven lexical module aliases. Reassigned bindings,
dynamic selections, and visibly overwritten namespace members are unresolved.
Direct and destructuring assignment, loop, update, and delete targets are checked
through namespace aliases; unrelated member writes do not discard observed API
identity. This static check does
not evaluate arbitrary call side effects or runtime registration. Missing findings
do not establish that an application has no Electron boundary usage.
IPC pairing is an inference, not an observation. A renderer `invoke` pairs only
with one unique `ipcMain.handle`/`handleOnce` candidate on the same exact literal
channel; a renderer send pairs only with one unique `ipcMain.on`/`once`
+14
View File
@@ -38,6 +38,20 @@ Inspect separately obtained components explicitly and preserve their identities.
A valid implementation marker establishes a candidate native boundary, not
recovered native semantics or runtime behavior.
### Resource admission
The static provider admits regular input files up to 128 MiB before retaining
their bytes. Reads remain bounded if a file grows after admission. Metadata,
CIL decoding, and projected inspection records have a separate conservative
64 MiB representation budget; shared heap strings are cached, but repeated
output occurrences still consume that budget. These are inspection resource
policies, not PE format limits or a hard bound on total V8 heap use.
Input overflow returns a resource constraint. Decode or projection overflow
reports `resource-limit` with unavailable or partial coverage; omitted facts do
not become empty observations. Ambiguous PE section mappings are rejected
rather than interpreted according to section order.
## Analysis objective
REA's managed-code track is intended to answer five different questions without
+20
View File
@@ -167,6 +167,9 @@ Native call tracing and process capture retain available observations in
`details.partial_observation` on failure, including when cleanup succeeds.
The observation reports its partial coverage; cleanup details describe host
state separately from the execution failure.
A process capture that fails during a finalization interval keeps every signal
attempt and its delivery result in `details.partial_observation`, with
`elapsed_ms` and `exit_code` null when no exit was observed.
Derived comparisons and reconstruction verification yield before computation
and before publication, so cancellation cannot race with successful Evidence.
@@ -305,6 +308,23 @@ that reference for a summary, one section/module, or a stable page, or call
bundle exports stream canonical JSON into an atomically published file. A broad
follow-up or `get_evidence_bundle` can also exceed the response budget; exporting
preserves the complete session without sending it through a single MCP frame.
To discover existing records without their large payloads, call
`get_evidence_bundle` with `detail: "summary"`. Its `result.records` contain
Evidence IDs, artifact/provider identities, analysis-profile digests and native
procedure metadata. Optional `filters` match exact `evidence_id`, `operation`,
`target_sha256`, `analysis_profile_digest` or `procedure_address`. Use a discovered
ID with `inspect_analysis_view`; neither discovery nor retained selection starts
a provider. `retention: "complete-record"` describes stored Evidence, while
`native_dossier.value_flow` independently reports truncation, unavailable or
unknown metadata. Missing omission counts remain null. Summary output carries
every matching record; actual transport admission still applies. Omitted
`detail` preserves complete canonical bundle delivery; filters require summary.
For a portable exported bundle, the CLI shares this projection:
`rea inspect-evidence-bundle '{"path":"/tmp/evidence.json","detail":"summary","filters":{"operation":"analyze_function"}}' --format json`.
The CLI validates and materializes that local bundle under its existing memory
budget; summary delivery does not make arbitrary large files cheap to import.
Cancelling an export stops further serialization and removes its staging file.
The destination changes only when a complete export is atomically published.
+6
View File
@@ -397,6 +397,12 @@ unpaired Unicode surrogate; a rejection identifies the field and UTF-16 index
and leaves every annotation unchanged. CRLF, supplementary Unicode characters,
and combining characters are preserved. Windows P0 remains read-only. Ghidra has no
GUI authority, and REA never falls back automatically to Hopper.
`set_address_name` and `set_addresses_names` give Ghidra sessions the same
compact naming results as Hopper: a local function entry renames that function,
and any other mapped address, such as a global, receives a new or renamed primary
label. A batch applies each address separately and reports one success boolean
per address. Use them to name many globals and helpers cheaply; use
`annotate_native_function` when you also want comments and a refreshed dossier.
Ghidra name edits preserve the existing namespace. Supply either a leaf name
such as `renamed` or a fully qualified name in that namespace, such as
`alpha::renamed`. The returned qualified name can be reused as an idempotent
+104
View File
@@ -114,6 +114,12 @@ from launch, must be ordered, and must fall within `timeout_ms`. Inputs marked
Environment overrides are recorded in the scenario commitment; inherited
values are not copied into Evidence.
Terminal dimensions must satisfy `columns × (rows + scrollback) ≤ 1,000,000`.
Every scheduled resize uses the same selected scrollback and must fit this
working-buffer budget. REA rejects oversized scenarios before launching the
command; it never clamps the requested dimensions. This limit is independent
of `limits.output_bytes`, which bounds retained terminal observations.
The capture contains raw PTY output chunks and rendered terminal states,
interaction dispatch outcomes, exit reason, sampled process-tree observations,
and settlement status. When filesystem paths are selected, REA records
@@ -175,6 +181,100 @@ run the same owned-process cleanup path. Settlement reports whether the
sampled process group quiesced or whether cleanup was needed or unverifiable;
sampling cannot prove that every short-lived or detached descendant was seen.
### Finalization interval
By default a deadline sends `SIGKILL` at once. Set `finalization_ms` to give the
target time to finish: when `timeout_ms` or `idle_timeout_ms` fires, REA sends
`SIGTERM`, keeps capturing output and selected files, and sends `SIGKILL` once
if the interval ends first. Cancellation is not delayed: it sends `SIGKILL`
immediately, also during finalization, and ends the run as cancelled.
```json
{
"executable": "node",
"arguments": ["./job.mjs"],
"filesystem_observation_paths": ["./reports"],
"timeout_ms": 30000,
"finalization_ms": 1500
}
```
REA binds the captured root's first start identity to its ownership token,
checking the identity before and after reading that token. A missing or
mismatched token, or an identity change during validation, leaves the identity
unverified rather than adopting a replacement process. Both finalization
signals recheck that retained identity immediately before the call; the check
and signal remain separate operating-system operations. REA does not fall back
to a bare PID kill for either finalization signal; failed identity or ownership
validation is recorded as `unverified` with its reason. Cancellation before any
deadline keeps the existing terminal kill. Descendants are not signalled during
finalization; owned-process cleanup handles them afterwards.
`exit.reason` keeps the initiating deadline (`timeout` or `idle_timeout`).
`exit.code` is `null` for every deadline exit; the code of an exit observed
during finalization is `exit.finalization.exit_code`, and the observed signal
stays in `exit.signal`. `exit.finalization` records what was observed and is
absent when no finalization was attempted:
```json
{
"code": null,
"signal": 0,
"reason": "timeout",
"finalization": {
"requested_ms": 1500,
"elapsed_ms": 12,
"exit_code": 0,
"signals": [
{ "signal": "SIGTERM", "sent_at_ms": 0, "delivery": "signaled" }
]
}
}
```
- `signals` lists each attempt in order: `SIGTERM` first, at most one `SIGKILL`.
`sent_at_ms` is the attempt time on a monotonic clock, relative to the start of
finalization, also for an attempt that could not be delivered.
- `delivery` is `signaled` (the identity matched and the signal call returned),
`gone` (no live process matched), `identity-changed` (the PID now belongs to a
different process, which was not signalled) or `unverified` (the identity could
not be established or the call failed). None of them proves the process
exited.
An `unverified` attempt may carry a `reason` string naming why: an unavailable
identity and its reason, an inspection or signal error, or delivery that did
not settle within the bound (1000 ms). An attempt recorded `unverified` after
that bound can still complete later; it stays identity-checked, and its late
result is not added to the record.
- `elapsed_ms` is when REA observed the exit, relative to the start of
finalization, or `null` when no exit was observed. The PTY layer can deliver the
exit slightly after the operating system ended the process.
- `exit_code` is the code the PTY reported when the exit was observed during
finalization and was not caused by a signal (`exit.signal` is `0`); a
finalizer that exits with `3` records `3`, one that exits cleanly records `0`.
It is `null` when no exit was observed or the target ended on a signal, and it
is never inferred. A target that exited on its own near the deadline is
recorded the same way.
- The observed signal stays in `exit.signal`; `exit.code` is `null` for every
deadline exit. The record does not attribute an exit to a particular signal,
and a `SIGKILL` from a scenario event next to the escalation cannot be told
apart from it.
- If the `SIGKILL` cannot be delivered, REA stops waiting for an exit, keeps every
observation collected so far with the delivery results, and releases the run
through owned-process cleanup.
- The `SIGKILL` is not sequenced after the `SIGTERM` delivery result: the two
deliveries are independent, because cancellation must not wait for a slow
identity check.
`timeout_ms + finalization_ms + settle_ms` must stay at or below
`Number.MAX_SAFE_INTEGER` (9007199254740991), the largest total exactly
representable as a JavaScript number; the scenario is rejected otherwise. This is
a representability limit, not a timer limit. The nominal lifecycle budget is that
sum; polling, snapshotting and cleanup lie outside it. Scenario `events` keep
being dispatched during finalization and are recorded as usual. `finalization_ms`
of `0`, the default, keeps today's behaviour and the committed identity of every
existing scenario. `elapsed_ms` and `sent_at_ms` are timing data and take part in
exit comparison like `settlement.elapsed_ms`.
Every capture requires `truncation_details`, with separate accounting for:
- `raw_terminal`: original UTF-8 PTY chunk bytes and observed/retained chunk
@@ -187,6 +287,10 @@ Every capture requires `truncation_details`, with separate accounting for:
input, so raw omissions also limit rendered coverage.
- `filesystem_before` and `filesystem_after`: file-count/depth limits,
enumeration failures, whole-file hash budget and bytes successfully hashed.
Directory names are read incrementally within the remaining entry capacity.
When a directory exceeds that capacity, the retained subset follows filesystem
enumeration order and is then sorted; it is explicitly incomplete and cannot
establish that an omitted path is absent.
Each retained regular file without a digest has an aliased path, size,
remaining budget and reason: `file_bytes_budget`,
`file_changed_or_short_read`, or `file_unavailable`. Its `system_code` is
File diff suppressed because it is too large Load Diff
+46
View File
@@ -15,6 +15,52 @@ the original run after a partial publication.
For a release that needs an independently frozen application checkpoint, use
the optional manual path below. It retains explicit prepare and publish phases.
## Contributor-aware release notes
Use [the rea-changelog-update skill](https://github.com/morluto/rea/blob/main/.agents/skills/rea-changelog-update/SKILL.md)
to turn the bot draft into grouped Highlights, Changes, and Fixes with inline
`Thanks @...` credit. `npm run release:notes -- inventory` collects the full
selected Git range and paginated GitHub PR/author/closing-issue metadata;
`record --summary` produces a compact provenance link for public notes, while
`record` produces the full record for review. Credit a contribution once even
when it appears in highlights, details and migrations; omit maintainer
`@morluto` self-thanks. Keep complete authorship metadata in the inventory
instead of duplicating a credit ledger in the release body. `check` validates credits
and optional PR-body equality, and `render` writes the verified release section.
Run a subcommand without its required options to see its usage. GitHub reads
require an authenticated `gh`; record/check/render work offline from the saved
inventory and complete local Git history.
Save a finalized inventory on the release branch at
`docs/releases/VERSION.contributions.json`. Review unknown identities and
reverted/backported work explicitly; GitHub metadata does not establish the
quality of a user-facing claim. Preserve the existing breaking-change section
and checkpoint checks. The contribution inventory is a reviewed snapshot, not
a signed attestation; the offline checker verifies local range coverage and
credit consistency, not GitHub's authorship independently.
Before editing the live PR, add `rea:release-notes-finalized` to prevent ordinary
main-push proposals from overwriting the editorial notes. Wait for any already
running proposal to finish. The guard applies to main-push automation; do not
run manual checkpoint preparation over finalized notes. Remove the label to
resume bot regeneration. The label does not freeze the base branch: new base
commits require re-inventorying and re-reviewing the notes.
Update both the release branch's `CHANGELOG.md` and the PR body, preserving
Release Please's surrounding metadata. Before creating a release, the workflow
checks an inventory when present, requires its target to equal the reviewed
merge's first parent, and compares the freshly fetched PR body to the changelog.
It rejects changes outside the release metadata paths and stale credit records
before any tag or publication. Review the diffs within allowed metadata paths
as well, especially package dependencies and scripts. Existing candidates without inventories retain their current
checks. Merging still starts publication; a notes edit does not authorize a
merge or release by itself.
The checker supports inline Release Please PR bodies up to GitHub's 65,536
character limit, including the surrounding header/footer. Overflow-link PR
bodies need a separate integration and fail explicitly; credits are never
silently truncated.
## 1. Select a source for a manual checkpoint
Use the next minor version proposed by Release Please, including releases with
+224 -32
View File
@@ -23,6 +23,11 @@ cases. Share setup without hiding inputs or expected evidence. For successful
`.ok` assertion adds no coverage. Protocol fixtures must reject unmodeled
commands and preserve actual producer reply shapes.
Stateless source-analysis matrices may share a file-owned stdio MCP connection
and retain representative compiled CLI parity checks. Every distinct source
scenario still runs through the public workflow; session-lifecycle scenarios
need their own sessions and cleanup assertions.
Report unavailable authority as a named skip. Await owned cleanup before a
verifier reports success. Native process inspectors are released at Vitest
worker teardown; fork termination does not run Node exit hooks, and per-file
@@ -99,9 +104,10 @@ result that its client offloads, and Pi exercises default codemode execution.
Use `REA_VERIFY_RUNTIME_ROOT` to select a production-only installed REA package.
Receipts retain client versions, result digests, host coverage and owned-process
cleanup. These POSIX lanes use a deterministic loopback model, so they do not
prove live model-provider or native Windows compatibility. Qwen and Pi use
caller-configured skill directories to isolate the fixture account from the OS
home; this does not prove their default home-directory discovery.
prove live model-provider or native Windows compatibility. Qwen, Pi and Hermes
bind both `HOME` and `USERPROFILE` to the disposable account. Qwen and Pi discover
the shared personal skill through their default search paths without an explicit
skill setting; Hermes discovers it through the selected `HERMES_HOME`.
Set `REA_VERIFY_HERMES_STICKY_PROFILE=1` to exercise native Hermes selection of
a named sticky profile with `HERMES_HOME` still pointing to its root.
@@ -159,11 +165,39 @@ Use `-- chat` for ordinary
chat or `REA_VERIFY_RUNTIME_ROOT` for a production-only installed package.
The native offline BYOK adapter uses a loopback OpenAI completions/SSE fixture
and `gpt-5.4` model metadata; model inference and token usage are synthetic.
Set `REA_VERIFY_COPILOT_MODEL` to examine another model configuration. In the
verified client, `gpt-4.1` blocks the complete catalog before HTTP with
Set `REA_VERIFY_COPILOT_MODEL` to examine another client model ID and
`REA_VERIFY_COPILOT_WIRE_MODEL` for its backend model name (defaults to the client
ID). Every actual HTTP request's model is checked and recorded separately.
Both `HOME` and `USERPROFILE` point to the disposable account; native `skill add`
still selects the installed skill directory explicitly.
The same native client passed full-profile chat and analysis with the actual
wire model `gpt-4.1` under a distinct BYOK model ID:
```bash
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- chat
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- call
```
These runs retain full canonical input/output schemas, all tools, installed-skill
loading and complete Evidence readback. They use the fixture's requested
one-million-token BYOK prompt setting; effective capacity and live API acceptance
remain unknown. With the built-in `gpt-4.1` model ID, the verified client still
blocks the full input-schema profile before HTTP with
`compaction_static_context_blocked`, even when requesting a larger BYOK prompt
capacity. The effective capacity is unknown; this lane does not establish a
fix for that client/model limit. It preserves the complete catalog and schemas.
capacity. The effective capacity is unknown. The default verifier preserves
the complete catalog and full schemas. To verify the existing compact profile:
```bash
REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- chat
REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- call
```
Both native workflows passed with the complete tool inventory. The verifier selects the
profile only in its disposable Copilot registration and records it in the
receipt. Compact advertisements can reduce nested validation structure; the
server still validates complete canonical inputs. This is a tested alternate
workflow, not a fix for the full-profile client/model limit tracked in #1554.
Live model APIs, native Windows and default OS-home skill discovery remain
unverified.
@@ -187,6 +221,30 @@ The loopback OpenAI completions/SSE custom model declares a one-million-token
context and synthetic usage. Live xAI APIs, native Windows and default OS-home
skill discovery remain unverified.
`verify:commandcode-client` requires installed Command Code (verified with npm
1.79.2); select it with `REA_VERIFY_COMMANDCODE_COMMAND`. Auth, MCP tokens and
sessions use the actual OS home, so provision a disposable POSIX account rather
than overriding `HOME`. Set `REA_VERIFY_COMMANDCODE_ACCOUNT_HOME` to that account's
actual home and create `.rea-client-verification` there with exactly
`Disposable REA client verification account` followed by a newline. The lane
owns that account's `.commandcode` configuration and shared REA skill. Build the
runtime as the checkout owner, then run `node scripts/verify-commandcode-client.mjs`
as the disposable account; the npm shortcut also needs a writable checkout.
The verifier does not create accounts or install clients. It guards setup
targets, preserves an unrelated registration and backups, checks idempotence,
and verifies native default shared-skill discovery and complete activation.
Default deferred-schema delivery stays enabled: the native prompt advertises
every REA tool, exact-name `search_tools` returns every input schema, and a native
shell query recovers the saved catalog's count and digest when the client spills
it. Actual analysis of a Unicode-path fixture delivers full named-schema Evidence
to the next model request. Use `-- chat` for ordinary chat or
`REA_VERIFY_RUNTIME_ROOT` for a production-only installed package. The loopback
OpenAI completions/SSE model is a keyless BYOK endpoint declaring a million-token
context with synthetic usage. A synthetic account-key value satisfies the
client's print-mode gate; native local-only mode refuses hosted API calls.
Updates, telemetry and cron are disabled. Hosted authentication, live models
and Windows remain unverified.
`verify:omp-client` requires installed OMP (verified with the official Linux x64
18.8.7 binary); select it with `REA_VERIFY_OMP_COMMAND`. This optional POSIX lane
isolates the default agent, global config and XDG roots, guards setup targets,
@@ -208,8 +266,9 @@ unverified; skill discovery uses an explicit isolated custom directory.
verified with `@deepseek-ai/dsh@0.2.0-rc.2`) and Git. Set
`REA_VERIFY_DEEPSEEK_COMMAND` to select its executable. This optional POSIX lane
installs the skill through `rea setup --skill`, writes the native Cordis MCP
patch into an isolated `DSH_HOME`, and uses `DSH_AGENTS_HOME` for native personal
skill discovery. A separate Git root prevents inherited project skills from
patch into an isolated `DSH_HOME`, and binds `HOME` and `USERPROFILE` to the
disposable account. The native default `~/.agents/skills` search path discovers
the installed skill without a `DSH_AGENTS_HOME` override. A separate Git root prevents inherited project skills from
masking a missing installation. The default `call` mode checks skill loading,
complete catalog discovery, the actual forwarded input schemas and regexes,
and full JavaScript Evidence for a Unicode path. Use `-- chat` for ordinary
@@ -337,6 +396,12 @@ regressions until a real fixture establishes equivalent coverage.
`verify:browser` exercises source-map failure isolation and expanded-output limits through the compiled CLI and stdio MCP with real Chrome. It also submits five 2 MiB source-map annotations, checks retained script identities and explicit map omissions, and closes its owned fixture target. Replacement and document-reset behavior has separate producer-boundary coverage.
The raw Chrome fixture launchers use `--password-store=basic` so Linux startup
does not wait for an unlock prompt from a reachable but locked desktop keyring.
Such a stall leaves the initial page unloaded with empty or placeholder CDP URLs;
it does not establish permission to capture that page. See
[the keyring diagnosis](https://github.com/morluto/rea/issues/1384#issuecomment-6093781686).
`verify:browser` also captures a source-owned noise canvas as a real PNG above
8 MiB through the CLI and stdio MCP, with complete byte/digest parity and real PNG
decoding. Its SDK client explicitly permits the larger inline JSON response;
@@ -390,14 +455,15 @@ installation (`application.java.min` through `application.java.max`). Current
12.1 releases require JDK 21 or newer and set no maximum. The lanes below still
prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
| Ghidra lane | Supported runner/target | Additional local tools |
| ------------------------------------------ | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE; `-- --x86` selects native x86 PE | Ghidra 12.1.4, full JDK 21, and the matching native artifact |
| Ghidra lane | Supported runner/target | Additional local tools |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:seeds` | POSIX host; source-owned 32-bit x86 raw fixture; function order and BSS labels | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:cross-format` | Any supported Ghidra host; also analyzes AArch64 ELF, x86-64 PE, and x86-64 Mach-O | `clang`, LLD, and `lld-link` in addition to host-lane prerequisites |
| `npm run verify:ghidra:windows` | Controlled Windows x64 with native x86-64 PE; `-- --x86` selects native x86 PE | Ghidra 12.1.4, full JDK 21, and the matching native artifact |
The optional Swift lane compiles a source-owned fixture and checks modern and
provider-demangled procedure names through real stdio MCP, with and without the
@@ -611,6 +677,13 @@ boundaries and budgets.
## Optional NativeAOT Ghidra analysis
The source parser lane is independent of the optional annotation JAR. Its
synthetic RTR 9.1 PE golden checks captured-byte digest admission, rehydration
into an immutable parser overlay, MethodTable/slot relationships, and frozen
literal extraction; provider tests exercise `inspect_native_load_image` and
address-based `inspect_native_data_type` with the database result still
unavailable. These tests do not establish Windows Ghidra host behavior.
This lane is separate from the default native lane. `build:fixtures:nativeaot`
requires an existing .NET SDK 8.0.416, the platform NativeAOT compiler/linker and
runtime pack 8.0.22. It builds benign sources into ignored `_reference/` and
@@ -711,12 +784,37 @@ Pull requests that change only root `README*.md` files, `docs/`, `AGENTS.md`,
or `CONTRIBUTING.md` run formatting and generated-document validation without
the source-test shards or native package lanes. Classification compares the PR
head with its merge base, so later base-branch changes do not expand that scope.
The required static and coverage aggregate jobs remain present and fail if
classification fails. Mixed changes, unknown paths and CI configuration changes
retain full checks. Main pushes retain the full baseline, including after
documentation merges; newer commits cancel superseded CI for the same PR or
main branch. Release publication has its own concurrency group and retains its
package verification and public-registry canary.
Static and coverage aggregate jobs remain present and fail if classification
fails. Every code change retains the complete four-shard Linux suite; individual
test files are not selected by imports or filenames.
`scripts/ci/plan.mjs` selects lanes using the ownership table in
`scripts/ci/scopes.mjs`. Provider implementation changes run their real-provider
verifiers and the Linux x64 installed-package/Inspector baseline. Test-only
changes run the Linux suite without native package verification. Website changes
run website checks. Provider workflow changes validate the workflow and run that provider's verifier.
Manual-provider/Android workflow changes run workflow validation only.
Shared process/contract/CLI boundaries, dependencies, packaging inputs, unknown
paths, and changes to the planner select the full baseline. Package script-only
edits select their owning verifier when its entrypoint is identifiable; unknown
or shared script changes select everything. Android build-script edits retain
the Linux suite and workflow validation; real Android verification stays manual.
Renames account for both paths.
Main pushes and manual dispatch retain the full baseline, including all portable
provider lanes and the native matrix below. A `ci:full` PR label selects the full
baseline on the next PR run; manual dispatch can request it immediately. Each
classification job records its selection and broad-fallback reasons in the run
summary. Representative Git-diff cases live in
`tests/boundary/filesystem/ciChangeScope.test.ts`.
`CI required` checks classification and every selected lane, rejecting failed,
cancelled, or unexpectedly skipped work. Existing required check names are
preserved; changing branch protection to require the new aggregate is a separate
repository-settings operation. Newer commits cancel superseded CI for the same
PR or main branch. Release publication has its own concurrency group and retains
its package verification and public-registry canary. Licensed/self-hosted
provider workflows and the Android smoke test retain their explicit triggers.
CI exercises the pinned Node.js runtime on native hosted runners:
@@ -730,17 +828,59 @@ CI exercises the pinned Node.js runtime on native hosted runners:
Package lanes assert the actual Node platform/architecture before verification
and record those values with the Node version. The four-host package matrix
runs at most two jobs concurrently with explicit timeouts and Node heap/thread
limits; Windows retains its curated capability lane. Package checks cover
installation, CLI/MCP discovery, target-free
analysis, configuration backups/recovery, Evidence and owned lifecycle; Inspector
checks execute source-owned loopback targets and special filename cases.
runs up to four jobs concurrently on independent hosted runners with explicit
timeouts and per-runner Node heap/thread limits; Windows retains its curated
capability lane. Package checks cover installation, CLI/MCP discovery,
target-free analysis, configuration backups/recovery, Evidence and owned
lifecycle; Inspector checks execute source-owned loopback targets and special
filename cases.
The Apple artifact CLI/MCP lane uses one `macos-15` job to avoid adding demand
for scarce macOS runner capacity. It runs the artifact-format and metadata
verifiers, followed by one consolidated focused invocation containing all
existing native and process-boundary regression paths.
The Linux source-test shards download a shared runtime and generated test
metadata snapshot produced by the build job with `npm run test:prepare`. The
snapshot includes `dist/`, generated skills, the MCP tool catalog, product
catalog, and managed-conformance metadata. Each shard still installs the locked
dependencies and runs its assigned coverage-enabled tests; it no longer
regenerates the same metadata independently.
These native baseline checks complement the Linux source-test shards and the
separate Apple-artifact and real-provider lanes. Actual Hopper, Ghidra, IDA,
browser and managed-tool claims require their corresponding verification lanes.
Runner labels follow the [GitHub hosted-runner reference](https://docs.github.com/en/actions/reference/runners/github-hosted-runners).
### Termux Android browser smoke test
The **Real Termux browser verification** workflow is manual-only while its
emulator setup is being validated. After the workflow reaches the default
branch, select a revision under **Actions → Real Termux browser verification →
Run workflow**. It boots an Android 11 (API 30) x86_64 emulator, installs the
checksum-verified Termux 0.118.3 APK, and lets the app initialize its bootstrap.
The host stages the selected revision using `git archive HEAD`; no workstation
build outputs or dependencies are copied into Android.
`scripts/verify/termux/emulator.sh` sends a `RUN_COMMAND` intent to Termux's app
service. Root access is limited to staging files, sending the intent, and reading
diagnostics; installation, compilation, Node, and Chromium run as Termux's app
UID. The in-app script installs Termux's Android-linked Node distribution,
checks its version against `.nvmrc`, pins npm from `packageManager`, runs
`npm ci`, and builds with `build:termux`. Termux repository packages, including
Chromium, are resolved at run time; their installed versions are logged.
The verifier leaves Playwright's cache environment overrides unset, requires
a healthy public MCP doctor result, and captures URL plus DOM from a loopback
HTTP fixture through `capture-browser-scenario`. It checks completed steps,
the exact URL, a DOM marker, and reported browser cleanup. This lane covers one
Android emulator and Termux combination, not physical ARM devices, Electron,
or the complete desktop browser suite. A workflow definition alone does not
establish passing Android coverage; inspect its execution receipt.
The `termux-browser-diagnostics` artifact retains the Termux command log, exit
status, and Android logcat on success or failure. The emulator is disposable.
## Developer commands
Use source feedback while editing, explicit boundary checks for the changed
@@ -808,9 +948,12 @@ provider evidence.
Vitest projects use up to two workers, bounded by host parallelism.
`process-boundary` runs later with serial files because process-tree sampling
shares host resources. Acceptance and process-global cases use isolated forks
without serial scheduling. Pure domain/service and MCP boundary projects share
immutable module graphs; each MCP session still owns its resources. Other
projects retain per-file isolation. See `vitest.config.ts` for current settings.
without serial scheduling. Pure domain/service, composition, conformance,
evaluation, and MCP boundary projects share module graphs; their tests avoid
process-global state and keep resource cleanup test-scoped. Each MCP session
still owns its resources. Adapter, non-MCP boundary, acceptance, and
process-global projects retain per-file isolation. See `vitest.config.ts` for
current settings.
Build/documentation writers hold checkout-local output locks. Tests do not
hold a broad command lock; `check:pr` finishes tests before document validation.
@@ -1029,6 +1172,39 @@ real local Codex CLI with:
npm run verify:agent
```
The lane runs scenarios sequentially in standalone Codex (`--no-daemon`) with a
disposable HOME and CODEX_HOME under `~/.cache/rea-agent-evaluations/`. Set
`REA_AGENT_EVAL_ROOT` to choose another parent directory outside the OS temporary
directory: current Codex refuses helper aliases under `/tmp`, preventing shell
commands and skill reads even if MCP works. Each owned run directory is removed
after verification unless fixture retention is requested. The lane
installs the packaged skill and MCP registration through public `rea setup`,
rather than copying a skill into a project that also inherits the caller's
skills and configuration. Only authentication is copied, when present, and it
is removed even with `REA_AGENT_EVAL_KEEP_FIXTURES=true`. No user configuration,
rules, plugins or history is copied; shell snapshots and additional agents are
disabled. Existing API-key environment authentication remains available.
The native missing-provider scenario preapproves only `open_binary` and
`close_binary` in the disposable client so it reaches REA's actual provider
boundary instead of stopping at Codex approval. This does not validate a real
native engine or authorize arbitrary runtime execution.
Select a smaller run with `REA_AGENT_EVAL_SCENARIOS`, a comma-separated list of
scenario IDs. `asar-zh` exercises the packaged desktop rubric from a Chinese
request. `javascript-module-view` exercises summary analysis, module paging and
item inspection in a tree with unrelated modules; its routing and text checks
remain heuristic, so review the retained transcript for selector choice and
coverage claims. Set `REA_CODEX_CLI` to an installed Codex executable and
`REA_AGENT_EVAL_MODEL` to an account-supported model. Use an external resource
limit for the whole process tree when testing on a constrained host; a Node
heap limit alone does not bound client and child-process memory.
`missing-target` leaves desktop fixtures visible without selecting one in the
request. It requires zero REA calls and a structured request for an app name or
artifact path. Its `expectedFirstTool` is `null`, and
`targetClarificationPassed` grades that limited routing outcome independently
of analysis prose heuristics. Analyzing a nearby example cannot pass this case.
The packaged desktop and parser-comparison scenarios now assess a closed set of
known fixture claims. Each final answer must be a strict JSON manifest containing
exactly the requested claim IDs, values, Evidence IDs, and Evidence authority and
@@ -1053,9 +1229,21 @@ navigation-context, and address-context scenarios have no factual rubric and
remain `not_assessed`; the managed workflow requires both artifact and member
inspection to answer its type and entry-point question.
The closed factual scenarios request complete producer Evidence on the initial
analysis so their source selectors can authenticate all configured facts.
Summary/page/item workflows are exercised separately by `javascript-module-view`;
its heuristic gate does not establish full-producer factual correctness.
All scenarios retain routing, workflow, validation, repetition, process-exit,
and token-use gates. Configured factual scenarios additionally require a factual
pass. Their answer-text heuristics are diagnostic and do not affect the gate.
Validation counts include complete text-only REA `invalid_request` envelopes
and failed SDK calls whose arguments violate the named current REA contract.
Client approval rejections and provider unavailability remain distinct failures;
truncated JSON previews are not parsed into invented error codes.
Repeated `binary_session` observations separated by a successful `open_binary`
or `close_binary` are treated as lifecycle verification. An unchanged retry,
including one after a failed lifecycle call, still counts as repetition.
Scenarios without a rubric retain the legacy text-heuristic gate.
`answerTermCoverageMet` checks case-insensitive substrings,
`epistemicCuePresent` checks keywords, and `finalCitesEvidence` checks only an ID's
@@ -1063,7 +1251,11 @@ presence. These metrics can still accept fabricated prose and must not be read
as factual assessments. Set `REA_AGENT_EVAL_TRANSCRIPT_DIR` to retain complete
tool results and final answers for review.
Report schema version 3 changes the top-level `factualCorrectness` from a constant
Report schema version 4 adds negative routing scenarios with nullable
`expectedFirstTool` and a `targetClarificationPassed` outcome. Consumers must
allow an expected absence of REA calls; this does not make an analysis scenario
pass without its required tools. Version 3 changed the top-level
`factualCorrectness` from a constant
string to an assessment summary with status, scope, and assessed/passed/failed/
not-assessed scenario counts. Scenario records include the factual assessment and
configured claim IDs. The evaluation scope is
+6 -3
View File
@@ -21,9 +21,12 @@ additional permission flag or degraded mode is required.
- the 25 read-only Ghidra inventory, memory, and function-analysis operations.
Loaded memory reads and file offsets preserve Ghidra's observed source mappings.
PE load-image inspection returns measurements with an explicit unsupported
attestation status; independent format-specific verification currently supports
DOS MZ only.
PE load-image inspection returns an explicit unsupported attestation status;
independent load-image verification currently supports DOS MZ only. For x64 PE
NativeAOT RTR 9.1, the read-only snapshot parser can report metadata through
`inspect_native_load_image` and `inspect_native_data_type` without a JAR or
database mutation. A real Windows-host NativeAOT Ghidra run is still required
to establish the complete host workflow.
This boundary does not establish general Windows feature parity. Ghidra GUI
state, annotations, other target architectures and formats, and Hopper-only
+7 -4
View File
@@ -85,19 +85,22 @@ if [[ "$version" =~ \+ ]]; then
fail "npm cannot install an exact version with build metadata. Pass a version without build metadata, then retry."
fi
prefix_args=()
install_args=(install --global)
if [[ "$platform" == "Linux" ]]; then
prefix_args=(--prefix "$HOME/.local")
install_args+=(--prefix "$HOME/.local")
install_bin="$HOME/.local/bin/rea"
else
npm_prefix="$(npm prefix --global)" || fail "the npm global prefix could not be read. Repair the npm configuration, then retry."
install_bin="$npm_prefix/bin/rea"
fi
install_args+=("$PACKAGE@$version")
printf 'REA install plan\n'
printf ' Version: %s\n' "$version"
printf ' Runtime: Node.js %s\n' "$node_version"
printf ' Command: npm install --global %s@%s\n' "$PACKAGE" "$version"
printf ' Command:'
printf ' %q' npm "${install_args[@]}"
printf '\n'
printf ' Binary: %s\n' "$install_bin"
printf ' Setup: %s\n' "$([[ "$start_setup" == true ]] && printf 'start when a terminal is available' || printf 'skipped')"
@@ -107,7 +110,7 @@ if [[ "$dry_run" == true ]]; then
fi
printf 'Installing %s@%s...\n' "$PACKAGE" "$version"
npm install --global ${prefix_args[@]+"${prefix_args[@]}"} "$PACKAGE@$version" || fail "npm could not install REA. Check registry access and npm permissions, then retry."
npm "${install_args[@]}" || fail "npm could not install REA. Check registry access and npm permissions, then retry."
[[ -x "$install_bin" ]] || fail "npm completed without installing the rea command. Check the npm global bin directory and PATH, then retry."
installed_version="$("$install_bin" --version 2>/dev/null | tr -d '[:space:]')" || fail "the installed REA version could not be read. Reinstall the requested version, then retry."
[[ "$installed_version" == "$version" ]] || fail "installed version $installed_version does not match $version. Reinstall the requested version, then retry."
+3
View File
@@ -3,9 +3,12 @@
"entry": [
"src/main.ts",
"src/cli.ts",
"src/composition/javascriptAnalysisWorkerProcess.ts",
"src/contracts/errorSchemas.ts",
"src/evaluation/AgentEvaluationFixtures.ts",
"src/application/RealHopperAssertions.ts",
"scripts/verify/termux/browser.mjs",
"scripts/verify-real-ghidra-noreturn.mjs",
"tests/processOwnershipRunner.ts"
],
"project": ["src/**/*.ts", "tests/**/*.ts", "scripts/**/*.mjs"],
+28
View File
@@ -0,0 +1,28 @@
The MIT License (MIT)
=====================
Copyright © 2023-2025 Washi
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
The read-only NativeAOT parser adapts format and hydration semantics from
Washi1337/ghidra-nativeaot commit
effeb734fc570c32650f88b159608979dc7b423e (MIT). MethodTable field layout was
cross-checked against .NET runtime src/coreclr/nativeaot/Runtime/inc/MethodTable.h
at dotnet/runtime commit a3b2d40328be0be3f8dd43f0e5cc925b8827b044 (MIT).
+12
View File
@@ -1,5 +1,7 @@
#include "authority.hpp"
#include <delayimp.h>
namespace rea {
static const napi_type_tag RESOURCE_TAG = {0x45fcbf0b49e84fe3ULL, 0x9e3e5074130de843ULL};
@@ -121,3 +123,13 @@ static napi_value initialize(napi_env env, napi_value exports) {
} // namespace rea
NAPI_MODULE(NODE_GYP_MODULE_NAME, rea::initialize)
// Node-API symbols are delay-imported from "node.exe". Resolve them against the
// host executable so Node.js, Bun, and other Node-API hosts load the addon
// regardless of their executable file name.
static FARPROC WINAPI reaHostHook(unsigned event, PDelayLoadInfo info) {
if (event != dliNotePreLoadLibrary || _stricmp(info->szDll, "node.exe") != 0) return nullptr;
return reinterpret_cast<FARPROC>(GetModuleHandleW(nullptr));
}
decltype(__pfnDliNotifyHook2) __pfnDliNotifyHook2 = reaHostHook;
+3 -2
View File
@@ -219,9 +219,10 @@ std::unique_ptr<Runtime> createRuntime(const std::wstring& parentPath, const std
require(BCryptGenRandom(nullptr, random.data(), random.size(), BCRYPT_USE_SYSTEM_PREFERRED_RNG) == 0,
"Generate runtime identity failed", parentPath, ERROR_GEN_FAILURE);
auto result = std::make_unique<Runtime>();
result->path = parent->path + L"\\" + prefix + hex(random.data(), random.size());
const auto name = prefix + hex(random.data(), random.size());
result->path = parent->path + (parent->path.back() == L'\\' ? L"" : L"\\") + name;
Security security;
result->directory = createRelative(parent->get(), prefix + hex(random.data(), random.size()),
result->directory = createRelative(parent->get(), name,
DELETE | FILE_LIST_DIRECTORY | FILE_READ_ATTRIBUTES | READ_CONTROL,
true, security, result->path);
try {
+20 -2
View File
@@ -1,12 +1,12 @@
{
"name": "rea-agents",
"version": "6.3.0",
"version": "6.4.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "rea-agents",
"version": "6.3.0",
"version": "6.4.0",
"license": "MIT",
"dependencies": {
"@babel/parser": "8.0.4",
@@ -17,6 +17,7 @@
"@lydell/node-pty": "1.1.0",
"@modelcontextprotocol/client": "2.3.1",
"@modelcontextprotocol/server": "2.3.1",
"@xmldom/xmldom": "0.9.12",
"@xterm/addon-serialize": "0.14.0",
"@xterm/headless": "6.0.0",
"@zip.js/zip.js": "2.15.0",
@@ -38,6 +39,7 @@
"plist": "5.0.0",
"semver": "7.8.5",
"smol-toml": "1.9.0",
"stream-chain": "4.2.6",
"stream-json": "3.7.0",
"write-file-atomic": "7.0.0",
"ws": "8.21.0",
@@ -64,6 +66,7 @@
"oxlint": "1.87.0",
"turbo": "2.11.7",
"typescript": "7.0.2",
"typescript-termux": "npm:typescript@5.9.3",
"vitepress": "1.6.4",
"vitest": "5.0.3"
},
@@ -7367,6 +7370,21 @@
"@typescript/typescript-win32-x64": "7.0.2"
}
},
"node_modules/typescript-termux": {
"name": "typescript",
"version": "5.9.3",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
},
"engines": {
"node": ">=14.17"
}
},
"node_modules/unbash": {
"version": "4.0.12",
"resolved": "https://registry.npmjs.org/unbash/-/unbash-4.0.12.tgz",
+16 -2
View File
@@ -1,7 +1,7 @@
{
"name": "rea-agents",
"mcpName": "io.github.morluto/rea",
"version": "6.3.0",
"version": "6.4.0",
"description": "Reverse engineer anything from your terminal or agent with one CLI and MCP server.",
"license": "MIT",
"repository": {
@@ -30,6 +30,9 @@
"access": "public"
},
"type": "module",
"imports": {
"#javascript-analysis-worker": "./dist/composition/javascriptAnalysisWorkerProcess.js"
},
"bin": {
"rea": "scripts/rea.mjs",
"rea-agents": "scripts/rea.mjs"
@@ -50,6 +53,7 @@
"bridge/ghidra/ReaGhidraBridge.java",
"bridge/ghidra/ReaGhidraExtensions.java",
"bridge/ghidra/ReaGhidraPrepareCom.java",
"bridge/ghidra/ReaGhidraApplySeeds.java",
"bridge/ghidra/ReaGhidraNoReturnFix.java",
"bridge/android/ReaJadxBridge.java",
"bridge/mitmproxy/capture.py",
@@ -77,9 +81,11 @@
"bridge/pwndbg/launch_impl.py",
"bridge/pwndbg/core_context.py",
"third_party/pwndbg/README.md",
"third_party/pwndbg/LICENSE.md"
"third_party/pwndbg/LICENSE.md",
"licenses/nativeaot-parser-MIT.txt"
],
"scripts": {
"release:notes": "node scripts/release-notes.mjs",
"deps:check": "node scripts/check-dependency-install.mjs",
"metadata:generate": "node scripts/run-exclusive.mjs artifacts node scripts/generate-package-metadata.mjs",
"metadata:check": "node scripts/generate-package-metadata.mjs --check",
@@ -101,6 +107,8 @@
"precheck:ci": "npm run deps:check",
"check:ci": "node scripts/run-exclusive.mjs artifacts turbo run typecheck lint format:check knip metadata:check jscpd scan:todos && npm run verify:architecture-guards",
"build:cached": "node scripts/check-dependency-install.mjs && node scripts/run-exclusive.mjs artifacts turbo run artifacts:skills",
"build:termux": "node scripts/run-exclusive.mjs artifacts npm run build:termux:run",
"build:termux:run": "npm run compile:termux && npm run artifacts:skills",
"format": "oxfmt .",
"format:check": "oxfmt --check .",
"lint": "oxlint && npm run verify:module-boundaries",
@@ -150,6 +158,7 @@
"verify:apktool": "npm run build:cached && node scripts/verify-apktool.mjs",
"verify:adb": "npm run build:cached && node scripts/verify-adb.mjs",
"verify:ghidra": "npm run build:cached && node scripts/verify-real-ghidra.mjs",
"verify:ghidra:seeds": "npm run build:cached && node scripts/verify/ghidra/seeds.mjs",
"verify:ghidra:mips": "npm run build:cached && node scripts/verify-real-ghidra-mips.mjs",
"verify:ghidra:swift": "npm run build:cached && node scripts/verify-real-ghidra-swift.mjs",
"verify:ida": "npm run build:cached && node scripts/verify-real-ida.mjs",
@@ -188,6 +197,7 @@
"verify:gemini-client": "npm run build:cached && node scripts/verify-gemini-client.mjs",
"verify:opencode-client": "npm run build:cached && node scripts/verify-opencode-client.mjs",
"verify:grok-client": "npm run build:cached && node scripts/verify-grok-client.mjs",
"verify:commandcode-client": "npm run build:cached && node scripts/verify-commandcode-client.mjs",
"verify:omp-client": "npm run build:cached && node scripts/verify-omp-client.mjs",
"verify:copilot-client": "npm run build:cached && node scripts/verify-copilot-client.mjs",
"verify:claude-client": "npm run build:cached && node scripts/verify-claude-client.mjs",
@@ -235,6 +245,7 @@
"verify:recorded:crash": "npm run build:cached && node scripts/verify-recorded-crash.mjs",
"mcp-catalog:check:built": "node scripts/generate-mcp-tool-catalog.mjs --check",
"compile": "node scripts/clean-build-output.mjs && tsc -p tsconfig.build.json",
"compile:termux": "node scripts/clean-build-output.mjs && node --max-old-space-size=3072 node_modules/typescript-termux/lib/tsc.js -p tsconfig.build.json",
"artifacts:mcp-catalog": "node scripts/generate-mcp-tool-catalog.mjs",
"artifacts:skills": "node scripts/generate-skill-metadata.mjs",
"artifacts:product-catalog": "node scripts/generate-product-catalog.mjs",
@@ -260,6 +271,7 @@
"@lydell/node-pty": "1.1.0",
"@modelcontextprotocol/client": "2.3.1",
"@modelcontextprotocol/server": "2.3.1",
"@xmldom/xmldom": "0.9.12",
"@xterm/addon-serialize": "0.14.0",
"@xterm/headless": "6.0.0",
"@zip.js/zip.js": "2.15.0",
@@ -280,6 +292,7 @@
"plist": "5.0.0",
"semver": "7.8.5",
"smol-toml": "1.9.0",
"stream-chain": "4.2.6",
"stream-json": "3.7.0",
"write-file-atomic": "7.0.0",
"ws": "8.21.0",
@@ -308,6 +321,7 @@
"oxlint": "1.87.0",
"turbo": "2.11.7",
"typescript": "7.0.2",
"typescript-termux": "npm:typescript@5.9.3",
"vitepress": "1.6.4",
"vitest": "5.0.3"
}
+1 -1
View File
@@ -78,7 +78,7 @@ const classes = await mkdtemp(join(output, "classes-"));
try {
const sources = [
...(await files(join(source, "src/main/java"), ".java")),
...(await files(join(root, "bridge/ghidra/extensions/nativeaot"), ".java")),
...(await files(join(root, "bridge/nativeaot"), ".java")),
];
const jars = await files(join(install, "Ghidra"), ".jar");
const args = [
+3 -2
View File
@@ -84,7 +84,8 @@ if (!(await exists(join(headers, "node_api.h")))) {
}
const artifact = "rea-windows-x64.node";
// MSVC short-import libraries do not produce a correct Node import table with
// GNU ld. Generate a GNU import library for the exact stable Node-API symbols.
// GNU ld. Generate a GNU delay-import library for the exact stable Node-API
// symbols; addon.cc binds it to the host executable, whatever its file name.
const sources = ["addon.cc", "filesystem.cc", "process.cc"];
const sourceText = (
await Promise.all(
@@ -106,7 +107,7 @@ await writeFile(
definition,
`LIBRARY node.exe\nEXPORTS\n${symbols.join("\n")}\n`,
);
run("x86_64-w64-mingw32-dlltool", ["-d", definition, "-l", library]);
run("x86_64-w64-mingw32-dlltool", ["-d", definition, "-y", library]);
run(compiler, [
"-std=c++17",
"-O2",
+198
View File
@@ -0,0 +1,198 @@
import { execFileSync } from "node:child_process";
import { appendFileSync } from "node:fs";
import { matchesGlob } from "node:path";
import { SCOPES } from "./scopes.mjs";
const flags = [
"code",
"runtime",
"package",
"apple",
"fixtures",
"readiness",
"docs",
"workflows",
...Object.keys(SCOPES),
];
// Actionlint validates workflow syntax only. The semantic suite asserts these
// workflows' release, publication and self-hosted runner safeguards.
const GUARDED_WORKFLOWS = new Set([
".github/workflows/release.yml",
".github/workflows/real-ghidra-windows.yml",
]);
// Sources compiled by build-conformance-fixtures.mjs. The readiness journey
// analyzes the built C fixture as well as its own applications.
const FIXTURE_SOURCES = [
"tests/conformance/c/**",
"tests/conformance/napi/**",
"tests/conformance/objc/**",
"tests/conformance/swift/**",
"tests/conformance/versions/**",
];
const READINESS_SOURCES = ["tests/conformance/readiness/**"];
const matchesAny = (file, patterns) =>
patterns.some((pattern) => matchesGlob(file, pattern));
const selected = Object.fromEntries(flags.map((flag) => [flag, false]));
const reasons = [];
const full = (reason) => {
for (const flag of flags) selected[flag] = true;
reasons.push(reason);
};
const git = (...args) =>
execFileSync("git", args, { encoding: "utf8" }).trimEnd();
function classifyPackage(base, head) {
const before = JSON.parse(git("show", `${base}:package.json`));
const after = JSON.parse(git("show", `${head}:package.json`));
const { scripts: beforeScripts, ...beforeRest } = before;
const { scripts: afterScripts, ...afterRest } = after;
if (JSON.stringify(beforeRest) !== JSON.stringify(afterRest)) {
full("Package metadata or dependencies changed");
return;
}
selected.code = true;
const changed = [
...new Set([
...Object.keys(beforeScripts ?? {}),
...Object.keys(afterScripts ?? {}),
]),
].filter((key) => beforeScripts?.[key] !== afterScripts?.[key]);
for (const script of changed) {
if (
/^(test(?::|$)|check(?::|$)|lint(?::|$)|format(?::|$)|knip$|jscpd$)/.test(
script,
)
)
continue;
if (
["compile:termux", "build:termux", "build:termux:run"].includes(script)
) {
selected.workflows = true;
reasons.push(
`Android build script changed; real Android verification remains manual: ${script}`,
);
continue;
}
// Verify scripts own provider setup as well as execution. Reuse their
// existing source-file ownership rather than maintaining a second map.
const entrypoint = (command) =>
/^(?:npm run build:cached && )?node (scripts\/[\w/-]+\.mjs)(?: [\w./=-]+)*$/.exec(
command ?? "",
)?.[1];
const beforeEntry = entrypoint(beforeScripts?.[script]);
const afterEntry = entrypoint(afterScripts?.[script]);
const entries = [beforeEntry, afterEntry].filter(Boolean);
const simple =
(!beforeScripts?.[script] || beforeEntry) &&
(!afterScripts?.[script] || afterEntry);
const scriptOwners =
simple &&
entries.every((entry) =>
Object.values(SCOPES).some((patterns) => patterns.includes(entry)),
)
? Object.entries(SCOPES)
.filter(([, patterns]) =>
entries.some((entry) => patterns.includes(entry)),
)
.map(([owner]) => owner)
: [];
if (scriptOwners.length)
for (const owner of scriptOwners) selected[owner] = true;
else full(`Shared or unknown package script: ${script}`);
}
}
const event = process.env.GITHUB_EVENT_NAME;
let files = [];
if (
event === "push" ||
event === "schedule" ||
event === "workflow_dispatch" ||
process.env.CI_FULL === "true"
) {
full("Full baseline requested by event or ci:full label");
} else if (event === "pull_request") {
const base = git("merge-base", process.env.BASE_SHA, process.env.HEAD_SHA);
const head = process.env.HEAD_SHA;
// Disable rename detection so both the removed and added paths participate.
files = git("diff", "--name-only", "--no-renames", "-z", base, head)
.split("\0")
.filter(Boolean);
for (const file of files) {
if (
/^(README(?:_[^/]+)?\.md|AGENTS\.md|CONTRIBUTING\.md)$/.test(file) ||
file.startsWith("docs/")
) {
selected.docs = true;
continue;
}
if (/^(src|tests)\/.*\.(?:test|spec)\.[cm]?[jt]sx?$/.test(file)) {
selected.code = true;
continue;
}
const owners = Object.entries(SCOPES)
.filter(([, patterns]) =>
patterns.some((pattern) => matchesGlob(file, pattern)),
)
.map(([owner]) => owner);
for (const owner of owners) selected[owner] = true;
if (
file.startsWith(".github/workflows/") ||
file === ".github/actionlint.yaml"
) {
selected.workflows = true;
if (GUARDED_WORKFLOWS.has(file)) selected.code = true;
if (file !== ".github/workflows/ci.yml") continue;
}
if (owners.length === 1 && owners[0] === "website") continue;
if (/^src\/(contracts|domain)\//.test(file)) selected.docs = true;
if (file === "package.json") {
classifyPackage(base, head);
continue;
}
selected.code = true;
if (matchesAny(file, FIXTURE_SOURCES)) {
selected.fixtures = true;
selected.readiness = true;
}
if (matchesAny(file, READINESS_SOURCES)) selected.readiness = true;
if (file.startsWith("tests/")) continue;
if (
/^(src\/process\/|src\/filesystem\/|src\/contracts\/tool|src\/server\/createServer|src\/cli\.ts|scripts\/ci\/)/.test(
file,
)
) {
full(`Shared boundary: ${file}`);
} else if (owners.length) {
if (file.startsWith("src/") || file.startsWith("bridge/"))
selected.runtime = true;
} else {
full(`Unclassified input: ${file}`);
}
}
} else throw new Error(`Unsupported CI event: ${event}`);
selected.runtime ||= selected.package;
const linux = { os: "ubuntu-latest", platform: "linux", architecture: "x64" };
const matrix = {
include: selected.package
? [
linux,
{ os: "ubuntu-24.04-arm", platform: "linux", architecture: "arm64" },
{ os: "macos-15", platform: "darwin", architecture: "arm64" },
{ os: "macos-15-intel", platform: "darwin", architecture: "x64" },
]
: [linux],
};
const plan = { ...selected, matrix, files, reasons };
if (process.env.GITHUB_OUTPUT)
appendFileSync(
process.env.GITHUB_OUTPUT,
Object.entries({ ...selected, matrix: JSON.stringify(matrix) })
.map(([key, value]) => `${key}=${value}\n`)
.join(""),
);
if (process.env.GITHUB_STEP_SUMMARY)
appendFileSync(
process.env.GITHUB_STEP_SUMMARY,
`### Selected CI lanes\n\n${flags.filter((flag) => selected[flag]).join(", ") || "Formatting only"}\n\n${reasons.join("\n\n")}\n`,
);
console.log(JSON.stringify(plan));
+25
View File
@@ -0,0 +1,25 @@
const jobs = JSON.parse(process.env.CI_JOBS);
const outputs = jobs.changes.outputs;
const required = ["changes", "static", "test"];
const lanes = {
code: ["build", "test-shard"],
runtime: ["windows-native-build", "package-e2e"],
package: ["windows-curated"],
apple: ["apple-artifacts"],
fixtures: ["conformance-fixtures"],
readiness: ["reconstruction-readiness"],
docs: ["docs"],
};
for (const [scope, value] of Object.entries(outputs)) {
if (value === "true") required.push(...(lanes[scope] ?? [scope]));
}
const failed = [...new Set(required)].filter(
(name) => jobs[name]?.result !== "success",
);
for (const [name, job] of Object.entries(jobs)) {
if (["failure", "cancelled"].includes(job.result) && !failed.includes(name))
failed.push(name);
}
if (failed.length)
throw new Error(`CI lanes did not pass: ${failed.join(", ")}`);
console.log("Every selected CI lane passed");
+228
View File
@@ -0,0 +1,228 @@
// Ownership of the real-provider and website verification lanes. Shared build
// inputs are classified once in plan.mjs rather than repeated in every workflow.
export const SCOPES = {
browser: [
".github/workflows/real-browser.yml",
"scripts/verify-real-browser.mjs",
"scripts/verify-browser-network.mjs",
"scripts/verify-browser-scripts.mjs",
"scripts/verify-browser-modules.mjs",
"scripts/verify-browser-dom-destinations.mjs",
"scripts/lib/browser-*.mjs",
"scripts/fixtures/browser-network-site.mjs",
"scripts/fixtures/browser-script-site.mjs",
"scripts/fixtures/browser-module-site.mjs",
"scripts/fixtures/browser-verifier-site.mjs",
"scripts/fixtures/page-cdp-proxy.mjs",
"src/browser/**",
"src/domain/browser*.ts",
"src/domain/webScript*.ts",
"src/domain/webModule*.ts",
"src/application/WebModule*.ts",
"src/composition/webModules.ts",
"src/artifacts/readStableArtifact.ts",
"src/artifacts/ArtifactPaths.ts",
"src/application/WebScriptExportService.ts",
"src/contracts/webScriptToolContracts.ts",
"src/server/registerWebScriptTool.ts",
"src/server/registerWebModuleTool.ts",
"src/server/createServer.ts",
"src/cli/webModuleCommands.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolEffects.ts",
"src/cli/webScriptCommands.ts",
],
binary_layout: [
".github/workflows/real-binary-layout.yml",
"bridge/pwntools/**",
"src/native/pwntools/**",
"src/domain/native/binaryLayout*",
"src/application/binaryDiagnostics/**",
"src/contracts/native/binaryDiagnosticsToolContracts.ts",
"src/composition/binaryDiagnostics.ts",
"src/server/registerBinaryDiagnosticsTools.ts",
"src/server/createServer.ts",
"src/cli/binaryDiagnosticsCommands.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolContracts.ts",
"src/contracts/toolEffects.ts",
"src/process/OwnedCommand.ts",
"src/process/ProviderProcess.ts",
"src/process/ProviderDeadline.ts",
"src/process/ProcessOwnership*.ts",
"src/process/PrivateRuntimeRoot.ts",
"src/artifacts/readStableArtifact*.ts",
"src/artifacts/ArtifactReader.ts",
"src/domain/analysisError*.ts",
"scripts/verify-binary-layout.mjs",
"scripts/lib/elf-section-name-fixtures.mjs",
"scripts/lib/large-result-mcp.mjs",
"scripts/fixtures/binary-layout*",
],
evm: [
".github/workflows/real-evm-interface.yml",
"src/evm/**",
"src/domain/evm/**",
"src/application/evm/**",
"src/contracts/evm/**",
"src/composition/evm.ts",
"src/cli/evmCommands.ts",
"src/server/registerEvmTools.ts",
"src/server/createServer.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolContracts.ts",
"src/contracts/toolEffects.ts",
"src/process/OwnedCommand.ts",
"src/process/ProviderProcess.ts",
"src/process/ProcessOwnership*.ts",
"src/process/ProviderDeadline.ts",
"src/process/PrivateRuntimeRoot.ts",
"src/artifacts/readStableArtifact*.ts",
"src/artifacts/ArtifactReader.ts",
"scripts/verify-evm-interface.mjs",
"scripts/fixtures/evm-interface.sol",
"third_party/evmole/**",
],
javascript_recovery: [
".github/workflows/real-javascript-recovery.yml",
"src/javascript/recovery/**",
"src/domain/javascript/**",
"src/application/javascript/JavaScriptRecovery*.ts",
"src/composition/javascriptRecovery.ts",
"src/contracts/javascript/javascriptRecoveryToolContracts.ts",
"src/cli/javascriptRecoveryCommands.ts",
"src/cli/javascriptApplicationAnalysis.ts",
"src/server/registerJavaScriptRecoveryTool.ts",
"src/server/createServer.ts",
"src/server/sessionAvailabilityPolicy.ts",
"src/application/CapabilityInventory.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolContracts.ts",
"src/contracts/toolEffects.ts",
"scripts/rea.mjs",
"src/process/**",
"src/artifacts/SafeOutputTree.ts",
"scripts/verify/javascript/recovery.mjs",
"scripts/fixtures/javascript-recovery/**",
"third_party/wakaru",
],
recorded_crash: [
".github/workflows/real-recorded-crash.yml",
"bridge/pwntools/**",
"bridge/pwndbg/**",
"src/native/pwntools/**",
"src/native/pwndbg/**",
"src/domain/native/recordedCrash*",
"src/application/binaryDiagnostics/**",
"src/contracts/native/binaryDiagnosticsToolContracts.ts",
"src/contracts/toolEffects.ts",
"src/contracts/toolOutputSchemaGroups.ts",
"src/composition/binaryDiagnostics.ts",
"src/server/registerRecordedCrashTools.ts",
"src/server/createServer.ts",
"src/server/sessionAvailabilityPolicy.ts",
"src/cli/binaryDiagnosticsCommands.ts",
"src/cliCommandNames.ts",
"src/process/**",
"src/artifacts/readStableArtifact*.ts",
"src/domain/analysisError*.ts",
"scripts/verify-recorded-crash.mjs",
"scripts/verify/recorded-crash/**",
"scripts/lib/recorded-core-fixtures.py",
"scripts/fixtures/recorded-crash.c",
"third_party/pwndbg/**",
],
web_network: [
".github/workflows/real-web-network-captures.yml",
"bridge/mitmproxy/**",
"src/browser/history/**",
"src/domain/webNetworkCapture*.ts",
"src/domain/analysisError*.ts",
"src/contracts/errorSchemas.ts",
"src/domain/explicitSensitive*.ts",
"src/application/WebNetworkCapture*",
"src/composition/webNetworkCaptures.ts",
"src/cli/webNetworkCaptureCommands.ts",
"src/server/registerWebNetworkCaptureTool.ts",
"src/server/createServer.ts",
"src/server/toolRegistrationOptions.ts",
"src/cli.ts",
"src/cliCommandNames.ts",
"src/contracts/toolContracts.ts",
"src/contracts/toolEffects.ts",
"src/process/OwnedCommand.ts",
"src/process/ProviderProcess.ts",
"src/process/ProviderDeadline.ts",
"src/process/ProcessOwnership*.ts",
"src/process/PrivateRuntimeRoot.ts",
"src/process/capture/ProcessCaptureLifecycle.ts",
"src/artifacts/readStableArtifact*.ts",
"src/artifacts/ArtifactReader.ts",
"src/contracts/webNetworkCaptureToolContracts.ts",
"scripts/verify-web-network-captures.mjs",
"scripts/fixtures/generate-mitmproxy-capture.py",
],
web_runtime: [
".github/workflows/real-web-runtime.yml",
"scripts/verify-browser-runtime.mjs",
"scripts/lib/browser-runtime-*.mjs",
"src/process/ProviderProcess.ts",
"src/process/ProcessOwnership*.ts",
"scripts/fixtures/browser-runtime-site.mjs",
"src/browser/execution/**",
"src/browser/CdpConnection.ts",
"src/browser/CdpEndpoint.ts",
"src/browser/CdpTarget*.ts",
"src/browser/CdpCaptureValues.ts",
"src/browser/CdpAuthorized*.ts",
"src/domain/webRuntime.ts",
"src/domain/webExecution.ts",
"src/domain/webEventListeners.ts",
"src/domain/browserObservationError*.ts",
"src/application/WebRuntime*.ts",
"src/composition/webRuntime.ts",
"src/contracts/webRuntimeToolContracts.ts",
"src/contracts/toolEffects.ts",
"src/server/registerWebRuntimeTools.ts",
"src/server/createServer.ts",
"src/cli/webRuntimeCommands.ts",
"src/cli/commandCancellation.ts",
"src/cli.ts",
],
web_source_map: [
".github/workflows/real-web-source-map.yml",
"scripts/verify-browser-source-maps.mjs",
"scripts/lib/browser-source-map-e2e.mjs",
"scripts/fixtures/browser-source-map-site.mjs",
"src/javascript/sourceMaps/**",
"src/browser/assets/**",
"src/domain/webSource*.ts",
"src/domain/webScript*.ts",
"src/application/WebSourceLocation*.ts",
"src/application/ValidateSelectedWebScript.ts",
"src/composition/webSourceLocations.ts",
"src/server/registerWebSourceLocationTool.ts",
"src/server/createServer.ts",
"src/contracts/webScriptToolContracts.ts",
"src/process/**",
"src/cli/webSourceLocationCommands.ts",
"src/cli.ts",
"src/artifacts/readStableArtifact.ts",
],
nativeaot: [
"bridge/nativeaot/**",
"scripts/generate-nativeaot-fixtures.mjs",
"tests/fixtures/nativeaot/**",
".github/workflows/nativeaot-fixtures.yml",
],
website: [
"website/**",
"scripts/*website*.py",
"scripts/test-website-scripts.mjs",
".github/workflows/website-check.yml",
],
};
@@ -0,0 +1,61 @@
/* Keep the configured launcher identity alive while its resource-limited
* Python child runs. Both remain in REA's owned process group. */
#define _POSIX_C_SOURCE 200809L
#include <errno.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
int main(int argc, char **argv) {
char *python = getenv("REA_VERIFY_LIMITED_PYTHON");
char *limit = getenv("REA_VERIFY_LIMIT_OPTION");
if (python == NULL || python[0] != '/' || limit == NULL) {
fputs("limited Python fixture requires an absolute Python and limit\n", stderr);
return 127;
}
struct rlimit core = {0, 0};
if (setrlimit(RLIMIT_CORE, &core) != 0) {
perror("setrlimit");
return 127;
}
char **arguments = calloc((size_t)argc + 4, sizeof(char *));
if (arguments == NULL) return 127;
arguments[0] = "/usr/bin/prlimit";
arguments[1] = limit;
arguments[2] = "--";
arguments[3] = python;
for (int index = 1; index < argc; index++) arguments[index + 3] = argv[index];
pid_t child = fork();
if (child == 0) {
execv(arguments[0], arguments);
perror("execv");
_exit(127);
}
free(arguments);
if (child < 0) {
perror("fork");
return 127;
}
int status;
while (waitpid(child, &status, 0) < 0) {
if (errno == EINTR) continue;
perror("waitpid");
return 127;
}
if (WIFEXITED(status)) return WEXITSTATUS(status);
if (WIFSIGNALED(status)) {
int signum = WTERMSIG(status);
signal(signum, SIG_DFL);
sigset_t signals;
sigemptyset(&signals);
sigaddset(&signals, signum);
sigprocmask(SIG_UNBLOCK, &signals, NULL);
raise(signum);
return 128 + signum;
}
return 127;
}
+15 -3
View File
@@ -28,6 +28,11 @@ import {
isolatedFixtureUrl,
} from "./browser-runtime-world-e2e.mjs";
// Arming, progress delivery, isolated-world proof and the page action cross
// separate transports. Leave time for their round trips on shared CI runners;
// coverage can include a late action even after request collection has ended.
const executionObservationMs = 2_000;
/** Exercise native browser runtime attribution through public CLI and stdio MCP, including arming. */
export async function verifyBrowserRuntime(
executable,
@@ -210,7 +215,7 @@ async function executionMcp(
const response = await client.callTool(
{
name: "observe_web_execution",
arguments: { ...input, observation_ms: 300 },
arguments: { ...input, observation_ms: executionObservationMs },
},
{
timeout: 40_000,
@@ -259,7 +264,7 @@ async function executionCli(entrypoint, input, env, action, isolatedContext) {
input.cdp_endpoint,
input.target_id,
"--observation-ms",
"300",
String(executionObservationMs),
"--json",
],
{ env, stdio: ["ignore", "pipe", "pipe"] },
@@ -382,7 +387,14 @@ function assertExecution(evidence, site, requireBlock = true) {
const request = result.requests.find((item) =>
item.url.endsWith("/evidence?marker=chosen"),
);
assert.ok(request, "The selected request initiator must be retained");
assert.ok(
request,
`The selected request initiator must be retained: ${JSON.stringify({
window: result.window,
requests: result.requests.map((item) => item.url),
excluded_requests: result.excluded_requests,
})}`,
);
assert.ok(
request.callsites.some(
(site) =>
@@ -18,6 +18,7 @@ export async function startRuntimeFixtureBrowser(executable, origin) {
command: executable,
arguments: [
"--headless=new",
"--password-store=basic",
...(process.env.REA_BROWSER_NO_SANDBOX === "true"
? ["--no-sandbox"]
: []),
@@ -0,0 +1,683 @@
import assert from "node:assert/strict";
import { createServer } from "node:http";
import { once } from "node:events";
import { setTimeout as delay } from "node:timers/promises";
import { CdpConnection } from "../../dist/browser/CdpConnection.js";
import { createBrowserScenarioProvider } from "../../dist/composition/browserScenario.js";
import { browserScenarioSchema } from "../../dist/domain/browserScenario.js";
import {
runScenarioCli,
scenarioProfiles,
} from "./browser-scenario-verifier.mjs";
const scenarioFor = (browser, origin) =>
browserScenarioSchema.parse({
browser,
start_url: { url: `${origin}/storage-main` },
actions: [
{
step_id: "frame-ready",
action: "wait_for",
locator: { kind: "css", selector: "#frame-report[data-ready='true']" },
state: "attached",
timeout_ms: 10_000,
},
{
step_id: "top-level-partition",
action: "goto",
destination: {
url: `${origin.replace("127.0.0.1", "localhost")}/storage-top`,
},
wait_until: "load",
},
{
step_id: "return-to-origin",
action: "goto",
destination: { url: `${origin}/storage-main` },
wait_until: "load",
},
],
storage: {
local_storage: [
{
origin,
entries: [
{
name: "rea_storage_probe",
value: { source: "literal", value: "main-initial" },
},
{
name: "rea_storage_remove",
value: { source: "literal", value: "main-remove-seed" },
},
],
},
{
origin: origin.replace("127.0.0.1", "localhost"),
entries: [
{
name: "rea_storage_probe",
value: { source: "literal", value: "frame-initial" },
},
{
name: "rea_storage_remove",
value: { source: "literal", value: "frame-remove-seed" },
},
],
},
],
session_storage: [
{
origin,
entries: [
{
name: "rea_storage_probe",
value: { source: "literal", value: "main-session-initial" },
},
{
name: "rea_storage_empty",
value: { source: "literal", value: "main-empty-seed" },
},
],
},
{
origin: origin.replace("127.0.0.1", "localhost"),
entries: [
{
name: "rea_storage_probe",
value: { source: "literal", value: "frame-session-initial" },
},
{
name: "rea_storage_empty",
value: { source: "literal", value: "frame-empty-seed" },
},
],
},
],
},
capture: { after_each_step: ["dom"], at_end: ["dom"] },
});
const raceWithTimeout = (operation, timeoutMs, onTimeout = () => undefined) => {
const controller = new AbortController();
return Promise.race([
operation,
delay(timeoutMs, undefined, { signal: controller.signal }).then(onTimeout),
]).finally(() => controller.abort());
};
const storagePage = (kind, port) => {
if (kind === "main")
return `<!doctype html><html><body><pre id="main-report"></pre><pre id="frame-report"></pre><iframe src="http://localhost:${port}/storage-frame"></iframe><script>
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
document.querySelector("#main-report").textContent = JSON.stringify(state);
localStorage.setItem("rea_storage_probe", "main-updated");
sessionStorage.setItem("rea_storage_probe", "main-session-updated");
localStorage.removeItem("rea_storage_remove");
sessionStorage.setItem("rea_storage_empty", "");
addEventListener("message", (event) => {
if (event.data?.kind !== "rea-storage-frame") return;
const report = document.querySelector("#frame-report");
report.textContent = JSON.stringify(event.data.state);
report.dataset.ready = "true";
});
</script></body></html>`;
if (kind === "frame")
return `<!doctype html><html><body><pre id="frame-state"></pre><script>
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
document.querySelector("#frame-state").textContent = JSON.stringify(state);
localStorage.setItem("rea_storage_probe", "frame-updated");
sessionStorage.setItem("rea_storage_probe", "frame-session-updated");
localStorage.removeItem("rea_storage_remove");
sessionStorage.setItem("rea_storage_empty", "");
parent.postMessage({ kind: "rea-storage-frame", state }, "*");
</script></body></html>`;
if (kind === "debugger")
return `<!doctype html><html><body><pre id="debugger-state"></pre><script>
window.reaDebuggerBefore = true;
debugger;
window.reaDebuggerAfter = true;
document.querySelector("#debugger-state").textContent = String(window.reaDebuggerAfter);
</script></body></html>`;
return `<!doctype html><html><body><pre id="top-report"></pre><script>
const state = { local: localStorage.getItem("rea_storage_probe"), session: sessionStorage.getItem("rea_storage_probe"), removed: localStorage.getItem("rea_storage_remove"), empty: sessionStorage.getItem("rea_storage_empty") };
document.querySelector("#top-report").textContent = JSON.stringify(state);
localStorage.setItem("rea_storage_probe", "top-updated");
sessionStorage.setItem("rea_storage_probe", "top-session-updated");
</script></body></html>`;
};
const startStorageSite = async () => {
let port = 0;
const server = createServer((request, response) => {
const path = new URL(request.url ?? "/", "http://127.0.0.1").pathname;
const kind =
path === "/storage-main"
? "main"
: path === "/storage-frame"
? "frame"
: path === "/storage-debugger"
? "debugger"
: path === "/storage-top"
? "top"
: undefined;
if (kind === undefined) {
response.writeHead(404).end();
return;
}
response.setHeader("content-type", "text/html; charset=utf-8");
response.end(storagePage(kind, port));
});
server.listen(0, "::");
await once(server, "listening");
port = server.address().port;
return {
origin: `http://127.0.0.1:${port}`,
close: async () => {
server.close();
// Chrome can leave speculative TCP connections without an HTTP request;
// server.close() alone waits indefinitely for those owned sockets.
server.closeAllConnections();
await once(server, "close");
},
};
};
const domText = (step) => {
assert.equal(step.artifacts.dom.state, "captured");
return step.artifacts.dom.value.text;
};
const assertContainsState = (text, id, expected) => {
const match = new RegExp(`<pre id="${id}"[^>]*>(.*?)</pre>`, "su").exec(text);
assert.ok(match, `Captured DOM omitted #${id}: ${text}`);
assert.deepEqual(JSON.parse(match[1]), expected);
};
/** Verify one-time local/session storage initialization on a real Chromium. */
export async function verifyScenarioStorage({
executable,
endpoint,
targetId,
}) {
const site = await startStorageSite();
try {
await verifyLaunchStorage(executable, site.origin);
await verifyConnectedStorage(endpoint, targetId, site.origin);
return {
mocked: false,
launch: true,
connect: true,
oopifInitialization: true,
firstInlineScriptSeeded: true,
updatesAndDeletesPersistAcrossNavigation: true,
newScenarioGetsFreshSeed: true,
externalTargetSurvives: true,
cancellationCleanup: true,
};
} finally {
await site.close();
}
}
const verifyLaunchStorage = async (executable, origin) => {
const scenario = scenarioFor(
{ mode: "launch", executable_path: executable },
origin,
);
const profilesBefore = await scenarioProfiles();
const result = await runScenarioCli(scenario);
const capture = result.normalized_result;
assert.ok(capture, "Launch-mode storage scenario returned no result");
assertLaunchStorageStates(capture);
assert.equal(capture.browser.cleanup, "terminated-owned-process");
const profilesAfter = await scenarioProfiles();
assert.ok([...profilesAfter].every((profile) => profilesBefore.has(profile)));
const freshScenario = browserScenarioSchema.parse({
...scenario,
actions: [scenario.actions[0]],
});
const fresh = await createBrowserScenarioProvider(
process.env,
).captureScenario(freshScenario);
assert.equal(fresh.ok, true);
if (!fresh.ok) throw fresh.error;
assertContainsState(domText(fresh.value.steps[0]), "main-report", {
local: "main-initial",
session: "main-session-initial",
removed: "main-remove-seed",
empty: "main-empty-seed",
});
};
const assertLaunchStorageStates = (capture) => {
assertContainsState(domText(capture.steps[0]), "main-report", {
local: "main-initial",
session: "main-session-initial",
removed: "main-remove-seed",
empty: "main-empty-seed",
});
assertContainsState(domText(capture.steps[1]), "frame-report", {
local: "frame-initial",
session: "frame-session-initial",
removed: "frame-remove-seed",
empty: "frame-empty-seed",
});
assertContainsState(domText(capture.steps[2]), "top-report", {
local: "frame-updated",
session: "frame-session-updated",
removed: null,
empty: "",
});
assertContainsState(domText(capture.steps[3]), "main-report", {
local: "main-updated",
session: "main-session-updated",
removed: null,
empty: "",
});
assertContainsState(domText(capture.steps[3]), "frame-report", {
local: "top-updated",
session: "top-session-updated",
removed: null,
empty: "",
});
};
const verifyConnectedStorage = async (endpoint, targetId, origin) => {
const scenario = scenarioFor(
{ mode: "connect", cdp_endpoint: endpoint, target_id: targetId },
origin,
);
const targets = await (
await fetch(`${endpoint}/json/list`, { signal: AbortSignal.timeout(2_000) })
).json();
const target = targets.find(({ id }) => id === targetId);
assert.ok(target, "External browser target is missing");
const observer = await CdpConnection.connect(
target.webSocketDebuggerUrl,
"capture_browser_scenario",
);
try {
await assertStorageKeyPartition(observer, origin);
const result = await runScenarioCli(scenario);
const capture = result.normalized_result;
assert.ok(capture, "Connect-mode storage scenario returned no result");
assertContainsState(domText(capture.steps[0]), "main-report", {
local: "main-initial",
session: "main-session-initial",
removed: "main-remove-seed",
empty: "main-empty-seed",
});
assert.equal(capture.browser.cleanup, "disconnected-external");
await verifyCallerDebuggerPause(observer, origin, scenario);
await verifyCancelledStorage(observer, endpoint, targetId, scenario);
} finally {
await raceWithTimeout(restoreTarget(observer, target.url), 1_000).catch(
() => undefined,
);
await raceWithTimeout(observer.close(), 1_000).catch(() => undefined);
}
};
const assertStorageKeyPartition = async (connection, origin) => {
const attachedSessions = new Set();
const unsubscribe = connection.onEvent((event) => {
if (
event.method === "Target.attachedToTarget" &&
typeof event.params === "object" &&
event.params !== null &&
"sessionId" in event.params &&
typeof event.params.sessionId === "string" &&
"targetInfo" in event.params &&
typeof event.params.targetInfo === "object" &&
event.params.targetInfo !== null &&
"type" in event.params.targetInfo &&
event.params.targetInfo.type === "iframe"
)
attachedSessions.add(event.params.sessionId);
});
await connection.send("Page.enable");
await connection.send("Target.setAutoAttach", {
autoAttach: true,
waitForDebuggerOnStart: false,
flatten: true,
});
try {
const start = await connection.send("Runtime.evaluate", {
expression: "performance.timeOrigin",
returnByValue: true,
});
await connection.send("Page.navigate", {
url: `${origin}/storage-main`,
});
await waitForPagePath(connection, "/storage-main", start.result.value);
let childSession;
let childFrameId;
const deadline = Date.now() + 10_000;
while (Date.now() < deadline && childFrameId === undefined) {
for (const sessionId of attachedSessions) {
try {
const tree = await connection.send(
"Page.getFrameTree",
{},
sessionId,
);
const frame = tree.frameTree?.frame;
if (
typeof frame?.id === "string" &&
frame.url ===
`${origin.replace("127.0.0.1", "localhost")}/storage-frame`
) {
childSession = sessionId;
childFrameId = frame.id;
break;
}
} catch {
// The iframe may detach between target discovery and frame inspection.
}
}
if (childFrameId === undefined) await delay(25);
}
assert.ok(
childSession && childFrameId,
"Storage fixture OOPIF was not attached",
);
const child = await connection.send(
"Storage.getStorageKey",
{ frameId: childFrameId },
childSession,
);
const rootTree = await connection.send("Page.getFrameTree");
const beforeRoot = await connection.send("Storage.getStorageKey", {
frameId: rootTree.frameTree.frame.id,
});
await connection.send("Page.navigate", {
url: `${origin.replace("127.0.0.1", "localhost")}/storage-top`,
});
await waitForPagePath(connection, "/storage-top", start.result.value);
const afterTree = await connection.send("Page.getFrameTree");
const topLevel = await connection.send("Storage.getStorageKey", {
frameId: afterTree.frameTree.frame.id,
});
assert.notEqual(
child.storageKey,
topLevel.storageKey,
`Chrome did not expose distinct partition keys: child=${JSON.stringify(child)} top=${JSON.stringify(topLevel)} rootBefore=${JSON.stringify(beforeRoot)}`,
);
} finally {
unsubscribe();
await connection.send("Target.setAutoAttach", {
autoAttach: false,
waitForDebuggerOnStart: false,
flatten: true,
});
}
};
const waitForPagePath = async (connection, path, priorTimeOrigin) => {
const deadline = Date.now() + 10_000;
while (Date.now() < deadline) {
const result = await connection.send("Runtime.evaluate", {
expression: `JSON.stringify({ path: location.pathname, time: performance.timeOrigin })`,
returnByValue: true,
});
const current = JSON.parse(result.result.value);
if (current.path === path && current.time !== priorTimeOrigin) return;
await delay(25);
}
throw new Error(`External browser did not navigate to ${path}`);
};
const verifyCallerDebuggerPause = async (caller, origin, scenario) => {
await caller.send("Runtime.enable");
await caller.send("Debugger.enable");
const controller = new AbortController();
const debuggerUrl = `${origin}/storage-debugger`;
const scripts = new Map();
const protocolTrace = [];
let pauseObserved = false;
let resolvePaused;
const paused = new Promise((resolve) => {
resolvePaused = (callFrameId) => {
pauseObserved = true;
resolve(callFrameId);
};
});
const unsubscribe = caller.onEvent((event) => {
if (event.method === "Debugger.scriptParsed") {
const params = event.params;
if (
typeof params === "object" &&
params !== null &&
"scriptId" in params &&
typeof params.scriptId === "string" &&
"url" in params &&
typeof params.url === "string"
)
scripts.set(params.scriptId, params.url);
if (
typeof params === "object" &&
params !== null &&
"url" in params &&
typeof params.url === "string" &&
params.url.includes("storage")
)
protocolTrace.push({ method: event.method, url: params.url });
} else if (event.method === "Debugger.paused") {
const params = event.params;
const callFrames =
typeof params === "object" && params !== null && "callFrames" in params
? params.callFrames
: undefined;
const frame = Array.isArray(callFrames) ? callFrames[0] : undefined;
const scriptId =
typeof frame === "object" && frame !== null && "location" in frame
? frame.location?.scriptId
: undefined;
protocolTrace.push({
method: event.method,
url: typeof scriptId === "string" ? scripts.get(scriptId) : undefined,
reason:
typeof params === "object" && params !== null && "reason" in params
? params.reason
: undefined,
});
if (
typeof frame === "object" &&
frame !== null &&
typeof frame.callFrameId === "string" &&
typeof scriptId === "string" &&
scripts.get(scriptId) === debuggerUrl
)
resolvePaused(frame.callFrameId);
}
});
const debuggerScenario = browserScenarioSchema.parse({
...scenario,
start_url: { url: debuggerUrl },
actions: [
{ step_id: "after-debugger", action: "wait_for_timeout", duration_ms: 1 },
],
});
let settled = false;
const pending = createBrowserScenarioProvider(process.env)
.captureScenario(debuggerScenario, { signal: controller.signal })
.finally(() => {
settled = true;
unsubscribe();
});
try {
const callFrameId = await raceWithTimeout(paused, 10_000, () => {
throw new Error(
`Caller debugger pause was not observed at ${debuggerUrl}; settled=${String(settled)} events=${JSON.stringify(protocolTrace)}`,
);
});
const pausedState = await caller.send("Debugger.evaluateOnCallFrame", {
callFrameId,
expression: "window.reaDebuggerAfter === true",
returnByValue: true,
});
assert.equal(
pausedState.result.value,
false,
"Caller-owned debugger frame resumed before the caller inspected it",
);
await delay(100);
assert.equal(
settled,
false,
"Storage observer resumed a caller-owned debugger pause",
);
await caller.send("Debugger.resume");
pauseObserved = false;
const result = await pending;
if (!result.ok) throw result.error;
const debuggerState = await caller.send("Runtime.evaluate", {
expression: "window.reaDebuggerAfter === true",
returnByValue: true,
});
assert.equal(debuggerState.result.value, true);
} catch (cause) {
if (!pauseObserved) controller.abort(cause);
else await caller.send("Debugger.resume").catch(() => undefined);
const completion = await raceWithTimeout(
pending.catch((pendingCause) => ({ ok: false, error: pendingCause })),
5_000,
);
if (completion === undefined)
throw new Error("Capture remained pending after cancellation", { cause });
const completionError =
typeof completion === "object" &&
completion !== null &&
"error" in completion
? completion.error
: typeof completion === "object" &&
completion !== null &&
"ok" in completion &&
completion.ok === false
? completion.error
: undefined;
const completionDetail =
completionError === undefined
? "capture did not report an error"
: (() => {
const causes = [];
let current = completionError;
for (
let depth = 0;
depth < 4 && current !== undefined;
depth += 1
) {
causes.push({
name: current.name,
tag: current._tag,
reason: current.reason,
message: current.message,
});
current = current.cause;
}
return JSON.stringify(causes);
})();
throw new Error(`${String(cause)}; capture=${completionDetail}`, {
cause,
});
} finally {
unsubscribe();
await raceWithTimeout(caller.send("Debugger.disable"), 1_000).catch(
() => undefined,
);
}
};
const verifyCancelledStorage = async (
observer,
endpoint,
targetId,
scenario,
) => {
const controller = new AbortController();
const cancelledScenario = browserScenarioSchema.parse({
...scenario,
actions: [
{
step_id: "cancelled-wait",
action: "wait_for",
locator: { kind: "css", selector: "#never-storage-ready" },
state: "visible",
timeout_ms: 30_000,
},
],
});
const priorDocument = await observer.send("Runtime.evaluate", {
expression: "performance.timeOrigin",
returnByValue: true,
});
const pending = createBrowserScenarioProvider(process.env).captureScenario(
cancelledScenario,
{ signal: controller.signal },
);
await waitForNewDocument(
observer,
"/storage-main",
priorDocument.result.value,
);
const state = await observer.send("Runtime.evaluate", {
expression:
"JSON.parse(document.querySelector('#main-report').textContent)",
returnByValue: true,
});
assert.deepEqual(state.result.value, {
local: "main-initial",
session: "main-session-initial",
removed: "main-remove-seed",
empty: "main-empty-seed",
});
controller.abort(new Error("cancel storage seed scenario"));
const result = await pending;
assert.ok(
!result.ok ||
result.value.steps.some(({ status }) => status === "cancelled"),
"Cancellation unexpectedly completed every scenario step",
);
const targets = await (await fetch(`${endpoint}/json/list`)).json();
assert.ok(
targets.some(({ id }) => id === targetId),
"Cancelled connect-mode storage cleanup closed the external target",
);
};
const waitForNewDocument = async (connection, path, priorTimeOrigin) => {
const deadline = Date.now() + 10_000;
while (Date.now() < deadline) {
const result = await connection.send("Runtime.evaluate", {
expression: `JSON.stringify({ path: location.pathname, time: performance.timeOrigin, state: document.querySelector('#main-report')?.textContent })`,
returnByValue: true,
});
const current = JSON.parse(result.result.value);
if (
current.path === path &&
current.time !== priorTimeOrigin &&
current.state !== undefined
)
return;
await new Promise((resolve) => setTimeout(resolve, 25));
}
throw new Error(`External browser did not navigate to ${path}`);
};
const restoreTarget = async (connection, url) => {
await connection.send("Page.navigate", { url });
const deadline = Date.now() + 10_000;
while (Date.now() < deadline) {
const result = await connection.send("Runtime.evaluate", {
expression: "location.href",
returnByValue: true,
});
if (result.result.value === url) return;
await new Promise((resolve) => setTimeout(resolve, 25));
}
throw new Error("External browser did not return to its original URL");
};
+18 -12
View File
@@ -65,7 +65,11 @@ function assertObservationEvents(result) {
throw new Error(
"Real Chrome attach-window console observation was missing",
);
if (result.network.websocket_events.length < 1)
if (
!result.network.websocket_connections.some(
(connection) => connection.events.length > 0,
)
)
throw new Error("Real Chrome WebSocket metadata was missing");
}
@@ -77,8 +81,8 @@ function assertObservationPrivacy(result) {
result.network.requests.some(
(request) => request.body_shapes.status !== "not_approved",
) ||
result.network.websocket_events.some(
(event) => event.payload_shape !== null,
result.network.websocket_connections.some((connection) =>
connection.events.some((event) => event.payload_shape !== null),
)
)
throw new Error(
@@ -246,15 +250,17 @@ export function assertSensitiveShapes(result) {
)
throw new Error("Real Chrome selected console text was not preserved");
if (
!result.network.websocket_events.some(
(event) =>
event.payload_shape?.format === "json" &&
event.payload_shape.json_shape?.properties.some(
({ path }) =>
path.length === 1 &&
path[0].kind === "property" &&
path[0].name === "token",
),
!result.network.websocket_connections.some((connection) =>
connection.events.some(
(event) =>
event.payload_shape?.format === "json" &&
event.payload_shape.json_shape?.properties.some(
({ path }) =>
path.length === 1 &&
path[0].kind === "property" &&
path[0].name === "token",
),
),
)
)
throw new Error("Real Chrome WebSocket JSON shape was missing");
@@ -0,0 +1,28 @@
import { analyzeJavaScriptApplication } from "../../dist/application/javascript/JavaScriptApplicationService.js";
import { ArtifactResourceScope } from "../../dist/artifacts/ArtifactResourceScope.js";
/** Analyze one JavaScript application and close its caller-owned resources. */
export const analyzeJavaScriptApplicationWithOwnedScope = async (input) => {
const scope = new ArtifactResourceScope();
let outcome;
try {
const result = await analyzeJavaScriptApplication(input, scope);
outcome = result.ok
? { kind: "completed", value: result.value }
: { kind: "failed", cause: result.error };
} catch (cause) {
outcome = { kind: "failed", cause };
}
try {
await scope.close();
} catch (cleanupCause) {
if (outcome.kind === "failed")
throw new AggregateError(
[outcome.cause, cleanupCause],
"JavaScript analysis and artifact cleanup both failed",
);
throw cleanupCause;
}
if (outcome.kind === "failed") throw outcome.cause;
return outcome.value;
};
@@ -62,6 +62,36 @@ export async function verifyGhidraAddressBoundaries(client, entry, cli) {
await query("read_bytes", { address: padded, length: 1 }),
await query("read_bytes", { address: entry, length: 1 }),
);
const aliasKeys = [entry, padded];
assert.deepEqual(
await query("set_addresses_names", {
names: {
[entry]: "rea_same_address_alias",
[padded]: "rea_same_address_alias",
},
}),
Object.fromEntries(aliasKeys.map((address) => [address, true])),
"Equivalent address spellings with one requested name share one edit",
);
const nameBeforeAliasedBatch = await query("address_name", {
address: entry,
});
assert.equal(nameBeforeAliasedBatch, "rea_same_address_alias");
assert.deepEqual(
await query("set_addresses_names", {
names: {
[entry]: "rea_first_address_alias",
[padded]: "rea_second_address_alias",
},
}),
Object.fromEntries(aliasKeys.map((address) => [address, false])),
"Conflicting spellings of one Ghidra address must be rejected together",
);
assert.equal(
await query("address_name", { address: entry }),
nameBeforeAliasedBatch,
"A rejected alias collision must leave the preexisting target name unchanged",
);
for (const [command, flags, operation] of [
["read-bytes", ["--length", "1"], "read_bytes"],
[
+15
View File
@@ -58,6 +58,15 @@ export async function verifyGhidraBoundaries(
`${name}: ${JSON.stringify(validate.errors)}`,
);
assert.deepEqual(reply.structuredContent, JSON.parse(mcpTextValue(reply)));
if (name === "list_strings" || name === "search_strings")
assert.ok(
reply.structuredContent.limitations.some((limitation) =>
/Only Ghidra-defined string Data.*empty search does not establish/u.test(
limitation,
),
),
`${name} omitted defined-string coverage`,
);
successfulCalls++;
return result;
};
@@ -378,6 +387,12 @@ export async function verifyGhidraBoundaries(
).some((item) => item.address === marker.address),
false,
);
assert.deepEqual(
await call("search_strings", {
pattern: `REA_GHIDRA_ABSENT_${randomUUID()}`,
}),
[],
);
assert.equal(
(await call("inspect_native_instruction", { address: strings[0].address }))
.status,
+22 -4
View File
@@ -21,7 +21,6 @@ export async function verifyGhidraEntryAliases({
new URL("../../tests/conformance/ghidra/entry-aliases.c", import.meta.url),
);
const path = join(workspace, "entry-aliases");
const prefix = process.platform === "darwin" ? "_" : "";
try {
await exec(compiler, ["-O0", "-g", "-fno-inline", source, "-o", path], {
env,
@@ -29,7 +28,8 @@ export async function verifyGhidraEntryAliases({
});
const bytes = await readFile(path);
await call("close_binary");
await call("open_binary", { path, provider_id: "ghidra" });
const imported = await call("open_binary", { path, provider_id: "ghidra" });
const prefix = imported.format === "mach-o" ? "_" : "";
const names = await call("list_names");
const alias = names.find(
({ value }) => value === `${prefix}rea_entry_alias`,
@@ -46,8 +46,10 @@ export async function verifyGhidraEntryAliases({
);
assert.ok(canonical, "Fixture alias is not at a function entry");
const xrefsSelectors = [];
// CLI xrefs keeps bare hexadecimal selectors as addresses. A Mach-O
// primary label such as "dead" is tested below as a procedure name instead.
for (const value of [
canonical.value,
`${prefix}rea_alias_target`,
alias.value,
`${prefix}rea_xrefs_data`,
`${prefix}rea_xrefs_unreferenced`,
@@ -89,6 +91,18 @@ export async function verifyGhidraEntryAliases({
);
assert.ok(interior);
assert.notEqual(interior.address, alias.address);
assert.equal(
interior.symbol.type,
"label",
"Fixture interior control must remain an imported label",
);
assert.equal(
(await call("list_procedures")).some(
({ address }) => address === interior.address,
),
false,
"Fixture interior control must not be a Ghidra procedure entry",
);
const failure = await reject("procedure_address", {
procedure: interior.value,
});
@@ -153,7 +167,11 @@ export async function verifyGhidraEntryAliases({
{ env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 },
);
const evidence = JSON.parse(stdout);
assert.deepEqual(evidence.normalized_result, expected);
assert.deepEqual(
evidence.normalized_result,
expected,
`CLI xrefs mismatch for ${JSON.stringify(selector)}`,
);
assert.equal(evidence.parameters.address, selector.at(-1));
}
}
+3 -1
View File
@@ -183,7 +183,9 @@ export async function verifyGhidraLargeResults({
assert.equal(dossier.procedure.name, "rea_frame_probe");
assert.ok(dossier.comments.some(({ text }) => text === comment));
assert.ok(
dossierRecord.raw_result.comments.some(({ text }) => text === comment),
dossierRecord.raw_result.value.comments.some(
({ text }) => text === comment,
),
);
const { stdout } = await exec(
process.execPath,
@@ -54,6 +54,15 @@ export async function verifyGhidraNamespaceAnnotations({
);
return { namespace, address: procedure.address };
});
const names = await call("list_names");
const globals = ["alpha", "outer::inner"].map((namespace) => {
const symbol = names.find(
(item) => item.value === `${namespace}::value` && item.symbol.primary,
);
assert.ok(symbol, `Missing demangled fixture global ${namespace}::value`);
return { namespace, address: symbol.address };
});
await verifyDuplicateAddressNames(call, procedures, names);
const ambiguous = await reject("procedure_address", { procedure: "same" });
assert.equal(ambiguous.code, "invalid_request");
for (const { address } of entries)
@@ -125,6 +134,17 @@ export async function verifyGhidraNamespaceAnnotations({
});
assert.deepEqual(unchanged.annotations, qualified.annotations);
}
await verifyAddressNames(call, [entries[0], entries[2]], "function");
await verifyAddressNames(call, globals, "label");
for (const { namespace, address } of entries) {
assert.equal(
await call("set_address_name", {
address,
name: `${namespace}::qualified`,
}),
true,
);
}
const { stdout } = await exec(
process.execPath,
[
@@ -159,6 +179,179 @@ export async function verifyGhidraNamespaceAnnotations({
}
}
async function verifyDuplicateAddressNames(call, procedures, names) {
const functionEntries = [
"rea_duplicate_function_a",
"rea_duplicate_function_b",
].map((name) => {
const procedure = procedures.find(
({ value }) => value === name || value === `_${name}`,
);
assert.ok(procedure, `Missing duplicate-name fixture function ${name}`);
return { address: procedure.address, original: procedure.value };
});
const labelEntries = ["rea_duplicate_label_a", "rea_duplicate_label_b"].map(
(name) => {
const symbol = names.find(
(item) =>
(item.value === name || item.value === `_${name}`) &&
item.symbol.primary,
);
assert.ok(symbol, `Missing duplicate-name fixture label ${name}`);
return { address: symbol.address, original: symbol.value };
},
);
for (const [kind, entries] of [
["function", functionEntries],
["label", labelEntries],
]) {
const [owner, duplicate] = entries;
const name = `rea_duplicate_${kind}`;
assert.equal(
await call("set_address_name", { address: owner.address, name }),
true,
);
assert.equal(
await call("set_address_name", { address: owner.address, name }),
true,
"Renaming the current owner to its existing name remains idempotent",
);
assert.equal(
await call("set_address_name", { address: duplicate.address, name }),
false,
);
assert.equal(
await call("address_name", { address: duplicate.address }),
duplicate.original,
"A rejected duplicate leaves the other address unchanged",
);
const batchName = `rea_batch_duplicate_${kind}`;
assert.deepEqual(
await call("set_addresses_names", {
names: {
[owner.address]: batchName,
[duplicate.address]: batchName,
},
}),
{ [owner.address]: true, [duplicate.address]: false },
);
assert.equal(
await call("address_name", { address: owner.address }),
batchName,
);
assert.equal(
await call("address_name", { address: duplicate.address }),
duplicate.original,
"A failed batch member rolls back only its own address",
);
const owners = (await call("list_names")).filter(
(item) => item.value === batchName && item.symbol.primary,
);
assert.deepEqual(
owners.map(({ address }) => address),
[owner.address],
);
}
}
async function verifyAddressNames(call, symbols, kind) {
for (const { namespace, address } of symbols) {
assert.equal(
await call("set_address_name", {
address,
name: `address_renamed_${kind}`,
}),
true,
);
const qualified = `${namespace}::address_qualified_${kind}`;
assert.equal(
await call("set_address_name", { address, name: qualified }),
true,
);
assert.equal(await call("address_name", { address }), qualified);
assert.equal(
await call("set_address_name", {
address,
name: `other::literal_${kind}`,
}),
true,
);
const literal = await call("address_name", { address });
assert.equal(literal, `${namespace}::other::literal_${kind}`);
assert.equal(
await call("set_address_name", { address, name: literal }),
true,
);
assert.equal(await call("address_name", { address }), literal);
assert.equal(
await call("set_address_name", { address, name: qualified }),
true,
);
for (let repeat = 0; repeat < 2; repeat++) {
const readback = await call("address_name", { address });
assert.deepEqual(
await call("set_addresses_names", { names: { [address]: readback } }),
{ [address]: true },
);
assert.equal(await call("address_name", { address }), qualified);
}
assert.equal(
await call("set_address_name", { address, name: `${namespace}::` }),
false,
);
const session = await call("binary_session");
for (const operation of [
"set_address_name",
"set_addresses_names",
"address_name",
"list_names",
])
assert.equal(
session.capabilities.find((item) => item.operation === operation)
?.available,
true,
`An invalid name must not mark ${operation} unavailable`,
);
assert.equal(await call("address_name", { address }), qualified);
}
const [valid, invalid] = symbols;
assert.deepEqual(
await call("set_addresses_names", {
names: {
[valid.address]: `${valid.namespace}::batch_renamed_${kind}`,
[invalid.address]: `${invalid.namespace}::`,
},
}),
{ [valid.address]: true, [invalid.address]: false },
);
assert.equal(
await call("address_name", { address: valid.address }),
`${valid.namespace}::batch_renamed_${kind}`,
);
assert.equal(
await call("address_name", { address: invalid.address }),
`${invalid.namespace}::address_qualified_${kind}`,
);
const names = await call("list_names");
for (const { namespace, address } of symbols) {
const expected =
address === valid.address
? `${namespace}::batch_renamed_${kind}`
: `${namespace}::address_qualified_${kind}`;
assert.ok(
names.some(
(item) =>
item.address === address &&
item.value === expected &&
item.symbol.primary,
),
`Missing refreshed primary symbol ${expected}`,
);
}
}
async function requireNamespaceDemangler(env) {
const inspection = inspectGhidraInstallation({
environment: process.env,
+149
View File
@@ -0,0 +1,149 @@
import { z } from "zod";
import {
releaseAuthorSchema as author,
releaseIssueSchema as issue,
releasePullRequestSchema,
} from "./release-notes.mjs";
const pageInfo = z.object({
hasNextPage: z.boolean(),
endCursor: z.string().nullable(),
});
const connection = (node) => z.object({ nodes: z.array(node), pageInfo });
const associated = z.object({
number: z.number().int().positive(),
mergeCommit: z.object({ oid: z.string() }).nullable(),
});
const pr = releasePullRequestSchema.omit({ issues: true }).extend({
closingIssuesReferences: connection(issue),
});
const PAGE = "pageInfo { hasNextPage endCursor }";
const ACTOR = "login __typename";
const AUTHORS = `authors(first:100, after:$cursor) { nodes { name email user { ${ACTOR} } } ${PAGE} }`;
const ASSOCIATED = `associatedPullRequests(first:100, after:$cursor) { nodes { number mergeCommit { oid } } ${PAGE} }`;
const ISSUES = `closingIssuesReferences(first:100, after:$cursor) { nodes { number url author { ${ACTOR} } repository { nameWithOwner } } ${PAGE} }`;
async function query(run, repository, selection, cursor = null) {
const [owner, name] = repository.split("/");
const response = JSON.parse(
await run(
"gh",
["api", "graphql", "--input", "-"],
JSON.stringify({
query: `query($owner:String!,$name:String!,$cursor:String) { repository(owner:$owner,name:$name) { ${selection} } }`,
variables: { owner, name, cursor },
}),
),
);
if (response.errors?.length)
throw new Error(
`GitHub inventory query failed: ${JSON.stringify(response.errors)}`,
);
if (!response.data?.repository)
throw new Error(`GitHub repository unavailable: ${repository}`);
return response.data.repository;
}
async function remainingPages(run, repository, selection, field) {
const initial = field.initial;
const nodes = [...initial.nodes];
let page = initial;
const cursors = new Set();
while (page.pageInfo.hasNextPage) {
const cursor = page.pageInfo.endCursor;
if (!cursor || cursors.has(cursor))
throw new Error(`GitHub returned a non-advancing ${field.name} cursor`);
cursors.add(cursor);
const response = await query(run, repository, selection, cursor);
page = field.schema.parse(response.item?.[field.name]);
nodes.push(...page.nodes);
}
return nodes;
}
/** GitHub's commit authors include co-authors; unresolved identities remain explicit. */
export async function collectGitHubCredits(run, repository, commits) {
const shipped = new Set(commits.map((commit) => commit.sha));
const numbers = new Set();
const result = [];
// Small query batches limit transient GraphQL work, never the history examined.
for (let offset = 0; offset < commits.length; offset += 20) {
const batch = commits.slice(offset, offset + 20);
const selection = batch
.map(
(commit, index) =>
`c${index}: object(oid:${JSON.stringify(commit.sha)}) { ... on Commit { oid ${AUTHORS} ${ASSOCIATED} } }`,
)
.join("\n");
const response = await query(run, repository, selection);
for (const [index, commit] of batch.entries()) {
const value = z
.object({
oid: z.literal(commit.sha),
authors: connection(author),
associatedPullRequests: connection(associated),
})
.parse(response[`c${index}`]);
const select = (fields) =>
`item: object(oid:${JSON.stringify(commit.sha)}) { ... on Commit { ${fields} } }`;
const authors = await remainingPages(run, repository, select(AUTHORS), {
name: "authors",
schema: connection(author),
initial: value.authors,
});
const associations = await remainingPages(
run,
repository,
select(ASSOCIATED),
{
name: "associatedPullRequests",
schema: connection(associated),
initial: value.associatedPullRequests,
},
);
const pullRequests = [
...new Set(
associations
.filter((item) => shipped.has(item.mergeCommit?.oid))
.map((item) => item.number),
),
];
for (const number of pullRequests) numbers.add(number);
result.push({ ...commit, authors, pullRequests });
}
}
const pullRequests = [];
const sorted = [...numbers].sort((a, b) => a - b);
for (let offset = 0; offset < sorted.length; offset += 20) {
const batch = sorted.slice(offset, offset + 20);
const fields = `number title url author { ${ACTOR} } mergeCommit { oid } ${ISSUES}`;
const response = await query(
run,
repository,
batch
.map(
(number, index) =>
`p${index}: pullRequest(number:${number}) { ${fields} }`,
)
.join("\n"),
);
for (const [index, number] of batch.entries()) {
const value = pr.parse(response[`p${index}`]);
if (value.number !== number || !shipped.has(value.mergeCommit.oid))
throw new Error(`PR #${number} changed outside the selected history`);
const issues = await remainingPages(
run,
repository,
`item: pullRequest(number:${number}) { ${ISSUES} }`,
{
name: "closingIssuesReferences",
schema: connection(issue),
initial: value.closingIssuesReferences,
},
);
const { closingIssuesReferences: _connection, ...metadata } = value;
pullRequests.push({ ...metadata, issues });
}
}
return { commits: result, pullRequests };
}
+118
View File
@@ -0,0 +1,118 @@
import { valid } from "semver";
import { z } from "zod";
import {
releaseInventorySchema,
verifyInventoryHistory,
verifyReleaseNotes,
} from "./release-notes.mjs";
const sha = z.string().regex(/^[a-f0-9]{40}$/u);
const repository = z.string().regex(/^[\w.-]+\/[\w.-]+$/u);
const pullRequest = z.object({
number: z.number().int().positive(),
body: z.string(),
merged_at: z.string().nullable(),
merge_commit_sha: sha.nullable(),
head: z.object({
ref: z.string(),
repo: z.object({ full_name: z.string() }).nullable(),
}),
labels: z.array(z.object({ name: z.string() })),
});
/** Read the reviewed Git snapshot and live PR before Release Please can mutate tags. */
export async function verifyNotesPublication(run, options) {
const repo = repository.parse(options.repository);
const source = sha.parse(
(
await run("git", [
"rev-parse",
"--verify",
"--end-of-options",
`${options.source}^{commit}`,
])
).trim(),
);
const read = (path) => run("git", ["show", `${source}:${path}`]);
const { version } = z
.object({ version: z.string() })
.parse(JSON.parse(await read("package.json")));
if (valid(version) !== version || version.includes("+"))
throw new Error("Invalid reviewed package version");
let candidates;
if (options.pr !== undefined) {
const number = z.coerce.number().int().positive().parse(options.pr);
candidates = [
pullRequest.parse(
JSON.parse(await run("gh", ["api", `repos/${repo}/pulls/${number}`])),
),
];
} else {
candidates = z
.array(z.array(pullRequest))
.parse(
JSON.parse(
await run("gh", [
"api",
"--paginate",
"--slurp",
`repos/${repo}/commits/${source}/pulls?per_page=100`,
]),
),
)
.flat();
}
const matching = candidates.filter(
(pr) =>
pr.merged_at !== null &&
pr.merge_commit_sha === source &&
pr.head.repo?.full_name.toLowerCase() === repo.toLowerCase() &&
pr.head.ref.startsWith("release-please--branches--"),
);
if (matching.length !== 1)
throw new Error(
"Expected exactly one merged Release Please PR for the reviewed source",
);
const pr = matching[0];
const path = `docs/releases/${version}.contributions.json`;
const present =
(
await run("git", ["ls-tree", "--name-only", source, "--", path])
).trim() === path;
if (!present) {
if (pr.labels.some((label) => label.name === "rea:release-notes-finalized"))
throw new Error(
"Finalized release PR is missing its contribution inventory",
);
return { status: "legacy", version, pullRequest: pr.number };
}
const inventory = releaseInventorySchema.parse(JSON.parse(await read(path)));
if (
inventory.version !== version ||
inventory.repository.toLowerCase() !== repo.toLowerCase()
)
throw new Error(
"Contribution inventory differs from the reviewed release version or repository",
);
const baseline = z
.object({ ".": z.string() })
.parse(
JSON.parse(
await run("git", ["show", `${source}^1:.release-please-manifest.json`]),
),
)["."];
if (valid(baseline) !== baseline || baseline.includes("+"))
throw new Error("Invalid release baseline version");
const baselineSha = (
await run("git", ["rev-parse", `refs/tags/rea-agents-${baseline}^{commit}`])
).trim();
if (inventory.base !== baselineSha)
throw new Error(
"Contribution inventory does not start at the published baseline tag",
);
await verifyInventoryHistory(run, inventory, source);
return {
status: "verified",
...verifyReleaseNotes(inventory, await read("CHANGELOG.md"), pr.body),
};
}
+442
View File
@@ -0,0 +1,442 @@
import { valid } from "semver";
import { z } from "zod";
const sha = z.string().regex(/^[a-f0-9]{40}$/u);
const actor = z.object({
login: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9-]*(?:\[bot\])?$/u),
__typename: z.string(),
});
const author = z.object({
name: z.string(),
email: z.string(),
user: actor.nullable(),
});
const commit = z.object({
sha,
subject: z.string(),
body: z.string(),
parents: z.array(sha),
authors: z.array(author),
pullRequests: z.array(z.number().int().positive()),
});
const issue = z.object({
number: z.number().int().positive(),
url: z.url(),
author: actor.nullable(),
repository: z.object({ nameWithOwner: z.string() }),
});
const pullRequest = z.object({
number: z.number().int().positive(),
title: z.string(),
url: z.url(),
author: actor.nullable(),
mergeCommit: z.object({ oid: sha }),
issues: z.array(issue),
});
export {
author as releaseAuthorSchema,
issue as releaseIssueSchema,
pullRequest as releasePullRequestSchema,
};
export const releaseInventorySchema = z.object({
schemaVersion: z.literal(1),
repository: z.string().regex(/^[\w.-]+\/[\w.-]+$/u),
version: z
.string()
.refine(
(value) => valid(value) === value && !value.includes("+"),
"Expected exact release SemVer",
),
base: sha,
target: sha,
commits: z.array(commit),
pullRequests: z.array(pullRequest),
});
export async function readReleaseHistory(run, base, target) {
if (
(await run("git", ["rev-parse", "--is-shallow-repository"])).trim() !==
"false"
)
throw new Error(
"Release inventory requires complete Git history; fetch --unshallow first",
);
const resolve = async (ref) =>
sha.parse(
(
await run("git", [
"rev-parse",
"--verify",
"--end-of-options",
`${ref}^{commit}`,
])
).trim(),
);
const baseSha = await resolve(base);
const targetSha = await resolve(target);
await run("git", ["merge-base", "--is-ancestor", baseSha, targetSha]);
const text = await run("git", [
"log",
"--reverse",
"--topo-order",
"-z",
"--format=%H%x00%s%x00%B%x00%P",
`${baseSha}..${targetSha}`,
]);
const fields = text.split("\0");
if (fields.pop() !== "" || fields.length % 4)
throw new Error("Malformed Git release history");
const commits = [];
for (let index = 0; index < fields.length; index += 4) {
commits.push({
sha: sha.parse(fields[index]),
subject: fields[index + 1],
body: fields[index + 2],
parents: fields[index + 3].split(" ").filter(Boolean),
});
}
return { base: baseSha, target: targetSha, commits };
}
// REA history maps Anthropic and Cursor's agent trailers to ordinary User actors.
// Keep those source identities in the inventory while excluding automation credit.
const AGENT_ACCOUNTS = new Set(["claude", "cursoragent"]);
// REA's maintainer requested external-contributor thanks without self-credit.
const MAINTAINER_ACCOUNTS = new Set(["morluto"]);
const human = (value) =>
value?.__typename === "User" &&
!value.login.endsWith("[bot]") &&
!AGENT_ACCOUNTS.has(value.login.toLowerCase()) &&
!MAINTAINER_ACCOUNTS.has(value.login.toLowerCase())
? value.login
: undefined;
const unique = (values) =>
[...new Set(values.filter(Boolean))].sort((a, b) =>
a.toLowerCase().localeCompare(b.toLowerCase()),
);
function prCredits(inventory, pr) {
return unique([
human(pr.author),
...pr.issues.map((item) => human(item.author)),
...inventory.commits
.filter(
(item) =>
item.parents.length === 1 && item.pullRequests.includes(pr.number),
)
.flatMap((item) => item.authors.map((identity) => human(identity.user))),
]);
}
function thanks(handles) {
return handles.length
? ` Thanks ${handles.map((handle) => `@${handle}`).join(", ")}.`
: "";
}
export function contributionRecord(raw) {
const inventory = releaseInventorySchema.parse(raw);
const lines = [
"### Complete contribution record",
"",
`History: \`${inventory.base}..${inventory.target}\`; ${inventory.pullRequests.length} merged PRs, ${inventory.commits.length} commits examined.`,
"",
];
for (const pr of inventory.pullRequests) {
const issues = pr.issues
.map(
(item) =>
`[${item.repository.nameWithOwner}#${item.number}](${item.url})`,
)
.join(", ");
lines.push(
`- **PR [#${pr.number}](${pr.url})**${issues ? ` Related ${issues}.` : ""}${thanks(prCredits(inventory, pr))}`,
);
}
const direct = inventory.commits.filter(
(item) => item.pullRequests.length === 0 && item.parents.length <= 1,
);
if (direct.length) {
lines.push("", "#### Direct contributions", "");
for (const item of direct)
lines.push(
`- [${item.sha.slice(0, 7)}](https://github.com/${inventory.repository}/commit/${item.sha})${thanks(unique(item.authors.map((identity) => human(identity.user))))}`,
);
}
return `${lines.join("\n")}\n`;
}
/** Keep the full authorship snapshot available without duplicating public thanks. */
export function contributionSummary(raw) {
const inventory = releaseInventorySchema.parse(raw);
return `Full contribution record: [${inventory.commits.length} commits and ${inventory.pullRequests.length} merged PR${inventory.pullRequests.length === 1 ? "" : "s"}](https://github.com/${inventory.repository}/blob/rea-agents-${inventory.version}/docs/releases/${inventory.version}.contributions.json).\n`;
}
export function selectedReleaseNotes(text, version) {
const headings = [...text.matchAll(/^## \[([^\]]+)\].*$/gmu)];
if (headings[0]?.[1] !== version)
throw new Error(`Notes must start with release [${version}]`);
return text
.slice(headings[0].index, headings[1]?.index ?? text.length)
.trim();
}
// Notes have one Markdown list item per line; fenced examples are never credit sources.
function proseBullets(text) {
let fence;
return text.split("\n").filter((line) => {
const marker = /^\s*(`{3,}|~{3,})/u.exec(line)?.[1];
if (marker) {
if (!fence) fence = marker;
else if (marker[0] === fence[0] && marker.length >= fence.length)
fence = undefined;
return false;
}
return !fence && /^[-*] /u.test(line);
});
}
function bulletPrs(bullet, repository) {
// Resolve explicit links first so cross-repository numbers cannot acquire local credit.
const local = [];
const withoutLinks = bullet.replace(
/\[([^\]]*)\]\(https:\/\/github\.com\/([^/]+\/[^/]+)\/(?:pull|issues)\/(\d+)\)/gu,
(_match, _label, repo, number) => {
if (repo.toLowerCase() === repository.toLowerCase())
local.push(Number(number));
return "";
},
);
for (const match of withoutLinks.matchAll(/(?<![\w/])#(\d+)\b/gu))
local.push(Number(match[1]));
return new Set(local);
}
function requiredCredits(inventory, bullet, references) {
const required = [];
for (const pr of inventory.pullRequests.filter((item) =>
references.has(item.number),
)) {
for (const handle of prCredits(inventory, pr))
required.push({ handle, context: `PR #${pr.number}` });
}
for (const issue of inventory.pullRequests.flatMap((item) => item.issues)) {
const mentioned =
(issue.repository.nameWithOwner.toLowerCase() ===
inventory.repository.toLowerCase() &&
references.has(issue.number)) ||
bullet.includes(`](${issue.url})`);
const handle = human(issue.author);
if (mentioned && handle)
required.push({
handle,
context: `issue ${issue.repository.nameWithOwner}#${issue.number}`,
});
}
for (const match of bullet.matchAll(
/https:\/\/github\.com\/([^/]+\/[^/]+)\/commit\/([a-f0-9]{7,40})\b/gu,
)) {
if (match[1].toLowerCase() !== inventory.repository.toLowerCase()) continue;
const matches = inventory.commits.filter((item) =>
item.sha.startsWith(match[2]),
);
if (matches.length !== 1)
throw new Error(`Unknown or ambiguous release commit ${match[2]}`);
if (matches[0].parents.length > 1) continue;
for (const identity of matches[0].authors) {
const handle = human(identity.user);
if (handle)
required.push({ handle, context: `commit ${matches[0].sha}` });
}
}
return required;
}
function verifyEditorialCredits(inventory, prose) {
const known = new Set(inventory.pullRequests.map((pr) => pr.number));
const issues = new Set(
inventory.pullRequests.flatMap((pr) =>
pr.issues
.filter(
(item) =>
item.repository.nameWithOwner.toLowerCase() ===
inventory.repository.toLowerCase(),
)
.map((item) => item.number),
),
);
const eligible = new Set(
unique([
...inventory.pullRequests.flatMap((pr) => prCredits(inventory, pr)),
...inventory.commits.flatMap((item) =>
item.authors.map((identity) => human(identity.user)),
),
]).map((handle) => handle.toLowerCase()),
);
const errors = [];
const requiredByContext = new Map();
const thankedByContext = new Map();
for (const bullet of proseBullets(prose)) {
const references = bulletPrs(bullet, inventory.repository);
const start = bullet.indexOf("Thanks ");
const thanked = new Set(
[
...bullet
.slice(start < 0 ? bullet.length : start)
.matchAll(/@([A-Za-z0-9-]+(?:\[bot\])?)/gu),
].map((match) => match[1].toLowerCase()),
);
for (const number of references)
if (!known.has(number) && !issues.has(number))
errors.push(`Unknown release reference #${number}`);
for (const { handle, context } of requiredCredits(
inventory,
bullet,
references,
)) {
const required = requiredByContext.get(context) ?? new Set();
required.add(handle);
requiredByContext.set(context, required);
const credited = thankedByContext.get(context) ?? new Set();
for (const login of thanked) credited.add(login);
thankedByContext.set(context, credited);
}
for (const handle of thanked)
if (!eligible.has(handle))
errors.push(`Unverified human credit @${handle}`);
}
for (const [context, required] of requiredByContext)
for (const handle of required)
if (!thankedByContext.get(context)?.has(handle.toLowerCase()))
errors.push(`Missing Thanks @${handle} for ${context}`);
return errors;
}
export function verifyReleaseNotes(raw, text, body) {
const inventory = releaseInventorySchema.parse(raw);
const notes = selectedReleaseNotes(text, inventory.version);
// Release Please's inline issue/PR body limit (pull-request-overflow-handler).
if (notes.length > 65_536 || (body !== undefined && body.length > 65_536))
throw new Error(
"Release notes exceed GitHub's inline PR body limit; overflow-link bodies are not supported by this checker",
);
if (notes.length > 125_000 || Buffer.byteLength(notes) > 125_000)
throw new Error(
"Release notes exceed GitHub's body budget; shorten prose without dropping credit",
);
const record = contributionRecord(inventory).trim();
const summary = contributionSummary(inventory).trim();
const start = notes.endsWith(summary)
? notes.length - summary.length
: notes.indexOf("### Complete contribution record");
if (
start < 0 ||
(notes.slice(start).trim() !== record &&
notes.slice(start).trim() !== summary)
)
throw new Error(
"Contribution record differs from inventory; regenerate it with release-notes record",
);
const prose = notes.slice(0, start);
const errors = verifyEditorialCredits(inventory, prose);
for (const heading of ["Highlights", "Changes", "Fixes"])
if (!prose.includes(`### ${heading}\n`))
errors.push(`Missing ${heading} section`);
if (body !== undefined) {
const bodyNotes = selectedReleaseNotes(body, inventory.version)
.replace(
/\n---\nThis PR was generated with \[Release Please\][\s\S]*$/u,
"",
)
.trim();
if (bodyNotes !== notes)
errors.push("Release PR body differs from CHANGELOG.md");
}
if (errors.length) throw new Error(errors.join("\n"));
return {
version: inventory.version,
pullRequests: inventory.pullRequests.length,
commits: inventory.commits.length,
unresolvedAuthors: inventory.commits.flatMap((item) =>
item.authors
.filter((identity) => identity.user === null)
.map((identity) => ({ sha: item.sha, ...identity })),
),
};
}
/** Reject stale inventories and product changes made after the editorial target. */
export async function verifyInventoryHistory(run, raw, source) {
const inventory = releaseInventorySchema.parse(raw);
const history = await readReleaseHistory(
run,
inventory.base,
inventory.target,
);
const original = inventory.commits.map(
({ authors: _authors, pullRequests: _prs, ...item }) => item,
);
if (JSON.stringify(history.commits) !== JSON.stringify(original))
throw new Error("Inventory does not match complete local Git history");
const shas = new Set(history.commits.map((item) => item.sha));
const numbers = new Set();
for (const pr of inventory.pullRequests) {
if (numbers.has(pr.number) || !shas.has(pr.mergeCommit.oid))
throw new Error(`Duplicate or out-of-range PR #${pr.number}`);
numbers.add(pr.number);
}
for (const item of inventory.commits)
for (const number of item.pullRequests)
if (!numbers.has(number))
throw new Error(`Commit refers to missing PR #${number}`);
if (source !== undefined) {
const resolved = sha.parse(
(
await run("git", [
"rev-parse",
"--verify",
"--end-of-options",
`${source}^{commit}`,
])
).trim(),
);
const parent = (await run("git", ["rev-parse", `${resolved}^1`])).trim();
if (parent !== inventory.target)
throw new Error(
"Release base moved after notes target; regenerate inventory for the reviewed merge's first parent",
);
await run("git", [
"merge-base",
"--is-ancestor",
inventory.target,
resolved,
]);
const allowed = new Set([
"CHANGELOG.md",
".release-please-manifest.json",
"package.json",
"package-lock.json",
"server.json",
"src/generatedPackageMetadata.ts",
"docs/installation.md",
`docs/releases/${inventory.version}.contributions.json`,
]);
const paths = (
await run("git", [
"diff",
"--name-only",
"--no-renames",
"-z",
inventory.target,
resolved,
])
)
.split("\0")
.filter(Boolean);
const product = paths.filter((path) => !allowed.has(path));
if (product.length)
throw new Error(
`Product changed after notes target; regenerate inventory: ${product.join(", ")}`,
);
}
}
+16 -1
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env node
import { access } from "node:fs/promises";
import { dirname, resolve } from "node:path";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
// Route production MCP before importing Incur. Incur owns registration helpers
@@ -15,6 +15,21 @@ const isMcpMode =
args.length === 1 && (args[0] === "--mcp" || args[0] === "mcp");
const isMcpDoctorMode = args[0] === "mcp" && args[1] === "doctor";
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
// Playwright's registry assumes one of its three desktop host platforms when
// it chooses the browser cache directory. Termux reports `android`, although
// its Node runtime and filesystem are POSIX-compatible. Keep the cache path
// explicit so REA can start on Termux. Browser launch still requires a
// caller-supplied Android-compatible Chromium executable.
if (
process.platform === "android" &&
process.env.PLAYWRIGHT_BROWSERS_PATH === undefined
) {
const cacheRoot =
process.env.XDG_CACHE_HOME ?? join(process.env.HOME ?? ".", ".cache");
process.env.PLAYWRIGHT_BROWSERS_PATH = join(cacheRoot, "ms-playwright");
}
const runtimeFiles = isMcpMode
? ["dist/main.js"]
: isMcpDoctorMode
+140
View File
@@ -0,0 +1,140 @@
#!/usr/bin/env node
import { execFile } from "node:child_process";
import { readFile, writeFile } from "node:fs/promises";
import { parseArgs, promisify } from "node:util";
import { collectGitHubCredits } from "./lib/release-notes-github.mjs";
import { verifyNotesPublication } from "./lib/release-notes-publication.mjs";
import {
contributionRecord,
contributionSummary,
readReleaseHistory,
releaseInventorySchema,
selectedReleaseNotes,
verifyInventoryHistory,
verifyReleaseNotes,
} from "./lib/release-notes.mjs";
const execFileAsync = promisify(execFile);
async function run(command, args, input) {
const pending = execFileAsync(command, args, {
encoding: "utf8",
maxBuffer: 64 * 1024 * 1024,
});
pending.child.stdin.end(input);
return (await pending).stdout;
}
async function createInventory(values) {
if (!values.repository || !values.base || !values.target || !values.version)
throw new Error(
"inventory requires --repository, --base, --target and --version",
);
// Parse caller-selected boundaries before making network requests.
const history = await readReleaseHistory(run, values.base, values.target);
releaseInventorySchema.omit({ commits: true, pullRequests: true }).parse({
schemaVersion: 1,
repository: values.repository,
version: values.version,
base: history.base,
target: history.target,
});
const credits = await collectGitHubCredits(
run,
values.repository,
history.commits,
);
const inventory = releaseInventorySchema.parse({
schemaVersion: 1,
repository: values.repository,
version: values.version,
base: history.base,
target: history.target,
...credits,
});
await verifyInventoryHistory(run, inventory);
await writeFile(values.manifest, `${JSON.stringify(inventory, null, 2)}\n`, {
flag: "wx",
});
process.stdout.write(
`Inventoried ${inventory.commits.length} commits and ${inventory.pullRequests.length} merged PRs. Saved ${values.manifest}\n`,
);
}
async function main() {
const { values, positionals } = parseArgs({
allowPositionals: true,
options: {
repository: { type: "string" },
pr: { type: "string" },
base: { type: "string" },
target: { type: "string" },
version: { type: "string" },
manifest: { type: "string" },
notes: { type: "string", default: "CHANGELOG.md" },
"pr-body": { type: "string" },
source: { type: "string" },
output: { type: "string" },
summary: { type: "boolean", default: false },
},
});
const [command] = positionals;
if (command === "publication" && positionals.length === 1) {
if (!values.repository || !values.source)
throw new Error(
"publication requires --repository and --source; --pr selects the merged release PR",
);
process.stdout.write(
`${JSON.stringify(await verifyNotesPublication(run, values), null, 2)}\n`,
);
return;
}
if (
positionals.length !== 1 ||
!["inventory", "record", "check", "render"].includes(command) ||
!values.manifest
)
throw new Error(
"Usage: release-notes inventory|record|check|render --manifest FILE [--repository OWNER/REPO --base TAG --target SHA --version VERSION] [--notes FILE --pr-body FILE --source SHA --output FILE] [--summary for record]",
);
if (command === "inventory") {
await createInventory(values);
return;
}
const inventory = releaseInventorySchema.parse(
JSON.parse(await readFile(values.manifest, "utf8")),
);
await verifyInventoryHistory(run, inventory, values.source);
if (command === "record") {
const record = values.summary
? contributionSummary(inventory)
: contributionRecord(inventory);
if (values.output)
await writeFile(values.output, record, {
flag: "wx",
});
else process.stdout.write(record);
return;
}
const notes = await readFile(values.notes, "utf8");
const body = values["pr-body"]
? await readFile(values["pr-body"], "utf8")
: undefined;
const report = verifyReleaseNotes(inventory, notes, body);
if (command === "check") {
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
return;
}
if (!values.output)
throw new Error(
"render requires --output (an existing file is never overwritten)",
);
const rendered = selectedReleaseNotes(notes, inventory.version);
await writeFile(values.output, `${rendered}\n`, { flag: "wx" });
}
await main().catch((error) => {
process.stderr.write(
`Release notes failed: ${error instanceof Error ? error.message : String(error)}\n`,
);
process.exitCode = 1;
});
+481
View File
@@ -0,0 +1,481 @@
// Run the website's browser scripts against a small document model, so missing
// page-owned messages are checked without a browser.
import assert from "node:assert/strict";
import { randomUUID } from "node:crypto";
import { test } from "node:test";
import { setImmediate as nextTurn } from "node:timers/promises";
import { pathToFileURL } from "node:url";
const ASSETS = new URL("../website/public/assets/", import.meta.url);
class Element {
constructor(tag, attributes = {}, children = [], text = "") {
this.tagName = tag.toUpperCase();
this.attributes = new Map(Object.entries(attributes));
this.children = [];
this.parent = null;
this.listeners = new Map();
this.ownText = text;
this.hidden = "hidden" in attributes;
this.disabled = "disabled" in attributes;
this.checked = "checked" in attributes;
this.value = attributes.value ?? "";
for (const child of children) this.append(child);
}
append(child) {
child.parent = this;
this.children.push(child);
}
get textContent() {
return (
this.ownText + this.children.map((child) => child.textContent).join("")
);
}
set textContent(value) {
this.ownText = String(value);
this.children = [];
}
get id() {
return this.getAttribute("id") ?? "";
}
get lang() {
return this.getAttribute("lang") ?? "";
}
get dataset() {
const values = {};
for (const [name, value] of this.attributes) {
if (!name.startsWith("data-")) continue;
const key = name
.slice(5)
.replace(/-([a-z])/g, (_, letter) => letter.toUpperCase());
values[key] = value;
}
return values;
}
getAttribute(name) {
return this.attributes.has(name) ? this.attributes.get(name) : null;
}
setAttribute(name, value) {
this.attributes.set(name, String(value));
}
removeAttribute(name) {
this.attributes.delete(name);
}
addEventListener(type, listener) {
this.listeners.set(type, [...(this.listeners.get(type) ?? []), listener]);
}
dispatch(type, event = {}) {
for (const listener of this.listeners.get(type) ?? []) {
listener({ preventDefault() {}, repeat: false, ...event });
}
}
click() {
this.dispatch("click");
}
focus() {}
descendants() {
return this.children.flatMap((child) => [child, ...child.descendants()]);
}
matches(selector) {
return selector
.split(",")
.some((part) => matchesComplex(this, part.trim()));
}
closest(selector) {
for (let node = this; node !== null; node = node.parent) {
if (node.matches(selector)) return node;
}
return null;
}
querySelectorAll(selector) {
return this.descendants().filter((node) => node.matches(selector));
}
querySelector(selector) {
return this.querySelectorAll(selector)[0] ?? null;
}
getContext() {
return this.context ?? null;
}
}
// Supports the selector forms the website scripts use: tag, #id, .class and
// [attribute], [attribute="value"] or [attribute^="value"], joined by spaces.
function matchesCompound(node, compound) {
const pattern =
/^([a-z0-9]*)((?:#[\w-]+|\.[\w-]+|\[[\w-]+(?:\^?="[^"]*")?\])*)$/i;
const parsed = pattern.exec(compound);
if (parsed === null) throw new Error(`Unsupported selector: ${compound}`);
const [, tag, rest] = parsed;
if (tag && node.tagName !== tag.toUpperCase()) return false;
for (const [, id, className, name, operator, value] of rest.matchAll(
/#([\w-]+)|\.([\w-]+)|\[([\w-]+)(?:(\^?=)"([^"]*)")?\]/g,
)) {
if (id !== undefined) {
if (node.id !== id) return false;
continue;
}
if (className !== undefined) {
const classes = (node.getAttribute("class") ?? "").split(/\s+/);
if (!classes.includes(className)) return false;
continue;
}
const actual = node.getAttribute(name);
if (actual === null) return false;
if (operator === "=" && actual !== value) return false;
if (operator === "^=" && !actual.startsWith(value)) return false;
}
return true;
}
function matchesComplex(node, selector) {
const parts = selector.split(/\s+/);
if (!matchesCompound(node, parts.at(-1))) return false;
let ancestor = node.parent;
for (const part of parts.slice(0, -1).reverse()) {
while (ancestor !== null && !matchesCompound(ancestor, part))
ancestor = ancestor.parent;
if (ancestor === null) return false;
ancestor = ancestor.parent;
}
return true;
}
const h = (tag, attributes, ...children) => {
const text = children.filter((child) => typeof child === "string").join("");
return new Element(
tag,
attributes ?? {},
children.filter((child) => child instanceof Element),
text,
);
};
// Install a fresh document and browser globals, then run one script.
async function run(
script,
body,
{ lang = "en", clipboard = "resolve", stars = 1234 } = {},
) {
const root = h("html", { lang }, h("body", {}, ...body));
const timers = [];
const written = [];
const canvasText = [];
for (const canvas of root.querySelectorAll("canvas")) {
canvas.context = new Proxy(
{ fillText: (text) => canvasText.push(text) },
{ get: (target, name) => target[name] ?? (() => {}), set: () => true },
);
}
const document = {
documentElement: root,
hidden: false,
getElementById: (id) =>
root.descendants().find((node) => node.id === id) ?? null,
querySelector: (selector) => root.querySelector(selector),
querySelectorAll: (selector) => root.querySelectorAll(selector),
createRange: () => ({ selectNodeContents() {} }),
addEventListener() {},
};
const window = {
scrollY: 0,
innerHeight: 900,
location: { hash: "" },
addEventListener() {},
getSelection: () => ({ removeAllRanges() {}, addRange() {} }),
matchMedia: () => ({ matches: false, addEventListener() {} }),
requestAnimationFrame: () => 1,
scrollTo() {},
setTimeout: (callback) => timers.push(callback),
};
Object.assign(globalThis, {
document,
window,
requestAnimationFrame: () => 1,
cancelAnimationFrame() {},
Path2D: class {},
fetch: async () => ({
ok: true,
json: async () => ({ stargazers_count: stars }),
}),
});
Object.defineProperty(globalThis, "navigator", {
configurable: true,
value: {
clipboard: {
writeText: async (text) => {
if (clipboard === "reject") throw new Error("denied");
written.push(text);
},
},
},
});
await import(
`${pathToFileURL(new URL(script, ASSETS).pathname)}?load=${randomUUID()}`
);
await nextTurn();
return { root, timers, written, canvasText };
}
const copyBlock = () => [
h("button", { "data-copy": "command" }, "Copy"),
h("pre", {}, h("code", { id: "command" }, "npx rea-agents setup")),
];
const copyStatus = (messages = {}) =>
h("p", {
id: "copy-status",
...Object.fromEntries(
Object.entries(messages).map(([key, value]) => [`data-${key}`, value]),
),
});
const englishCopyMessages = {
copied: "Copied",
selected: "Selected",
"status-copied": "Code copied to clipboard.",
"status-selected": "Code selected. Use your keyboard to copy it.",
};
test("copy buttons report and restore localized labels", async () => {
const { root, timers, written } = await run("main.js", [
...copyBlock(),
copyStatus(englishCopyMessages),
]);
const button = root.querySelector("[data-copy]");
button.click();
await nextTurn();
assert.deepEqual(written, ["npx rea-agents setup"]);
assert.equal(button.textContent, "Copied");
assert.equal(
root.querySelector("#copy-status").textContent,
"Code copied to clipboard.",
);
timers.forEach((callback) => callback());
assert.equal(button.textContent, "Copy");
});
test("copy buttons keep their label when messages are missing", async () => {
for (const [body, clipboard] of [
[[...copyBlock(), copyStatus()], "resolve"],
[[...copyBlock(), copyStatus()], "reject"],
[
[...copyBlock(), copyStatus({ copied: "", "status-copied": "" })],
"resolve",
],
[copyBlock(), "resolve"],
]) {
const { root, timers, written } = await run("main.js", body, { clipboard });
const button = root.querySelector("[data-copy]");
button.click();
await nextTurn();
assert.equal(written.length, clipboard === "resolve" ? 1 : 0);
assert.equal(button.textContent, "Copy");
assert.equal(root.querySelector("#copy-status")?.textContent ?? "", "");
timers.forEach((callback) => callback());
assert.equal(button.textContent, "Copy");
}
});
const githubLink = (labels = {}) =>
h(
"a",
{
class: "nav-github",
"aria-label": "REA on GitHub",
title: "REA on GitHub",
...labels,
},
h("span", { class: "nav-github-count", hidden: "" }),
);
test("the star label follows the page language and plural rules", async () => {
const labels = {
"data-label-one": "REA on GitHub ({count} star)",
"data-label-other": "REA on GitHub ({count} stars)",
};
for (const [stars, lang, expected] of [
[1, "en", "REA on GitHub (1 star)"],
[1234, "en", "REA on GitHub (1,234 stars)"],
[1234, "de", "REA on GitHub (1.234 stars)"],
]) {
const { root } = await run("main.js", [githubLink(labels)], {
stars,
lang,
});
await nextTurn();
assert.equal(
root.querySelector(".nav-github").getAttribute("aria-label"),
expected,
);
}
// Languages without a "one" category use the "other" label.
const { root } = await run(
"main.js",
[githubLink({ "data-label-other": "{count}" })],
{ stars: 1, lang: "ja" },
);
await nextTurn();
assert.equal(
root.querySelector(".nav-github").getAttribute("aria-label"),
"1",
);
});
test("the star label stays unchanged without a localized label", async () => {
const { root } = await run("main.js", [githubLink()], { stars: 1 });
await nextTurn();
const link = root.querySelector(".nav-github");
assert.equal(link.getAttribute("aria-label"), "REA on GitHub");
assert.equal(link.getAttribute("title"), "REA on GitHub");
assert.equal(link.querySelector(".nav-github-count").hidden, true);
});
const percentDemo = (messages) =>
h(
"figure",
{ "data-percent-demo": "", ...messages },
h(
"div",
{ class: "percent-operation", hidden: "" },
h(
"button",
{ "data-percent-operation": "add", "aria-pressed": "true" },
"200 + 10%",
),
h(
"button",
{ "data-percent-operation": "multiply", "aria-pressed": "false" },
"200 × 10%",
),
),
h("span", { "data-percent-expression": "" }, "200 +"),
h("strong", { "data-percent-display": "" }, "20"),
h(
"div",
{ class: "percent-controls", hidden: "" },
h("button", { "data-percent-key": "percent" }, "%"),
h("button", { "data-percent-key": "equals", disabled: "" }, "="),
h("button", { "data-percent-key": "reset" }, "Reset"),
),
h(
"p",
{ "data-percent-status": "" },
"10% of 200 = 20. Then 200 + 20 = 220.",
),
);
test("the percent demo uses page messages and keeps text when one is missing", async () => {
const { root } = await run("home.js", [
percentDemo({
"data-msg-input": "Press % to see what happens to 10.",
"data-msg-result": "200 {operator} {percent} = {result}.",
}),
]);
const status = root.querySelector("[data-percent-status]");
const key = (name) =>
root.querySelector(`[data-percent-key="${name}"]`).click();
assert.equal(status.textContent, "Press % to see what happens to 10.");
key("percent");
assert.equal(status.textContent, "Press % to see what happens to 10.");
key("equals");
assert.equal(status.textContent, "200 + 20 = 220.");
const bare = await run("home.js", [percentDemo({})]);
const bareStatus = bare.root.querySelector("[data-percent-status]");
for (const name of ["percent", "equals", "reset"]) {
bare.root.querySelector(`[data-percent-key="${name}"]`).click();
assert.equal(
bareStatus.textContent,
"10% of 200 = 20. Then 200 + 20 = 220.",
);
}
});
const dinoDemo = (messages) =>
h(
"figure",
{ "data-dino-demo": "", ...messages },
h("svg", { class: "dino-fallback" }),
h("canvas", { hidden: "" }),
h(
"div",
{ class: "dino-controls", hidden: "" },
h("button", { "data-dino-start": "" }, "Play"),
h("button", { "data-dino-jump": "" }, "Jump ↑"),
h("button", { "data-dino-reset": "" }, "Restart"),
h("output", { "data-dino-speed": "" }, "6.0"),
h("input", { "data-dino-slider": "", value: "6" }),
h("input", { "data-dino-auto": "", checked: "" }),
),
h(
"p",
{ "data-dino-status": "" },
"Original rule: start at 6, then increase toward 13.",
),
);
test("the dinosaur game never shows a missing message", async () => {
const { root, canvasText } = await run("dino-demo.js", [
dinoDemo({}),
h("button", { "data-speed-check": "", hidden: "" }, "Replay"),
h("p", { "data-speed-check-status": "" }, "Not replayed yet."),
]);
const start = root.querySelector("[data-dino-start]");
const status = root.querySelector("[data-dino-status]");
const slider = root.querySelector("[data-dino-slider]");
start.click();
start.click();
slider.value = "9.5";
slider.dispatch("input");
root.querySelector("[data-dino-reset]").click();
assert.equal(start.textContent, "Play");
assert.equal(
status.textContent,
"Original rule: start at 6, then increase toward 13.",
);
assert.deepEqual(canvasText, []);
root.querySelector("[data-speed-check]").click();
assert.equal(
root.querySelector("[data-speed-check-status]").textContent,
"Not replayed yet.",
);
});
test("the dinosaur game fills placeholders from page messages", async () => {
const { root, canvasText } = await run("dino-demo.js", [
dinoDemo({
"data-msg-pause": "Pause",
"data-msg-fixed-running":
"Fixed speed: {speed}. Space or Jump clears a cactus.",
"data-msg-distance": "DISTANCE {distance}",
}),
]);
const start = root.querySelector("[data-dino-start]");
const slider = root.querySelector("[data-dino-slider]");
start.click();
assert.equal(start.textContent, "Pause");
slider.value = "9.5";
slider.dispatch("input");
assert.equal(
root.querySelector("[data-dino-status]").textContent,
"Fixed speed: 9.5. Space or Jump clears a cactus.",
);
assert.ok(canvasText.includes("DISTANCE 0000"));
});
+212 -2
View File
@@ -2,7 +2,10 @@
import contextlib
import importlib.util
from html import escape
import io
import json
import re
from pathlib import Path
import shutil
import tempfile
@@ -137,7 +140,7 @@ class WebsiteMaintenanceChecks(unittest.TestCase):
def test_shared_edits_preserve_body_and_catch_drift_before_rewriting(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
for name in ("public", "templates"):
for name in ("public", "templates", "locales"):
shutil.copytree(ROOT / "website" / name, root / "website" / name)
page = root / "website/public/blog/touhou-reconstruction/index.html"
original = page.read_text()
@@ -159,7 +162,7 @@ class WebsiteMaintenanceChecks(unittest.TestCase):
def test_new_page_escapes_metadata_stays_a_draft_and_joins_sitemap_when_ready(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
for name in ("public", "templates"):
for name in ("public", "templates", "locales"):
shutil.copytree(ROOT / "website" / name, root / "website" / name)
site = root / "website/public"
page = WEBSITE.new_page(root, "blog/new-example", 'A <test> & "result"', 'Check "quotes" & <markup>.')
@@ -242,5 +245,212 @@ class WebsiteMaintenanceChecks(unittest.TestCase):
worker.join()
def site_copy(directory):
"""Copy the authored website inputs without existing translated pages."""
root = Path(directory)
for name in ("public", "templates", "locales"):
shutil.copytree(ROOT / "website" / name, root / "website" / name)
for code in WEBSITE.LANGUAGES:
if code != "en":
shutil.rmtree(root / "website/public" / code, ignore_errors=True)
return root
def zh_hant(key):
"""Return one zh-Hant UI string, so assertions follow the locale file."""
strings = json.loads((ROOT / "website/locales/zh-hant.json").read_text())["strings"]
return escape(strings[key])
def quietly(function, *arguments):
"""Run a command helper without its progress output."""
with contextlib.redirect_stdout(io.StringIO()):
return function(*arguments)
def mark_reviewed(path):
"""Apply the reviewer's change: reviewed status and date, without noindex.
A reviewed translation has its own search title, so give the test page one.
"""
source = re.sub(r"<title>(.*?)</title>", r"<title>\1 (zh-Hant)</title>", path.read_text(), count=1)
source = source.replace('status="draft"', 'status="reviewed" reviewed="2026-10-11"', 1)
path.write_text(source.replace(' <meta name="robots" content="noindex" />\n', "", 1))
def translation_errors(root):
"""Return the verifier's translation errors and warnings for a site copy."""
return VERIFY.check_translations(root, VERIFY.parse_pages(root / "website/public"))
class WebsiteTranslationChecks(unittest.TestCase):
"""Create translations through the contributor command and check what readers get."""
def test_translate_creates_an_unlinked_noindex_draft(self):
with tempfile.TemporaryDirectory() as directory:
root = site_copy(directory)
site = root / "website/public"
english = (site / "get-started/index.html").read_text()
page = quietly(WEBSITE.new_translation, root, "zh-hant", "get-started")
source = page.read_text()
self.assertEqual(page, site / "zh-hant/get-started/index.html")
self.assertIn('<html lang="zh-Hant">', source)
self.assertIn('<meta name="robots" content="noindex" />', source)
self.assertEqual(
WEBSITE.translation_record(source),
{"source-sha256": WEBSITE.source_digest(english), "status": "draft"},
)
self.assertIn('href="../../assets/styles.css"', source)
self.assertIn('<a class="brand" href="../../"', source)
self.assertIn(
'<a href="../../zh-hant/get-started/" class="nav-start" aria-current="page">'
f'{zh_hant("navGetStarted")}</a>',
source,
)
self.assertIn(
f'<a href="../../showcase/" hreflang="en">{zh_hant("navShowcases")}'
'<span class="nav-badge" lang="en">EN</span></a>',
source,
)
self.assertIn('<li><a href="../../get-started/" hreflang="en" lang="en">English</a></li>', source)
self.assertIn('<meta property="og:locale" content="zh_TW" />', source)
self.assertNotIn('rel="alternate"', source)
self.assertIn(WEBSITE.NOTICE_START + WEBSITE.NOTICE_END, source)
self.assertEqual(english, (site / "get-started/index.html").read_text())
errors, warnings = translation_errors(root)
self.assertEqual(errors, [])
self.assertLessEqual({file for file, _ in warnings}, {"website/locales/zh-hant.json"})
self.assertEqual(VERIFY.check_site(site)[3], [])
with contextlib.redirect_stdout(io.StringIO()):
PREPARE.prepare_sitemap(site)
self.assertNotIn("zh-hant", (site / "sitemap.xml").read_text())
for language, route, message in (
("zh-hant", "get-started", "already exists"),
("en", "faq", "Choose one of"),
("ja", "missing", "No English page"),
("ja", "zh-hant/get-started", "without a language directory"),
("ja", "../outside", "directory route"),
):
with self.subTest(language=language, route=route):
with self.assertRaisesRegex(ValueError, message):
quietly(WEBSITE.new_translation, root, language, route)
def test_reviewed_translations_join_menus_hreflang_and_the_sitemap(self):
with tempfile.TemporaryDirectory() as directory:
root = site_copy(directory)
site = root / "website/public"
mark_reviewed(quietly(WEBSITE.new_translation, root, "zh-hant", "get-started"))
quietly(WEBSITE.sync_pages, root)
english = (site / "get-started/index.html").read_text()
for tag, url in (
("en", "https://rea.tools/get-started/"),
("zh-Hant", "https://rea.tools/zh-hant/get-started/"),
("x-default", "https://rea.tools/get-started/"),
):
self.assertIn(f'<link rel="alternate" hreflang="{tag}" href="{url}" />', english)
self.assertIn('<meta property="og:locale:alternate" content="zh_TW" />', english)
self.assertIn(
'<li><a href="../zh-hant/get-started/" hreflang="zh-Hant" lang="zh-Hant">繁體中文</a></li>', english
)
self.assertNotIn("language-menu", (site / "faq/index.html").read_text())
translated = (site / "zh-hant/get-started/index.html").read_text()
self.assertIn('<meta property="og:locale:alternate" content="en_US" />', translated)
# A reviewed home page becomes the menu's fallback for untranslated pages.
mark_reviewed(quietly(WEBSITE.new_translation, root, "zh-hant", "/"))
quietly(WEBSITE.sync_pages, root)
self.assertIn(
'<li><a href="../zh-hant/" hreflang="zh-Hant" lang="zh-Hant">繁體中文</a></li>',
(site / "faq/index.html").read_text(),
)
translated = (site / "zh-hant/get-started/index.html").read_text()
self.assertIn('<a class="brand" href="../../zh-hant/"', translated)
home = (site / "zh-hant/index.html").read_text()
self.assertIn('href="get-started/"', home)
self.assertIn('href="../showcase/"', home)
self.assertIn('href="../assets/styles.css"', home)
with contextlib.redirect_stdout(io.StringIO()):
PREPARE.prepare_sitemap(site)
sitemap = (site / "sitemap.xml").read_text()
self.assertIn("https://rea.tools/zh-hant/get-started/", sitemap)
self.assertIn("https://rea.tools/zh-hant/", sitemap)
self.assertEqual(VERIFY.check_site(site)[3], [])
self.assertEqual(translation_errors(root)[0], [])
def test_english_commands_sync_and_text_changes_show_a_notice(self):
with tempfile.TemporaryDirectory() as directory:
root = site_copy(directory)
site = root / "website/public"
english_path = site / "get-started/index.html"
page = quietly(WEBSITE.new_translation, root, "zh-hant", "get-started")
english = english_path.read_text()
identifier = WEBSITE.synced_code_ids(english)[0]
block = WEBSITE.code_block(identifier)
# A command-only change is copied and keeps the translation current.
command = block.search(english).group(2) + " --test-flag"
english_path.write_text(block.sub(lambda match: match.group(1) + command + match.group(3), english))
with self.assertRaisesRegex(ValueError, "Shared HTML is stale: zh-hant/get-started/index.html"):
quietly(WEBSITE.sync_pages, root, True)
quietly(WEBSITE.sync_pages, root)
self.assertEqual(block.search(page.read_text()).group(2), command)
self.assertNotIn('class="translation-notice"', page.read_text())
self.assertEqual(translation_errors(root), ([], []))
# A text change marks the translation out of date.
unchanged = english_path.read_text()
changed = unchanged.replace(" </main>", " <p>A new English paragraph.</p>\n </main>", 1)
english_path.write_text(changed)
self.assertEqual(quietly(WEBSITE.sync_pages, root, True), [page])
errors, warnings = translation_errors(root)
self.assertEqual(errors, [])
self.assertTrue(any("has changed since this translation" in message for _, message in warnings))
quietly(WEBSITE.sync_pages, root)
link = f'<a href="../../get-started/" hreflang="en">{zh_hant("staleNoticeLink")}</a>'
self.assertIn(
f'<p class="translation-notice" role="note">{zh_hant("staleNotice").replace("{link}", link)}</p>',
page.read_text(),
)
report = quietly(WEBSITE.translation_report, root)
self.assertIn(f' after updating the translation, record source-sha256="{WEBSITE.source_digest(changed)}"', report)
english_path.write_text(unchanged)
quietly(WEBSITE.sync_pages, root)
self.assertNotIn('class="translation-notice"', page.read_text())
page.write_text(block.sub(lambda match: match.group(1) + "edited" + match.group(3), page.read_text()))
self.assertTrue(any(f"{identifier} must match" in error for error in translation_errors(root)[0]))
def test_translations_keep_structure_and_cjk_spacing(self):
with tempfile.TemporaryDirectory() as directory:
root = site_copy(directory)
page = quietly(WEBSITE.new_translation, root, "zh-hant", "get-started")
original = page.read_text()
first, second = WEBSITE.copy_targets(original)[:2]
def added(paragraph):
return (" </main>", f" {paragraph}\n </main>")
cases = (
(*added("<p>完成後\n 重新啟動。</p>"), "renders as a gap"),
('<main id="main"', '<main id="moved"', "element IDs differ"),
(f'data-copy="{first}"', f'data-copy="{second}"', "copy buttons differ"),
(' <meta name="robots" content="noindex" />\n', "", "draft translation needs"),
('status="draft"', 'status="final"', "status must be draft or reviewed"),
)
for old, new, message in cases:
with self.subTest(message=message):
self.assertIn(old, original)
page.write_text(original.replace(old, new, 1))
self.assertTrue(any(message in error for error in translation_errors(root)[0]))
page.write_text(original.replace(*added("<p>重新啟動 REA 用戶端。</p>"), 1))
self.assertEqual(translation_errors(root)[0], [])
page.write_text(original.replace(*added("<p><code>extra-command</code></p>"), 1))
errors, warnings = translation_errors(root)
self.assertEqual(errors, [])
self.assertTrue(any("inline code differs" in message for _, message in warnings))
if __name__ == "__main__":
unittest.main()
+197 -32
View File
@@ -1,9 +1,18 @@
import { spawn } from "node:child_process";
import { cp, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path";
import {
chmod,
copyFile,
mkdir,
mkdtemp,
readFile,
rm,
writeFile,
} from "node:fs/promises";
import { homedir } from "node:os";
import { delimiter, dirname, join, resolve } from "node:path";
import { createInterface } from "node:readline";
import { fileURLToPath } from "node:url";
import { z } from "zod";
import { evaluateCodexEvents } from "../dist/evaluation/CodexAgentEval.js";
import {
@@ -15,32 +24,95 @@ import {
agentEvaluationPassed,
summarizeFactualCorrectness,
} from "../dist/evaluation/AgentEvaluationReport.js";
import { MCP_STARTUP_POLICY } from "../dist/mcpStartupPolicy.js";
import { PRODUCT_IDENTITY } from "../dist/identity.js";
import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs";
const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
const verifierRun = createVerifierRun();
const evaluationRoot = await mkdtemp(join(tmpdir(), "rea-agent-eval-"));
const fixtureRoot = join(evaluationRoot, "targets");
const skillDestination = join(
evaluationRoot,
".agents/skills/reverse-engineer-anything",
);
const timeoutMs = Number(process.env.REA_AGENT_EVAL_TIMEOUT_MS ?? 480_000);
const codex = process.env.REA_CODEX_CLI ?? "codex";
const optionalModel = process.env.REA_AGENT_EVAL_MODEL;
if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 10_000)
throw new Error("REA_AGENT_EVAL_TIMEOUT_MS must be an integer >= 10000");
const codexVersion = await readCommandVersion(codex);
// Codex refuses helper aliases when CODEX_HOME is under the OS temp directory.
// Keep the disposable account in a private cache, then remove the owned run.
const evaluationParent = resolve(
process.env.REA_AGENT_EVAL_ROOT ??
join(homedir(), ".cache", "rea-agent-evaluations"),
);
await mkdir(evaluationParent, { recursive: true, mode: 0o700 });
const evaluationRoot = await mkdtemp(join(evaluationParent, "run-"));
const fixtureRoot = join(evaluationRoot, "targets");
const account = join(evaluationRoot, "account");
const codexHome = join(evaluationRoot, "codex");
const authPath = join(codexHome, "auth.json");
const skillDestination = join(
account,
".agents/skills/reverse-engineer-anything",
);
const evaluationEnvironment = {
...Object.fromEntries(
Object.entries(process.env).filter(
([name]) => !/^(?:REA_|CODEX_|GHIDRA_|HOPPER_|IDA_)/u.test(name),
),
),
HOME: account,
USERPROFILE: account,
CODEX_HOME: codexHome,
XDG_CONFIG_HOME: join(account, ".config"),
XDG_DATA_HOME: join(account, ".local/share"),
XDG_CACHE_HOME: join(account, ".cache"),
REA_PROCESS_RUN_ID: verifierRun.run_id,
PATH: `${dirname(process.execPath)}${delimiter}${process.env.PATH ?? ""}`,
};
try {
await mkdir(fixtureRoot, { recursive: true });
await cp(
join(repositoryRoot, "skills/reverse-engineer-anything"),
skillDestination,
{ recursive: true },
const codexVersion = await readCommandVersion(codex);
await Promise.all([
mkdir(fixtureRoot, { recursive: true }),
mkdir(account),
mkdir(codexHome, { mode: 0o700 }),
]);
// Reuse authentication only. User config, plugins, rules, history and skills
// must not make a fresh installed-skill trial pass accidentally.
try {
await copyFile(
join(process.env.CODEX_HOME ?? join(homedir(), ".codex"), "auth.json"),
authPath,
);
await chmod(authPath, 0o600);
} catch (cause) {
if (cause?.code !== "ENOENT") throw cause;
// Environment/API-key authentication can work without a local auth file.
}
await writeFile(
join(codexHome, "config.toml"),
'approval_policy = "never"\nsandbox_mode = "read-only"\nweb_search = "disabled"\n[features]\nmulti_agent = false\nshell_snapshot = false\n[history]\npersistence = "none"\n',
{ mode: 0o600 },
);
for (const approval of ["--dry-run", "--yes"]) {
await runProcess(
process.execPath,
[
join(repositoryRoot, "scripts/rea.mjs"),
"setup",
"--client",
"codex",
approval,
"--json",
],
evaluationRoot,
120_000,
evaluationEnvironment,
);
}
const installedSkill = await readFile(
join(skillDestination, "SKILL.md"),
"utf8",
);
if (!installedSkill.includes(` version: "${PRODUCT_IDENTITY.skillVersion}"`))
throw new Error("Setup did not install the current packaged REA skill");
const requestedScenarioIds = new Set(
(process.env.REA_AGENT_EVAL_SCENARIOS ?? "")
.split(",")
@@ -57,7 +129,11 @@ try {
expectedFirstTool: "open_binary",
requiresEvidence: false,
requiredAnswerTermGroups: [["unavailable", "could not", "provider"]],
prompt: `Explain what the shipped native program at ${targets.native} does. Base the answer on artifact analysis, and clearly state any facts that remain unavailable.`,
// The verifier authorizes these named session operations, not arbitrary
// writes or runtime execution. Missing-engine behavior stays observable.
approvedTools: ["open_binary", "close_binary"],
requiredToolSubsequence: ["open_binary", "close_binary"],
prompt: `Explain what static artifact evidence establishes about the shipped native program at ${targets.native}. Do not execute the target or install tools. Clearly state unavailable provider evidence, and close any analysis session you opened before finishing.`,
},
{
id: "asar",
@@ -70,6 +146,38 @@ try {
],
prompt: `Explain how the desktop application at ${targets.javascript} exposes APIs to its renderer. Base the answer on the shipped application artifact, and state what remains unknown.`,
},
{
id: "asar-zh",
fixtureScenarioId: "asar",
expectedFirstTool: "analyze_javascript_application",
requiresEvidence: true,
requiredAnswerTermGroups: [["profileapi"], ["profile:read"]],
prompt: `請靜態分析已交付的 Electron 程式 ${targets.javascript},說明 renderer 如何透過 preload 讀取 profile,以及 main handler 的關係。不要執行目標或安裝分析引擎;引用完整的產出 Evidence,保留 runtime 未知事項。`,
},
{
id: "javascript-module-view",
expectedFirstTool: "analyze_javascript_application",
requiresEvidence: true,
requiredToolSubsequence: [
"analyze_javascript_application",
"inspect_analysis_view",
"inspect_analysis_view",
],
requiredAnswerTermGroups: [
["main.js"],
["node", "identity"],
["observations", "recorded"],
],
prompt: `Inspect the recorded module identity, exports and observations for main.js in the shipped JavaScript tree ${targets.largeJavascript}. Use REA without directly reading target files or executing the target. Keep results focused on that module, cite Evidence, and state the limits of what its module view establishes.`,
},
{
id: "missing-target",
expectedFirstTool: null,
requiresEvidence: false,
requiredAnswerTermGroups: [],
prompt:
'Reverse engineer my desktop app and explain how it saves user settings. Use static analysis; do not install software. End with one JSON object containing status ("needs_target" or "analyzed"), requested_input ("app_name_or_artifact_path" or null), and question (a string or null).',
},
{
id: "javascript-export-shape",
expectedFirstTool: "analyze_javascript_application",
@@ -145,12 +253,14 @@ try {
const results = [];
for (const scenario of scenarios) {
process.stderr.write(`Running Codex agent evaluation: ${scenario.id}\n`);
const fixtureClaims = agentFixtureClaims(scenario.id, targets);
const answerInstructions = agentFixtureAnswerInstructions(scenario.id);
const fixtureId = scenario.fixtureScenarioId ?? scenario.id;
const fixtureClaims = agentFixtureClaims(fixtureId, targets);
const answerInstructions = agentFixtureAnswerInstructions(fixtureId);
const execution = await runCodex(
answerInstructions.length === 0
? scenario.prompt
: `${scenario.prompt}\n\n${answerInstructions}`,
: `${scenario.prompt}\n\nThis closed factual rubric needs complete producer Evidence in the transcript. Request complete detail on each initial application analysis when supported; summary or view records alone are not accepted by this rubric.\n\n${answerInstructions}`,
scenario.approvedTools ?? [],
);
const transcriptDirectory = process.env.REA_AGENT_EVAL_TRANSCRIPT_DIR;
if (transcriptDirectory !== undefined) {
@@ -175,6 +285,12 @@ try {
...(fixtureClaims === undefined ? {} : { fixtureClaims }),
},
);
// Fixtures are deliberately visible beside prior investigation artifacts.
// Their existence does not identify the unnamed app in this request.
const targetClarificationPassed =
scenario.expectedFirstTool === null
? validTargetClarification(metrics.finalMessage)
: undefined;
const result = {
id: scenario.id,
expectedFirstTool: scenario.expectedFirstTool,
@@ -185,6 +301,9 @@ try {
exitCode: execution.exitCode,
stderr: execution.stderr,
...metrics,
...(targetClarificationPassed === undefined
? {}
: { targetClarificationPassed }),
};
results.push(result);
process.stderr.write(
@@ -194,13 +313,20 @@ try {
const factualCorrectness = summarizeFactualCorrectness(results);
const summary = {
schemaVersion: 3,
schemaVersion: 4,
evaluationScope: "routing_workflow_and_configured_fixture_claims",
factualCorrectness,
verifier_run: await completeVerifierRun(verifierRun),
codex,
codexVersion,
model: optionalModel ?? null,
installation: {
method: "rea_setup_codex",
isolatedHome: true,
isolatedCodexHome: true,
skillVersion: PRODUCT_IDENTITY.skillVersion,
authentication: "caller_auth_only_removed_after_run",
},
scenarios: results,
totals: {
scenarios: results.length,
@@ -253,12 +379,32 @@ try {
`Agent routing/workflow/answer checks failed: ${failed.map(({ id }) => id).join(", ")}`,
);
} finally {
// Even a deliberately retained fixture directory must never retain auth.
await rm(authPath, { force: true });
if (process.env.REA_AGENT_EVAL_KEEP_FIXTURES !== "true")
await rm(evaluationRoot, { recursive: true, force: true });
}
async function createTargets(root, includeManaged) {
const fixtures = await createAgentEvaluationFixtures(root, repositoryRoot);
const largeJavascript = join(root, "large-app");
await mkdir(join(largeJavascript, "lib"), { recursive: true });
await Promise.all([
writeFile(
join(largeJavascript, "package.json"),
'{"name":"large-app","main":"main.js"}\n',
),
writeFile(
join(largeJavascript, "main.js"),
'const { ipcMain } = require("electron");\nipcMain.handle("profile:read", (_event, id) => ({ id }));\n',
),
...Array.from({ length: 180 }, (_, index) =>
writeFile(
join(largeJavascript, "lib", `formatter-${String(index)}.js`),
`export const format = (value) => String(value).trim();\n`,
),
),
]);
const managedOutput = join(root, "managed-output");
if (includeManaged) {
@@ -292,15 +438,29 @@ async function createTargets(root, includeManaged) {
return {
native: "/bin/true",
...fixtures,
largeJavascript,
managed: join(managedOutput, "AgentEval.dll"),
};
}
async function runCodex(prompt) {
function validTargetClarification(message) {
const schema = z.strictObject({
status: z.literal("needs_target"),
requested_input: z.literal("app_name_or_artifact_path"),
question: z.string().refine((value) => value.trim().length > 0),
});
try {
return schema.safeParse(JSON.parse(message)).success;
} catch (cause) {
if (cause instanceof SyntaxError) return false;
throw cause;
}
}
async function runCodex(prompt, approvedTools) {
const arguments_ = [
"--no-daemon",
"exec",
"--ignore-user-config",
"--ignore-rules",
"--ephemeral",
"--skip-git-repo-check",
"--sandbox",
@@ -312,18 +472,16 @@ async function runCodex(prompt) {
evaluationRoot,
"-c",
'approval_policy="never"',
"-c",
`mcp_servers.rea.command=${JSON.stringify(process.execPath)}`,
"-c",
`mcp_servers.rea.args=${JSON.stringify([join(repositoryRoot, "scripts/rea.mjs"), "mcp"])}`,
"-c",
`mcp_servers.rea.startup_timeout_sec=${String(MCP_STARTUP_POLICY.codexStartupTimeoutSeconds)}`,
...approvedTools.flatMap((name) => [
"-c",
`mcp_servers.rea.tools.${name}.approval_mode="approve"`,
]),
...(optionalModel === undefined ? [] : ["--model", optionalModel]),
prompt,
];
const child = spawn(codex, arguments_, {
cwd: evaluationRoot,
env: process.env,
env: evaluationEnvironment,
stdio: ["ignore", "pipe", "pipe"],
});
const events = [];
@@ -362,9 +520,16 @@ async function runCodex(prompt) {
return { events, exitCode, stderr: stderr.join("").slice(0, 32_768) };
}
async function runProcess(command, arguments_, cwd, timeout) {
async function runProcess(
command,
arguments_,
cwd,
timeout,
env = process.env,
) {
const child = spawn(command, arguments_, {
cwd,
env,
stdio: ["ignore", "ignore", "pipe"],
});
let stderr = "";
+84 -40
View File
@@ -15,6 +15,10 @@ import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { PrivateRuntimeRoot } from "../dist/process/PrivateRuntimeRoot.js";
import {
OwnedCommandFailure,
runOwnedCommand,
} from "../dist/process/OwnedCommand.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import { mcpTextValue } from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
@@ -56,6 +60,13 @@ const root = await PrivateRuntimeRoot.create({
const environment = Object.fromEntries(
Object.entries(process.env).filter(([, value]) => typeof value === "string"),
);
const limitedPython = join(root.path, "limited-python");
const limitedEnvironment = (option) => ({
...environment,
REA_PWNTOOLS_PYTHON: limitedPython,
REA_VERIFY_LIMITED_PYTHON: python,
REA_VERIFY_LIMIT_OPTION: option,
});
const client = new Client({ name: "binary-layout-verifier", version: "1" });
const transport = new StdioClientTransport({
command: process.execPath,
@@ -73,6 +84,46 @@ let cases = 0;
let bootstrapCases = 0;
const failures = [];
try {
await execute(
"gcc",
[
"-std=c11",
"-Wall",
"-Wextra",
"-Werror",
fileURLToPath(
new URL("./fixtures/binary-layout-limited-python.c", import.meta.url),
),
"-o",
limitedPython,
],
{ timeout: 30_000, maxBuffer: 1024 * 1024 },
);
// A live limited child must retain its configured launcher's identity so
// timeout cleanup can still verify and terminate the entire owned group.
await assert.rejects(
runOwnedCommand(
{
command: limitedPython,
arguments: [
"-I",
"-c",
"import time; print('limited child ready', flush=True); time.sleep(60)",
],
cwd: root.path,
runId: `${run.run_id}-limited-python-timeout`,
hostEnvironment: limitedEnvironment("--as=2147483648"),
},
{ timeoutMs: 2000, diagnosticBytes: 1024 },
),
(error) => {
assert.ok(error instanceof OwnedCommandFailure);
assert.equal(error.reason, "timeout");
assert.equal(error.cleanupFailure, null);
assert.ok(error.snapshot.stdout.text.includes("limited child ready"));
return true;
},
);
const bootstrapRoot = join(root.path, "bootstrap-boundary");
await mkdir(bootstrapRoot);
const bootstrapPath = join(bootstrapRoot, "layout.py");
@@ -796,19 +847,12 @@ try {
assert.deepEqual(await readFile(extendedPath), extendedObject);
cases++;
}
for (const [name, option, expected] of [
["address-space", "--as=2147483648", '"address_space_bytes": 2147483648'],
["cpu", "--cpu=20", '"cpu_seconds": 20'],
["file-size", "--fsize=33554432", '"file_size_bytes": 33554432'],
for (const [option, expected] of [
["--as=2147483648", '"address_space_bytes": 2147483648'],
["--cpu=20", '"cpu_seconds": 20'],
["--fsize=33554432", '"file_size_bytes": 33554432'],
]) {
const wrapper = join(root.path, `python-limit-${name}`);
const quotedPython = "'" + python.replaceAll("'", "'\"'\"'") + "'";
await writeFile(
wrapper,
`#!/bin/sh\nexec /usr/bin/prlimit ${option} -- ${quotedPython} "$@"\n`,
{ mode: 0o700 },
);
const limitedEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: wrapper };
const selectedEnvironment = limitedEnvironment(option);
const limitedClient = new Client({
name: "limited-layout-verifier",
version: "1",
@@ -816,7 +860,7 @@ try {
const limitedTransport = new StdioClientTransport({
command: process.execPath,
args: [entrypoint, "mcp"],
env: limitedEnvironment,
env: selectedEnvironment,
stderr: "pipe",
});
try {
@@ -826,7 +870,7 @@ try {
mode,
join(root.path, "protected"),
undefined,
limitedEnvironment,
selectedEnvironment,
limitedClient,
);
assert.ok(value.limitations.some((item) => item.includes(expected)));
@@ -862,17 +906,29 @@ try {
sectionHeavy.writeUInt16LE(sectionCount, 60);
const sectionHeavyPath = join(root.path, "section-heavy.o");
await writeFile(sectionHeavyPath, sectionHeavy);
const memoryWrapper = join(root.path, "python-memory-constraint");
const quotedPython = "'" + python.replaceAll("'", "'\"'\"'") + "'";
await writeFile(
memoryWrapper,
`#!/bin/sh\nexec /usr/bin/prlimit --as=100663296 -- ${quotedPython} "$@"\n`,
{ mode: 0o700 },
// pwntools materializes each PT_LOAD zero-filled tail during inspection.
// One allocation larger than the whole address-space budget fails promptly;
// a large section count instead races CPU work against the command deadline.
const memoryHeavy = Buffer.from(await readFile(join(root.path, "protected")));
const memoryPhOffset = Number(memoryHeavy.readBigUInt64LE(32));
const memoryPhSize = memoryHeavy.readUInt16LE(54);
const memoryPhCount = memoryHeavy.readUInt16LE(56);
const lastLoadIndex = Array.from(
{ length: memoryPhCount },
(_, index) => index,
).findLast(
(index) =>
memoryHeavy.readUInt32LE(memoryPhOffset + index * memoryPhSize) === 1,
);
const memoryEnvironment = {
...environment,
REA_PWNTOOLS_PYTHON: memoryWrapper,
};
assert.notEqual(lastLoadIndex, undefined);
const memorySizeOffset = memoryPhOffset + lastLoadIndex * memoryPhSize + 40;
memoryHeavy.writeBigUInt64LE(
memoryHeavy.readBigUInt64LE(memorySizeOffset) + 128n * 1024n * 1024n,
memorySizeOffset,
);
const memoryHeavyPath = join(root.path, "memory-zero-tail");
await writeFile(memoryHeavyPath, memoryHeavy);
const memoryEnvironment = limitedEnvironment("--as=100663296");
const memoryClient = new Client({
name: "memory-layout-verifier",
version: "1",
@@ -888,7 +944,7 @@ try {
for (const mode of ["cli", "mcp"]) {
const error = await inspect(
mode,
sectionHeavyPath,
memoryHeavyPath,
"resource_constraint",
memoryEnvironment,
memoryClient,
@@ -903,7 +959,7 @@ try {
assert.ok(error.remediation.action.includes("memory"));
cases++;
}
assert.deepEqual(await readFile(sectionHeavyPath), sectionHeavy);
assert.deepEqual(await readFile(memoryHeavyPath), memoryHeavy);
} finally {
try {
await memoryClient.close();
@@ -924,13 +980,7 @@ try {
cpuHeavy.writeBigUInt64LE(BigInt(cpuSectionCount), sectionOffset + 32);
const cpuHeavyPath = join(root.path, "cpu-section-heavy.o");
await writeFile(cpuHeavyPath, cpuHeavy);
const cpuWrapper = join(root.path, "python-cpu-constraint");
await writeFile(
cpuWrapper,
`#!/bin/sh\nexec /usr/bin/prlimit --cpu=1: --core=0: -- ${quotedPython} "$@"\n`,
{ mode: 0o700 },
);
const cpuEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: cpuWrapper };
const cpuEnvironment = limitedEnvironment("--cpu=1:");
const cpuClient = new Client({ name: "cpu-layout-verifier", version: "1" });
const cpuTransport = new StdioClientTransport({
command: process.execPath,
@@ -963,13 +1013,7 @@ try {
}
}
for (const limit of [1024, 1]) {
const wrapper = join(root.path, `python-file-size-${limit}`);
await writeFile(
wrapper,
`#!/bin/sh\nexec /usr/bin/prlimit --fsize=${limit}: --core=0: -- ${quotedPython} "$@"\n`,
{ mode: 0o700 },
);
const fileEnvironment = { ...environment, REA_PWNTOOLS_PYTHON: wrapper };
const fileEnvironment = limitedEnvironment(`--fsize=${limit}:`);
const fileClient = new Client({
name: "file-size-layout-verifier",
version: "1",
+452
View File
@@ -0,0 +1,452 @@
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { homedir } from "node:os";
import { isAbsolute, join, relative } from "node:path";
import { fileURLToPath } from "node:url";
import Ajv2020 from "ajv/dist/2020.js";
import {
TOOL_CONTRACTS,
toolContract,
} from "../dist/contracts/toolContracts.js";
import { exec } from "./lib/verify-package-core.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
import { createOpenAiModelFixture } from "./verify/clients/model-fixture.mjs";
assert(
process.platform !== "win32",
"This verifier requires a disposable POSIX account; Windows is unverified",
);
const account = process.env.REA_VERIFY_COMMANDCODE_ACCOUNT_HOME;
assert(
account && isAbsolute(account) && account === homedir(),
"Run as a disposable account whose actual OS home matches REA_VERIFY_COMMANDCODE_ACCOUNT_HOME; do not override HOME",
);
assert.equal(
await readFile(join(account, ".rea-client-verification"), "utf8"),
"Disposable REA client verification account\n",
);
const repo = fileURLToPath(new URL("..", import.meta.url));
const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
const command = process.env.REA_VERIFY_COMMANDCODE_COMMAND ?? "cmd";
const mode = process.argv[2] ?? "call";
assert(
["chat", "call"].includes(mode),
"Usage: verify-commandcode-client.mjs [chat|call]",
);
const verifier = createVerifierRun();
const lab = await mkdtemp(join(account, "rea-commandcode-client-"));
const workspace = join(lab, "workspace");
const target = join(workspace, "fixture α测试");
const profile = join(account, ".commandcode");
const configPath = join(profile, "mcp.json");
const skillDirectory = join(
account,
".agents",
"skills",
"reverse-engineer-anything",
);
await Promise.all([
mkdir(target, { recursive: true }),
mkdir(profile, { recursive: true }),
mkdir(join(lab, "tmp")),
]);
await exec("git", ["init", "--quiet", workspace], { timeout: 30_000 });
await writeFile(
join(target, "app.js"),
'export function result() { return { clientCompatibility: "REA_CLIENT_FIXTURE", count: 7 }; }\n',
);
const environment = {
PATH: process.env.PATH,
NODE_OPTIONS: process.env.NODE_OPTIONS,
USERPROFILE: account,
TMPDIR: join(lab, "tmp"),
CMD_LOCAL_ONLY: "1",
// Native print mode checks for an account key even with a keyless BYOK model.
// This synthetic value is never a real credential; local-only refuses hosted API calls.
COMMAND_CODE_API_KEY: "local-fixture-only",
COMMANDCODE_SKIP_UPDATES: "1",
COMMANDCODE_DISABLE_CRON: "1",
OTEL_SDK_DISABLED: "true",
REA_PROCESS_RUN_ID: verifier.run_id,
};
const expectedNames = TOOL_CONTRACTS.map(
(tool) => "mcp__rea__" + tool.name,
).sort();
const schemaValidator = new Ajv2020({ strict: false, validateFormats: false });
const textValues = (value) =>
typeof value === "string"
? [value]
: value !== null && typeof value === "object"
? Object.values(value).flatMap(textValues)
: [];
let clientVersion;
let expectedSkill;
let inventoryChecked = false;
let skillLoaded = false;
let schemasChecked = 0;
let catalogArtifact;
let catalogDigest;
let catalogProjection;
let artifactProjectionSeen = false;
let evidenceSeen = false;
let evidenceBytes;
let finalSeen = false;
let stage = 0;
let failure;
const call = (name, args) => ({
role: "assistant",
content: null,
tool_calls: [
{
id: `call_commandcode_${++stage}`,
type: "function",
function: { name, arguments: JSON.stringify(args) },
},
],
});
const validateCatalog = async (text) => {
const entries = [
...text.matchAll(
/^### (mcp__rea__[A-Za-z0-9_]+)\n[\s\S]*?\nParameters:\n```json\n([\s\S]*?)\n```/gm,
),
];
assert.deepEqual(
entries.map((entry) => entry[1]).sort(),
expectedNames,
"Native exact-name search must return every REA input schema",
);
for (const entry of entries) {
const schema = JSON.parse(entry[2]);
assert(
schemaValidator.validateSchema(schema),
`${entry[1]}: invalid native schema: ${JSON.stringify(schemaValidator.errors)}`,
);
}
schemasChecked = entries.length;
catalogDigest = createHash("sha256").update(text).digest("hex");
await writeFile(join(lab, "native-catalog.md"), text);
};
const fixture = await createOpenAiModelFixture({
directory: lab,
prefix: "commandcode",
onRequest: async ({ body, tools }) => {
assert(tools.some((tool) => tool.name === "search_tools"));
const strings = textValues(body.messages);
if (!inventoryChecked) {
const names = [
...new Set(
strings.flatMap(
(text) => text.match(/mcp__rea__[A-Za-z0-9_]+/g) ?? [],
),
),
].sort();
assert.deepEqual(
names,
expectedNames,
"Default native deferred-tool prompt must advertise the complete REA inventory",
);
assert.equal(
tools.filter((tool) => tool.name.startsWith("mcp__rea__")).length,
0,
"Keep native lazy schema delivery enabled",
);
assert(
tools
.find((tool) => tool.name === "activate_skill")
?.parameters.properties.name.enum.includes(
"reverse-engineer-anything",
),
);
inventoryChecked = true;
}
if (mode === "chat") {
assert.equal(
body.messages.filter((message) => message.role === "tool").length,
0,
);
finalSeen = true;
return { role: "assistant", content: "REA_CLIENT_COMPATIBILITY_OK" };
}
if (stage === 0)
return call("activate_skill", { name: "reverse-engineer-anything" });
assert(
strings.some((text) => text.includes(expectedSkill)),
"The model must receive the complete activated skill body",
);
skillLoaded = true;
const last = body.messages
.filter((message) => message.role === "tool")
.at(-1)?.content;
assert.equal(typeof last, "string");
if (stage === 1)
return call("search_tools", {
query: "select:" + expectedNames.join(","),
});
if (stage === 2) {
const spill = last.match(
/\[full output saved to: (.+?) — read it with read_file/,
);
if (spill) {
catalogArtifact = spill[1];
const child = relative(environment.TMPDIR, catalogArtifact);
assert(
isAbsolute(catalogArtifact) &&
child &&
!child.startsWith("..") &&
!isAbsolute(child),
"Native spill must remain inside this run's temporary directory",
);
const text = await readFile(catalogArtifact, "utf8");
await validateCatalog(text);
catalogProjection = { schemas: schemasChecked, sha256: catalogDigest };
const query =
"const fs=require('node:fs'),crypto=require('node:crypto');const text=fs.readFileSync(process.argv[1],'utf8');const schemas=[...text.matchAll(/^### (mcp__rea__[A-Za-z0-9_]+)\\n/gm)].length;console.log(JSON.stringify({rea_catalog:{schemas,sha256:crypto.createHash('sha256').update(text).digest('hex')}}));";
return call("shell_command", {
command: process.execPath,
args: ["-e", query, catalogArtifact],
});
}
await validateCatalog(last);
return call("mcp__rea__analyze_javascript_application", {
input_path: target,
format: "directory",
});
}
if (catalogArtifact && !artifactProjectionSeen) {
const line = last
.split("\n")
.find((item) => item.startsWith('{"rea_catalog":'));
assert(line, "Native shell must return the selected saved-catalog facts");
assert.deepEqual(JSON.parse(line).rea_catalog, catalogProjection);
artifactProjectionSeen = true;
return call("mcp__rea__analyze_javascript_application", {
input_path: target,
format: "directory",
});
}
const evidence = toolContract(
"analyze_javascript_application",
).outputSchema.parse(JSON.parse(last));
assert.equal(evidence.subject.local_path, target);
const shapes = evidence.normalized_result.graph.nodes
.flatMap((node) => node.observations)
.map((observation) => observation.properties)
.filter(
(properties) =>
properties.semantic_role === "export-return-shapes" &&
properties.exported_name === "result",
);
assert(
shapes.length > 0 &&
shapes.every(
(shape) => shape.return_shape_coverage?.projection_complete,
),
);
const fields = shapes.flatMap((shape) =>
shape.static_return_shapes.flatMap((item) => item.fields),
);
assert(
fields.some(
(field) =>
field.path === "/clientCompatibility" &&
field.state === "literal" &&
field.value === "REA_CLIENT_FIXTURE",
),
);
assert(
fields.some(
(field) =>
field.path === "/count" &&
field.state === "literal" &&
field.value === 7,
),
);
evidenceBytes = Buffer.byteLength(last);
await writeFile(join(lab, "native-evidence.json"), last);
evidenceSeen = true;
finalSeen = true;
return { role: "assistant", content: "REA_CLIENT_COMPATIBILITY_OK" };
},
});
await writeFile(
join(profile, "providers.json"),
JSON.stringify({
provider: {
"rea-local": {
baseURL: fixture.baseUrl,
apiKey: false,
models: {
"rea-client-fixture": { contextWindow: 1_000_000, maxOutput: 4096 },
},
},
},
}),
);
const original =
JSON.stringify(
{
mcpServers: { other: { command: "unrelated-server", enabled: false } },
preserved: "keep unrelated settings",
},
null,
2,
) + "\n";
await writeFile(configPath, original);
const rea = (args) =>
exec(
process.execPath,
[join(runtimeRoot, "scripts/rea.mjs"), ...args, "--json"],
{ env: environment, cwd: workspace, timeout: 60_000 },
);
try {
clientVersion = (
await exec(command, ["--version"], {
env: environment,
cwd: workspace,
timeout: 30_000,
})
).stdout.trim();
for (const args of [
["setup", "--client", "commandcode", "--dry-run"],
["setup", "--client", "commandcode", "--yes"],
["doctor", "--client", "commandcode", "--skill"],
]) {
const response = await rea(args);
await writeFile(
join(lab, `${args[0]}${args.includes("--dry-run") ? "-plan" : ""}.json`),
response.stdout,
);
const result = JSON.parse(response.stdout);
if (args[0] === "doctor") assert.equal(result.healthy, true);
else
assert.equal(
result.status,
args.includes("--dry-run") ? "planned" : "ready",
);
if (args.includes("--dry-run")) {
assert(
result.plannedActions.some((action) => action.target === configPath),
);
assert(
result.plannedActions.every(
(action) =>
action.target === configPath ||
action.target === skillDirectory ||
action.target.startsWith(skillDirectory + "/"),
),
"Refuse setup writes outside the disposable account's configuration and skill",
);
}
}
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
const configured = await readFile(configPath, "utf8");
assert.equal(JSON.parse(configured).preserved, "keep unrelated settings");
assert.equal(
JSON.parse(configured).mcpServers.other.command,
"unrelated-server",
);
const repeated = JSON.parse(
(await rea(["setup", "--client", "commandcode", "--yes"])).stdout,
);
assert.deepEqual(repeated.appliedActions, []);
assert.equal(await readFile(configPath, "utf8"), configured);
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
expectedSkill = (await readFile(join(skillDirectory, "SKILL.md"), "utf8"))
.replace(/^---\r?\n[\s\S]*?\r?\n---\r?\n/, "")
.trim();
const execution = exec(
command,
[
"-p",
mode === "chat"
? "Say hello briefly."
: `Use the REA skill to analyze ${target}.`,
"-m",
"rea-local/rea-client-fixture",
"--local-only",
"--skip-onboarding",
"--trust",
"--output-format",
"json",
"--max-turns",
"8",
"--yolo",
],
{
env: environment,
cwd: workspace,
timeout: 120_000,
maxBuffer: 10 * 1024 * 1024,
},
);
execution.child.stdin.end();
const response = await execution;
await writeFile(join(lab, `${mode}.stdout`), response.stdout);
await writeFile(join(lab, `${mode}.stderr`), response.stderr);
assert.equal(fixture.failure, undefined);
const events = response.stdout
.trim()
.split("\n")
.map((line) => JSON.parse(line));
assert(
events.some(
(event) =>
event.type === "result" &&
event.subtype === "success" &&
event.finalText === "REA_CLIENT_COMPATIBILITY_OK",
),
);
assert(inventoryChecked && finalSeen);
if (mode === "call")
assert(
skillLoaded &&
schemasChecked === expectedNames.length &&
evidenceSeen &&
(!catalogArtifact || artifactProjectionSeen),
);
} catch (cause) {
failure = cause;
if (typeof cause?.stdout === "string")
await writeFile(join(lab, `${mode}.stdout`), cause.stdout);
if (typeof cause?.stderr === "string")
await writeFile(join(lab, `${mode}.stderr`), cause.stderr);
} finally {
await fixture.close();
const receipt = {
client: "Command Code",
clientVersion,
mode,
passed: failure === undefined,
artifacts: lab,
runtimeRoot,
host: {
platform: process.platform,
architecture: process.arch,
node: process.versions.node,
},
modelFixture:
"loopback OpenAI completions/SSE, keyless BYOK, synthetic account-key gate and usage; no hosted authentication or live-model claim",
discovery:
"actual disposable OS account home; native user MCP configuration and default shared skill discovery; Windows unverified",
interface:
"default deferred schemas, complete exact-name search, native saved-catalog query when needed, actual JavaScript MCP call",
inventoryChecked,
catalogSize: inventoryChecked ? expectedNames.length : 0,
schemasChecked,
skillLoaded,
catalogArtifact,
catalogDigest,
artifactProjectionSeen,
evidenceSeen,
evidenceBytes,
finalSeen,
requests: fixture.requests,
probes: fixture.probes,
fixtureFailure: fixture.failure,
verifier: await completeVerifierRun(verifier),
};
await writeFile(join(lab, "receipt.json"), JSON.stringify(receipt, null, 2));
console.log(JSON.stringify(receipt));
}
if (failure !== undefined) throw failure;
+32 -2
View File
@@ -22,6 +22,13 @@ const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
const command = process.env.REA_VERIFY_COPILOT_COMMAND ?? "copilot";
const mode = process.argv[2] ?? "call";
const model = process.env.REA_VERIFY_COPILOT_MODEL ?? "gpt-5.4";
const wireModel = process.env.REA_VERIFY_COPILOT_WIRE_MODEL ?? model;
const inputSchemaProfile =
process.env.REA_VERIFY_COPILOT_SCHEMA_PROFILE ?? "full";
assert(
["full", "compact"].includes(inputSchemaProfile),
"REA_VERIFY_COPILOT_SCHEMA_PROFILE must be full or compact",
);
assert(
["chat", "call"].includes(mode),
"Usage: verify-copilot-client.mjs [chat|call]",
@@ -44,6 +51,7 @@ await exec("git", ["init", "--quiet", workspace], { timeout: 30_000 });
const environment = {
PATH: process.env.PATH,
NODE_OPTIONS: process.env.NODE_OPTIONS,
HOME: account,
USERPROFILE: account,
COPILOT_HOME: profile,
COPILOT_OFFLINE: "true",
@@ -54,7 +62,7 @@ const environment = {
COPILOT_PROVIDER_BEARER_TOKEN: "",
COPILOT_PROVIDER_HEADERS: "",
COPILOT_PROVIDER_MODEL_ID: model,
COPILOT_PROVIDER_WIRE_MODEL: model,
COPILOT_PROVIDER_WIRE_MODEL: wireModel,
COPILOT_PROVIDER_MAX_PROMPT_TOKENS: "1000000",
COPILOT_PROVIDER_MAX_OUTPUT_TOKENS: "4096",
REA_PROCESS_RUN_ID: verifier.run_id,
@@ -112,10 +120,17 @@ const extractEvidence = (value) => {
}
return undefined;
};
const wireModels = [];
const fixture = await createOpenAiModelFixture({
directory: lab,
prefix: "copilot",
onRequest: async ({ body, tools, requestIndex }) => {
assert.equal(
body.model,
wireModel,
"Copilot must send the selected wire model",
);
wireModels.push(body.model);
// Auxiliary requests without tools cannot execute the analyst's workflow.
if (tools.length === 0)
return { role: "assistant", content: "REA fixture title" };
@@ -311,6 +326,16 @@ try {
assert.deepEqual(JSON.parse(repeated.stdout).appliedActions, []);
assert.equal(await readFile(configPath, "utf8"), configured);
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
if (inputSchemaProfile === "compact") {
// Select the existing presentation in this disposable native-client profile.
const customized = JSON.parse(configured);
assert(customized.mcpServers.rea);
customized.mcpServers.rea.env = {
...customized.mcpServers.rea.env,
REA_MCP_INPUT_SCHEMA_PROFILE: inputSchemaProfile,
};
await writeFile(configPath, JSON.stringify(customized, null, 2));
}
const installedSkill = await readFile(
join(skillDirectory, "SKILL.md"),
"utf8",
@@ -408,6 +433,8 @@ try {
clientVersion,
mode,
model,
wireModel,
inputSchemaProfile,
passed: failure === undefined,
artifacts: lab,
runtimeRoot,
@@ -421,7 +448,9 @@ try {
skillDiscovery:
"native COPILOT_HOME MCP configuration and caller-added isolated skill directory; default OS-home discovery unverified",
capacityBoundary:
"gpt-4.1 model configuration blocks the full catalog before HTTP even with a ten-million-token requested prompt override; gpt-5.4 is the verified local fixture configuration",
inputSchemaProfile === "compact"
? "Explicit compact profile: complete inventory and canonical server validation; advertised schemas may be reduced. This does not fix the full-profile gpt-4.1 context boundary."
: "Complete canonical input and output schemas. Client model ID and wire model are recorded separately; effective token capacity and live provider compatibility remain unknown.",
catalogSize,
schemasChecked,
spillReadRequested,
@@ -429,6 +458,7 @@ try {
skillLoaded,
artifactDigest,
finalSeen,
wireModels,
requests: fixture.requests,
probes: fixture.probes,
fixtureFailure: fixture.failure,
+2 -2
View File
@@ -40,9 +40,9 @@ await Promise.all([
]);
const environment = {
PATH: process.env.PATH ?? "",
HOME: account,
USERPROFILE: account,
DSH_HOME: profile,
DSH_AGENTS_HOME: join(account, ".agents"),
OPENAI_API_KEY: "local-fixture-key",
REA_PROCESS_RUN_ID: verifier.run_id,
};
@@ -328,7 +328,7 @@ try {
"loopback custom OpenAI-compatible adapter; no live DeepSeek provider claim",
modelContextWindow: 1000000,
skillDiscovery:
"native DSH_AGENTS_HOME personal directory; isolated project root",
"native HOME shared personal directory; isolated project root",
catalogSize: catalog?.length,
patternCount,
uniquePatterns: patterns.size,
+2 -8
View File
@@ -54,6 +54,7 @@ await writeFile(
);
const environment = {
PATH: process.env.PATH ?? "",
HOME: account,
USERPROFILE: account,
[client === "pi" ? "PI_CODING_AGENT_DIR" : "HERMES_HOME"]: profile,
PI_OFFLINE: "1",
@@ -71,13 +72,6 @@ const original =
})
: "# Keep caller settings.\nmcp_servers:\n other:\n command: unrelated-server\n enabled: false\n";
await writeFile(configPath, original);
if (client === "pi") {
// The CLI fixture account and native os.homedir() differ; use Pi's caller-owned skill setting.
await writeFile(
join(profile, "settings.json"),
JSON.stringify({ skills: [join(account, ".agents", "skills")] }),
);
}
const clientVersion = (
await exec(command, ["--version"], { env: environment, timeout: 30_000 })
).stdout.trim();
@@ -448,7 +442,7 @@ try {
"loopback deterministic OpenAI-compatible endpoint; no live provider claim",
skillDiscovery:
client === "pi"
? "caller-configured isolated shared directory; default OS home discovery unverified"
? "native HOME shared personal directory; no explicit skill setting"
: "native HERMES_HOME personal directory",
stickyHermes,
catalogSize: catalog?.length,
+145 -97
View File
@@ -38,104 +38,147 @@ let update;
let repositorySkill;
let report;
const runStage = async (name, action) => {
const started = performance.now();
process.stderr.write(`[verify:package] ${name}: start\n`);
const result = await action();
process.stderr.write(
`[verify:package] ${name}: complete (${Math.round(performance.now() - started)} ms)\n`,
);
return result;
};
try {
repositorySkill = await verifyRepositorySkill({ root, workspace });
({ tarball } = await verifyPackagePack({ root, workspace }));
const environmentData = await verifyPackageEnvironment({
root,
workspace,
evidenceRoot,
referenceRoot,
});
const { cli, packageRunnerCli } = await verifyPackageInstall({
tarball,
prefix: environmentData.prefix,
workspace,
environment: environmentData.environment,
});
update = await verifyPackageUpdate({
prefix: environmentData.prefix,
packageRoot: join(
environmentData.prefix,
"lib",
"node_modules",
PRODUCT_IDENTITY.packageName,
),
tarball,
workspace,
environment: environmentData.environment,
});
mcpStartup = await measureMcpStartup({
command: cli,
args: ["mcp"],
environment: {
...environmentData.environment,
},
policy: MCP_STARTUP_POLICY,
});
mcpModuleLoading = await profileMcpModuleLoading({
command: cli,
args: ["mcp"],
environment: {
...environmentData.environment,
},
policy: MCP_STARTUP_POLICY,
packageName: PRODUCT_IDENTITY.packageName,
});
const { supportedSetupHost, hopperSetupSupported } =
await verifyPackageDiscovery({
repositorySkill = await runStage("repository skill", () =>
verifyRepositorySkill({ root, workspace }),
);
({ tarball } = await runStage("package archive", () =>
verifyPackagePack({ root, workspace }),
));
const environmentData = await runStage("package environment", () =>
verifyPackageEnvironment({
root,
workspace,
evidenceRoot,
referenceRoot,
}),
);
const { cli, packageRunnerCli } = await runStage("package install", () =>
verifyPackageInstall({
tarball,
prefix: environmentData.prefix,
workspace,
environment: environmentData.environment,
}),
);
update = await runStage("package update", () =>
verifyPackageUpdate({
prefix: environmentData.prefix,
packageRoot: join(
environmentData.prefix,
"lib",
"node_modules",
PRODUCT_IDENTITY.packageName,
),
tarball,
workspace,
environment: environmentData.environment,
}),
);
mcpStartup = await runStage("MCP startup", () =>
measureMcpStartup({
command: cli,
args: ["mcp"],
environment: {
...environmentData.environment,
},
policy: MCP_STARTUP_POLICY,
}),
);
mcpModuleLoading = await runStage("MCP module loading", () =>
profileMcpModuleLoading({
command: cli,
args: ["mcp"],
environment: {
...environmentData.environment,
},
policy: MCP_STARTUP_POLICY,
packageName: PRODUCT_IDENTITY.packageName,
}),
);
const { supportedSetupHost, hopperSetupSupported } = await runStage(
"package discovery",
() =>
verifyPackageDiscovery({
cli,
environment: environmentData.environment,
}),
);
const { artifactArchive } = await runStage("artifact and Electron CLI", () =>
verifyPackageArtifactAndElectron({
cli,
workspace,
environment: environmentData.environment,
}),
);
await runStage("managed analysis CLI", () =>
verifyManaged({
cli,
workspace,
environment: environmentData.environment,
}),
);
await runStage("unknown provider CLI", () =>
verifyUnknownProvider({
cli,
environment: environmentData.environment,
});
const { artifactArchive } = await verifyPackageArtifactAndElectron({
cli,
workspace,
environment: environmentData.environment,
});
await verifyManaged({
cli,
workspace,
environment: environmentData.environment,
});
await verifyUnknownProvider({
cli,
environment: environmentData.environment,
});
await verifyPackagePlatform({
cli,
environment: environmentData.environment,
});
await verifyPackageCapabilitiesAndSearch({
cli,
environment: environmentData.environment,
});
await verifyPackageEvidence({
cli,
evidenceRoot,
referenceRoot,
environment: environmentData.environment,
});
await verifyPackageSetup({
cli,
packageRunnerCli,
environment: environmentData.environment,
home: environmentData.home,
npxLog: environmentData.npxLog,
claudeConfig: environmentData.claudeConfig,
codexConfig: environmentData.codexConfig,
cursorConfig: environmentData.cursorConfig,
codexTarget: environmentData.codexTarget,
cursorTarget: environmentData.cursorTarget,
supportedSetupHost,
hopperSetupSupported,
root,
});
await verifyPackageMcp({
cli,
environment: environmentData.environment,
evidenceRoot,
artifactArchive,
});
}),
);
await runStage("platform CLI", () =>
verifyPackagePlatform({
cli,
environment: environmentData.environment,
}),
);
await runStage("capabilities and search CLI", () =>
verifyPackageCapabilitiesAndSearch({
cli,
environment: environmentData.environment,
}),
);
await runStage("evidence CLI", () =>
verifyPackageEvidence({
cli,
evidenceRoot,
referenceRoot,
environment: environmentData.environment,
}),
);
await runStage("setup and uninstall CLI", () =>
verifyPackageSetup({
cli,
packageRunnerCli,
environment: environmentData.environment,
home: environmentData.home,
npxLog: environmentData.npxLog,
claudeConfig: environmentData.claudeConfig,
codexConfig: environmentData.codexConfig,
cursorConfig: environmentData.cursorConfig,
codexTarget: environmentData.codexTarget,
cursorTarget: environmentData.cursorTarget,
supportedSetupHost,
hopperSetupSupported,
root,
}),
);
await runStage("MCP tools and evidence", () =>
verifyPackageMcp({
cli,
environment: environmentData.environment,
evidenceRoot,
artifactArchive,
}),
);
report = {
cli: true,
analysisCli: true,
@@ -173,8 +216,13 @@ try {
mcpModuleLoading,
};
} finally {
await rm(workspace, { recursive: true, force: true });
await runStage("workspace cleanup", () =>
rm(workspace, { recursive: true, force: true }),
);
}
const verifierLineage = await runStage("verifier lineage", () =>
completeVerifierRun(verifierRun),
);
process.stdout.write(
`${JSON.stringify({ verifier_run: await completeVerifierRun(verifierRun), ...report })}\n`,
`${JSON.stringify({ verifier_run: verifierLineage, ...report })}\n`,
);
@@ -175,6 +175,7 @@ const callerDirectory = join(workspace, "caller cwd with spaces");
const scriptCollisions = [
"ReaGhidraBridge.java",
"ReaGhidraPrepareCom.java",
"ReaGhidraApplySeeds.java",
"ReaGhidraNoReturnFix.java",
];
await mkdir(callerDirectory);
+3 -5
View File
@@ -45,6 +45,7 @@ await Promise.all([
]);
const baseEnvironment = {
PATH: process.env.PATH ?? "",
HOME: account,
USERPROFILE: account,
QWEN_HOME: profile,
QWEN_RUNTIME_DIR: runtime,
@@ -71,9 +72,6 @@ const original = JSON.stringify(
},
telemetry: { enabled: false },
model: { name: "rea-client-fixture" },
// Qwen uses os.homedir(), while REA's isolated CLI account uses USERPROFILE.
// This caller-owned setting lets the real client load the REA-installed bundle.
skills: { directories: [sharedSkills] },
mcpServers: { other: { command: "unrelated-server", disabled: true } },
},
null,
@@ -112,7 +110,7 @@ const configured = JSON.parse(
await readFile(join(profile, "settings.json"), "utf8"),
);
assert.deepEqual(configured.model, { name: "rea-client-fixture" });
assert.deepEqual(configured.skills, { directories: [sharedSkills] });
assert.equal(configured.skills, undefined);
assert.deepEqual(configured.mcpServers.other, {
command: "unrelated-server",
disabled: true,
@@ -384,7 +382,7 @@ try {
modelFixture:
"loopback deterministic OpenAI-compatible endpoint; no live model-provider claim",
skillDiscovery:
"caller-configured isolated shared directory; default OS home discovery unverified",
"native HOME shared personal directory; no explicit skill setting",
catalogSize: TOOL_CONTRACTS.length,
requests: fixture.requests,
probes: fixture.probes,
+17 -1
View File
@@ -45,6 +45,7 @@ import { verifyBrowserDomDestinations } from "./lib/browser-dom-destinations-e2e
import { verifyBrowserCaptureMetadataBudget } from "./lib/browser-capture-metadata-budget-e2e.mjs";
import { artifactCliEvidence, artifactMcpResult } from "./lib/artifact-e2e.mjs";
import { verifyScenarioEnvironment } from "./lib/browser-scenario-environment-e2e.mjs";
import { verifyScenarioStorage } from "./lib/browser-scenario-storage-e2e.mjs";
const REAL_BROWSER_STARTUP_TIMEOUT_MS = 60_000;
const SCENARIO_SECRET_VALUE = "rea-browser-verifier-secret";
@@ -64,6 +65,7 @@ try {
executable,
[
"--headless=new",
"--password-store=basic",
...(process.env.REA_BROWSER_NO_SANDBOX === "true"
? ["--no-sandbox"]
: []),
@@ -275,11 +277,18 @@ try {
throw new Error("Scenario attachment terminated its external browser");
const profilesBefore = await scenarioProfiles();
process.stderr.write("Browser verifier: scenario environment\n");
const scenarioEnvironment = await verifyScenarioEnvironment(
endpoint,
target,
site.origin,
);
process.stderr.write("Browser verifier: scenario storage\n");
const scenarioStorage = await verifyScenarioStorage({
executable,
endpoint,
targetId: target,
});
const launchedScenario = await createBrowserScenarioProvider(
process.env,
).captureScenario(
@@ -332,7 +341,9 @@ try {
`Browser scenario included unselected structured value: ${unselectedValue}`,
);
process.stderr.write("Browser verifier: large screenshot\n");
const largeScreenshot = await verifyLargeScreenshotE2e(endpoint, site.origin);
process.stderr.write("Browser verifier: popup coverage\n");
const popupEvents = await verifyPopupEventCoverage(executable);
const networkEvidence = await verifyBrowserNetworkEvidence(
executable,
@@ -354,7 +365,11 @@ try {
scripts: observed.value.scripts.items.length,
networkRequests: observed.value.network.requests.length,
consoleEvents: observed.value.console.events.length,
websocketEvents: observed.value.network.websocket_events.length,
websocketConnections: observed.value.network.websocket_connections.length,
websocketFrames: observed.value.network.websocket_connections.reduce(
(total, connection) => total + connection.events.length,
0,
),
bundleScripts: bundle.capture.scripts_analyzed,
sourceMaps: bundle.observations.source_maps.processed,
bundle_cli_and_stdio_mcp: true,
@@ -373,6 +388,7 @@ try {
browserScenarioLaunchCleanup: "terminated-owned-process",
scenarioFailure,
scenarioEnvironment,
scenarioStorage,
verified: true,
};
} finally {
+164
View File
@@ -21,6 +21,10 @@ import {
} from "./verify-real-ghidra-assertions.mjs";
async function functionCall(client, operation, parameters) {
return (await functionObservationCall(client, operation, parameters)).value;
}
async function functionObservationCall(client, operation, parameters) {
const input = parseGhidraFunctionInput(operation, parameters);
if (!input.ok) throw input.error;
const called = await client.callTool(operation, input.value);
@@ -153,6 +157,57 @@ export async function verifyDebugFunctionOperations(
const denseSwitchDossier = await functionCall(client, "analyze_function", {
procedure: denseSwitchProcedure.address,
});
const terminalProcedure = requireProcedure(
procedures,
"rea_ghidra_inventory_terminal_call",
);
const terminalObservations = await Promise.all(
[
"procedure_info",
"procedure_assembly",
"read_function_instructions",
"procedure_pseudo_code",
"analyze_function",
].map((operation) =>
functionObservationCall(client, operation, {
...(operation === "analyze_function" ? {} : { document: null }),
procedure: terminalProcedure.address,
}),
),
);
const terminalLimitations = terminalObservations.map((result) =>
result.limitations.find(
(limitation) =>
/terminal call at 0x[0-9a-f]+/i.test(limitation) &&
/ to 0x[0-9a-f]+/i.test(limitation) &&
limitation.includes("abort") &&
limitation.includes("hasNoReturn=true"),
),
);
if (
terminalLimitations.some((limitation) => limitation === undefined) ||
terminalLimitations.some(
(limitation) =>
!limitation.includes("external=") ||
!limitation.includes("thunk=") ||
!limitation.includes("fallthrough is excluded") ||
!limitation.includes("not an independent verification"),
)
)
throw new Error(
`Ghidra no-return boundary evidence was missing callee, address, or flag provenance: ${JSON.stringify(terminalObservations.map(({ limitations }) => limitations))}`,
);
const returningObservation = await functionObservationCall(
client,
"procedure_info",
{ document: null, procedure: leaf.address },
);
if (
returningObservation.limitations.some((limitation) =>
limitation.includes("puts"),
)
)
throw new Error("Ghidra marked an ordinary returning call as terminal");
// The debug fixture is compiled -O0, and GCC 16 and later lower dense
// switches to compare chains at -O0, so this dossier asserts only its
// complete shape; the optimized jump-table variant proves dense jump-table
@@ -179,6 +234,8 @@ export async function verifyDebugFunctionOperations(
instruction_window: {
returned: instructionWindow.instructions.length,
},
terminal_call_limitations: terminalLimitations,
returning_call_limitations: returningObservation.limitations,
cancellation,
timeout,
concurrency: concurrent,
@@ -617,6 +674,113 @@ export async function verifyRelativeSwitch(client, procedures, entrySize) {
};
}
/**
* Oracle for relative-switch-signed.S: signed halfword byte offsets from a base
* inside the case bodies, a CBZ route for the zero entry, and a B.CS bound.
*/
const SIGNED_RELATIVE_SWITCH_CASES = [
{ value: 0, immediate: 200 },
{ value: 1, immediate: 201 },
{ value: 2, immediate: 202 },
{ value: 3, immediate: 250 },
{ value: 4, immediate: 204 },
];
export async function verifyRelativeSignedSwitch(client, selectorBias = 0) {
const names = await inventoryCall(client, "list_names", {
document: null,
address: null,
});
const procedure = requireProcedure(names, "rea_relative_switch_signed");
const dossier = await functionCall(client, "analyze_function", {
procedure: procedure.address,
});
const table = dossier.native_api?.jump_tables.find((item) =>
item.data_sources.some(
(source) =>
source.entry_size_bytes === 2 &&
source.entry_count === SIGNED_RELATIVE_SWITCH_CASES.length,
),
);
if (table === undefined)
throw new Error(
`Exact signed halfword relative table unavailable: ${JSON.stringify(dossier.native_api)}`,
);
const expectedValues = SIGNED_RELATIVE_SWITCH_CASES.map(
({ value }) => value + selectorBias,
);
if (
table.mappings.some(
(mapping) =>
mapping.case_value !== null &&
!expectedValues.includes(mapping.case_value),
)
)
throw new Error(
`Signed relative switch emitted an unsupported case label: ${JSON.stringify(table)}`,
);
if (table.mappings.length !== SIGNED_RELATIVE_SWITCH_CASES.length)
throw new Error(
`Signed relative switch has unexpected mappings: ${JSON.stringify(table)}`,
);
if (selectorBias !== 0) {
if (
!table.limitations.some((limitation) =>
limitation.includes("table indices are not proven"),
) ||
!table.mappings.some((mapping) => mapping.case_value === null) ||
table.mappings.some((mapping) => mapping.case_value === selectorBias + 3)
)
throw new Error(
`Normalized selector must retain unknown case evidence: ${JSON.stringify(table)}`,
);
}
for (const { value: tableIndex, immediate } of SIGNED_RELATIVE_SWITCH_CASES) {
const value = tableIndex + selectorBias;
// Without a proven relationship between the machine index and the typed
// switch selector, the CBZ-only case must stay unknown rather than being
// assigned the normalized index or Ghidra's unreachable base destination.
if (selectorBias !== 0 && tableIndex === 3) continue;
const mapping = table.mappings.find((item) => item.case_value === value);
if (
mapping === undefined ||
mapping.confidence !== "high" ||
(selectorBias !== 0 &&
!mapping.evidence.some(
(evidence) => evidence.source === "ghidra-clang-case-token",
))
)
throw new Error(
`Signed relative switch case ${value} was not verified: ${JSON.stringify(table)}`,
);
const instruction = await functionCall(
client,
"inspect_native_instruction",
{ address: mapping.target_address },
);
if (
instruction.status !== "decoded" ||
!instruction.operands.some((operand) =>
operand.components.some(
(component) =>
component.kind === "immediate" &&
component.value === `0x${immediate.toString(16)}`,
),
)
)
throw new Error(
`Signed relative switch target does not implement source case ${value}: ${JSON.stringify(instruction)}`,
);
}
return {
entry_size: 2,
signed: true,
selector_bias: selectorBias,
dispatch_address: table.dispatch_address,
mappings: table.mappings.length,
};
}
export async function verifyNativeValueTrace(client, procedures, target) {
const entry = requireProcedure(procedures, "rea_ghidra_inventory_entry");
const names = await inventoryCall(client, "list_names", {
+109 -1
View File
@@ -2,7 +2,7 @@
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdtemp, mkdir, readFile, rm } from "node:fs/promises";
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
@@ -60,6 +60,9 @@ const env = {
JAVA_HOME: installation.javaHome,
HOPPER_LAUNCHER_PATH: "/rea-unconfigured-deep-provider/hopper",
};
for (const setting of ["GHIDRA_HEADLESS_MAXMEM", "GHIDRA_MAXMEM"]) {
if (process.env[setting] !== undefined) env[setting] = process.env[setting];
}
const reports = [];
let primaryFailure;
try {
@@ -227,6 +230,103 @@ try {
fixture.sha256,
"Analysis modified the source executable",
);
// Carved firmware keeps all program headers/load bytes but loses sections.
const read16 = (offset) =>
byteOrder === "little"
? sourceBytes.readUInt16LE(offset)
: sourceBytes.readUInt16BE(offset);
const read32 = (offset) =>
byteOrder === "little"
? sourceBytes.readUInt32LE(offset)
: sourceBytes.readUInt32BE(offset);
const sectionOffset = read32(32);
const cut = sectionOffset - 1;
const programOffset = read32(28);
const programCount = read16(44);
assert.equal(read16(42), 32);
assert.ok(cut >= programOffset + programCount * 32);
for (let i = 0; i < programCount; i += 1) {
const offset = programOffset + i * 32;
assert.ok(read32(offset + 4) + read32(offset + 16) <= cut);
}
const carvedBytes = sourceBytes.subarray(0, cut);
const carvedPath = join(workspace, `${byteOrder}-carved.elf`);
const carvedSha256 = createHash("sha256")
.update(carvedBytes)
.digest("hex");
await writeFile(carvedPath, carvedBytes);
const carvedResolved = await parseBinaryTarget(carvedPath);
if (!carvedResolved.ok) throw carvedResolved.error;
assert.deepEqual(
carvedResolved.value.mips.abiFlags,
independent.mips.abiFlags,
);
assert.match(
carvedResolved.value.mips.limitations.join(" "),
/section header table/u,
);
const carvedTarget = await call("open_binary", {
path: carvedPath,
provider_id: "ghidra",
});
assert.equal(carvedTarget.sha256, carvedSha256);
const carvedImage = await call("inspect_native_load_image");
assertMipsLoadedImage(carvedImage.observations, byteOrder, carvedSha256);
const carvedProcedures = await call("list_procedures");
assert.ok(
carvedProcedures.some(
(p) =>
BigInt(p.address) === BigInt(independent.symbols.rea_mips_entry),
),
);
const carvedDossier = functionDossierSchema.parse(
await call("analyze_function", {
procedure: independent.symbols.rea_mips_entry,
}),
);
assert.ok(carvedDossier.pseudocode.trim().length > 0);
const carvedSnapshotPath = join(
workspace,
`${byteOrder}-carved.snapshot.json`,
);
await call("close_binary", { snapshot_path: carvedSnapshotPath });
const carvedSnapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(carvedSnapshotPath, "utf8")),
);
assert.equal(carvedSnapshot.target.sha256, carvedSha256);
assert.ok(
carvedSnapshot.evidence_bundle.records.some((record) =>
parseEvidence(record).limitations.some((item) =>
item.includes("section header table"),
),
),
);
const carvedCli = await execute(
process.execPath,
[
entrypoint,
"function",
carvedPath,
independent.symbols.rea_mips_entry,
"--provider",
"ghidra",
"--json",
],
{ env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 },
);
const carvedEvidence = parseEvidence(JSON.parse(carvedCli.stdout));
assert.equal(carvedEvidence.subject?.digest.sha256, carvedSha256);
assert.ok(
carvedEvidence.limitations.some((item) =>
item.includes("section header table"),
),
);
assert.equal(
functionDossierSchema.parse(carvedEvidence.normalized_result).procedure
.address,
carvedDossier.procedure.address,
);
assert.deepEqual(await readFile(carvedPath), carvedBytes);
reports.push({
...fixture,
independent_reader: independent,
@@ -241,6 +341,14 @@ try {
"CLI/MCP and snapshot identity",
],
procedures: procedures.length,
carved: {
sha256: carvedSha256,
source_bytes: carvedBytes.length,
section_header_offset: sectionOffset,
effective_language: carvedImage.observations.language_id,
limitations: carvedResolved.value.mips.limitations,
procedures: carvedProcedures.length,
},
status: "passed",
});
} catch (cause) {
+227
View File
@@ -0,0 +1,227 @@
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { parseBinaryTarget } from "../dist/application/BinaryTargetResolver.js";
import { parseConfig } from "../dist/config/parseConfig.js";
import { GhidraClient } from "../dist/ghidra/GhidraClient.js";
import { inspectGhidraInstallation } from "../dist/ghidra/GhidraInstallation.js";
import { SUPPORTED_GHIDRA_VERSION } from "../dist/ghidra/GhidraInstallationPolicy.js";
import { GhidraProvider } from "../dist/ghidra/GhidraProvider.js";
import { silentLogger } from "../dist/logger.js";
import { cleanupOwnedProcessGroup } from "../dist/process/ProcessOwnership.js";
import {
ProviderProcessSupervisor,
spawnOwnedProviderProcess,
} from "../dist/process/ProviderProcess.js";
import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs";
const exec = promisify(execFile);
const run = createVerifierRun();
const procedureName = (name) =>
process.platform === "darwin" ? `_${name}` : name;
assert.ok(
process.platform === "linux" || process.platform === "darwin",
"This regression lane requires Linux or macOS",
);
assert.ok(
process.arch === "x64" || process.arch === "arm64",
"This regression lane requires x86-64 or arm64",
);
const config = parseConfig(process.env);
if (!config.ok) throw config.error;
const installation = inspectGhidraInstallation({
environment: process.env,
installDir: config.value.ghidraInstallDir,
javaHome: config.value.ghidraJavaHome,
});
assert.equal(installation.status, "available", JSON.stringify(installation));
assert.equal(installation.providerVersion, SUPPORTED_GHIDRA_VERSION);
const fixtures = fileURLToPath(
new URL("../tests/conformance/ghidra", import.meta.url),
);
const bridge = fileURLToPath(new URL("../bridge/ghidra", import.meta.url));
const workspace = await mkdtemp(join(tmpdir(), "rea-ghidra-noreturn-"));
let cleaned = true;
let report;
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
try {
const targetPath = join(
workspace,
process.platform === "darwin" ? "noreturn.dylib" : "noreturn.so",
);
const targetOptions =
process.platform === "darwin"
? [
"-dynamiclib",
"-undefined",
"dynamic_lookup",
"-U_FORTIFY_SOURCE",
"-D_FORTIFY_SOURCE=0",
]
: ["-shared"];
await exec(process.env.REA_CC ?? "cc", [
...targetOptions,
"-fPIC",
"-O0",
"-fno-builtin",
join(fixtures, "no-return.c"),
"-o",
targetPath,
]);
const targetDigest = digest(await readFile(targetPath));
const sourceDigests = Object.fromEntries(
await Promise.all(
["ReaGhidraBridge.java", "ReaGhidraNoReturnFix.java"].map(
async (name) => [name, digest(await readFile(join(bridge, name)))],
),
),
);
const project = join(workspace, "project");
await mkdir(project);
const launch = await spawnOwnedProviderProcess({
command: installation.analyzeHeadlessPath,
arguments: [
project,
"NoReturn",
"-import",
targetPath,
"-readOnly",
"-deleteProject",
"-scriptPath",
`${fixtures};${bridge}`,
"-postScript",
join(fixtures, "ReaNoReturnProbe.java"),
bridge,
],
runId: run.run_id,
expectedCommand: null,
});
const supervisor = new ProviderProcessSupervisor({
...launch,
ownsProcessLifetime: true,
cleanup: () => cleanupOwnedProcessGroup(launch.ownership),
});
let probe;
try {
assert.ok(
await supervisor.waitForExit(120000),
"No-return probe timed out",
);
assert.ok(
await supervisor.waitForOutputClose(1000),
"No-return probe output did not close",
);
const snapshot = supervisor.snapshot();
const output = `${snapshot.stdout.text}\n${snapshot.stderr.text}`;
assert.equal(snapshot.exitCode, 0, output);
const reports = [
...output.matchAll(/REA_NO_RETURN_PROBE_JSON (\{[^\r\n]*\})/gu),
];
assert.equal(reports.length, 1, output);
probe = JSON.parse(reports[0][1]);
assert.equal(probe.status, "passed");
assert.equal(probe.ghidra_version, SUPPORTED_GHIDRA_VERSION);
assert.equal(probe.target_sha256, targetDigest);
for (const [name, expected] of Object.entries(sourceDigests))
assert.equal(probe[name], expected);
assert.ok(probe.checks.includes("independent_entry_preserved"));
assert.ok(probe.checks.includes("propagated_flag_repaired_rea_tls_outer"));
assert.ok(probe.checks.includes("recovered_pseudocode"));
} finally {
const stopped = await supervisor.stop();
cleaned = stopped.status === "verified-cleanup";
assert.ok(
cleaned,
`Owned cleanup incomplete; retained ${workspace}: ${JSON.stringify(stopped)}`,
);
}
// Exercise the actual Java server, authenticated transport, request queue and Evidence
// projection. This read-only TCP lane also runs where AF_UNIX is unavailable; it does
// not claim Windows authority or validation of the default Linux Unix-socket transport.
const provider = new GhidraProvider(
config.value,
silentLogger,
process.env,
undefined,
(options) =>
new GhidraClient({ ...options, transport: "authenticated-loopback-tcp" }),
);
const target = await parseBinaryTarget(targetPath);
if (!target.ok) throw target.error;
const profile = await provider.resolveAnalysisProfile(target.value);
if (!profile.ok) throw profile.error;
assert.ok(profile.value.profile);
assert.equal(
profile.value.profile.parameters.no_return_repair,
"returning-imports-decoded-return-v2",
);
const client = provider.createClient(target.value, profile.value.profile);
cleaned = false;
const observations = [];
const failures = [];
try {
for (const operation of [
"procedure_pseudo_code",
"procedure_assembly",
"procedure_info",
"read_function_instructions",
"analyze_function",
]) {
const result = await client.execute(operation, {
procedure: procedureName("rea_abort"),
});
if (!result.ok) throw result.error;
const warnings = result.value.limitations.filter((value) =>
/terminal call/u.test(value),
);
assert.ok(
warnings.some(
(value) => /abort/u.test(value) && /fallthrough/u.test(value),
),
operation,
);
observations.push({ operation, warnings });
}
const healthy = await client.execute("procedure_pseudo_code", {
procedure: procedureName("rea_tls"),
});
if (!healthy.ok) throw healthy.error;
assert.ok(
!healthy.value.limitations.some((value) => /terminal call/u.test(value)),
);
} catch (cause) {
failures.push(cause);
}
const closed = await client.close();
cleaned = closed.ok;
if (!closed.ok) failures.push(closed.error);
if (failures.length > 0)
throw new AggregateError(
failures,
"No-return Evidence verification failed",
);
assert.equal(digest(await readFile(targetPath)), targetDigest);
report = {
status: "passed",
probe,
observations,
scope:
"Real Ghidra ELF or Mach-O analysis with injected flags; real read-only TCP Evidence projection. No target execution; exact Amethyst artifact and Windows are unverified.",
};
} finally {
if (cleaned) await rm(workspace, { recursive: true, force: true });
}
console.log(
JSON.stringify({
...report,
cleanup: "complete",
verifier_run: await completeVerifierRun(run),
}),
);
+1 -1
View File
@@ -322,7 +322,7 @@ async function functionOperation(operation, parameters) {
const result = parseGhidraFunctionResult(operation, called.value);
if (!result.ok) throw result.error;
observed.add(operation);
return result.value;
return result.value.value;
}
function assertSession(session, targetSha256, profileDigest) {
+53 -17
View File
@@ -38,6 +38,7 @@ import {
verifyJumpTableDenseSwitch,
verifyNativeTypeLayout,
verifyRelativeSwitch,
verifyRelativeSignedSwitch,
verifyNativeValueTrace,
} from "./verify-real-ghidra-function.mjs";
const exec = promisify(execFile);
@@ -173,6 +174,36 @@ try {
]);
}
}
const signedRelativeSource = fileURLToPath(
new URL(
"../tests/conformance/ghidra/relative-switch-signed.S",
import.meta.url,
),
);
for (const selectorBias of [0, 2, 10]) {
for (const format of [
"elf",
...(expectedNativeTarget.architecture === "arm64" ? ["mach-o"] : []),
]) {
const targetPath = join(
fixtureRoot,
`relative-signed-${selectorBias}-${format}.o`,
);
await exec(clang, [
...(format === "elf" ? ["--target=aarch64-linux-gnu"] : []),
`-DREA_SELECTOR_BIAS=${selectorBias}`,
"-c",
signedRelativeSource,
"-o",
targetPath,
]);
crossTargets.push([
targetPath,
`relative-signed-${selectorBias}`,
{ format, architecture: "arm64" },
]);
}
}
await exec(clang, [
"--target=aarch64-linux-gnu",
@@ -522,25 +553,30 @@ async function verifyTarget(targetPath, variant, expectedTarget = null) {
client,
})
: null;
const probes = variant.startsWith("relative-")
? await verifyRelativeSwitch(
const probes = variant.startsWith("relative-signed-")
? await verifyRelativeSignedSwitch(
client,
procedures,
Number(variant.slice(-1)),
Number(variant.split("-").at(-1)),
)
: variant === "type-layout"
? await verifyNativeTypeLayout(client, names)
: variant === "jump-table"
? await verifyJumpTableDenseSwitch(client, procedures)
: variant === "custom" || variant === "aarch64-jump-table"
? null
: await verifyInventoryOperations({
client,
variant,
procedures,
names,
strings,
});
: variant.startsWith("relative-")
? await verifyRelativeSwitch(
client,
procedures,
Number(variant.slice(-1)),
)
: variant === "type-layout"
? await verifyNativeTypeLayout(client, names)
: variant === "jump-table"
? await verifyJumpTableDenseSwitch(client, procedures)
: variant === "custom" || variant === "aarch64-jump-table"
? null
: await verifyInventoryOperations({
client,
variant,
procedures,
names,
strings,
});
const nativeValues =
variant === "debug"
? await verifyNativeValueTrace(client, procedures, parsedTarget.value)
+1 -5
View File
@@ -1,7 +1,6 @@
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { access, mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { access, writeFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { promisify } from "node:util";
@@ -25,8 +24,6 @@ const electronPath = join(fixtureRoot, "electron");
const environment = {
...process.env,
};
const temporaryRoot = await mkdtemp(join(tmpdir(), "rea-readiness-"));
await access(nativePath);
const nativeEvidence = await runCli(["inspect-artifact", nativePath, "--json"]);
const javascriptCliEvidence = await runCli([
@@ -135,7 +132,6 @@ try {
);
} finally {
await client.close();
await rm(temporaryRoot, { recursive: true, force: true });
}
async function runCli(arguments_) {
+221 -8
View File
@@ -1,6 +1,8 @@
"""Check website references, search metadata and the sole production publisher."""
from collections import Counter
from html.parser import HTMLParser
import os
from pathlib import Path
import re
import struct
@@ -10,7 +12,21 @@ from urllib.parse import unquote, urlsplit
import xml.etree.ElementTree as ET
from zipfile import BadZipFile, ZipFile
from website import NOTES_FIXTURES, SITE_ORIGIN, authored_markdown, page_url, sync_pages
from website import (
LANGUAGES,
NOTES_FIXTURES,
SITE_ORIGIN,
SiteState,
authored_markdown,
code_block,
copy_targets,
page_language,
page_url,
source_digest,
sync_pages,
synced_code_ids,
translation_record,
)
LEGACY_ORIGIN = "morluto.github.io"
SITE_TEXT_URL = re.compile(
@@ -167,8 +183,9 @@ def check_search_metadata(site, pages):
title = page.titles[0].strip() if len(page.titles) == 1 else ""
if not title:
errors.append(f"{label}: needs exactly one nonempty head title")
if page.lang != "en" or page.h1_count != 1:
errors.append(f"{label}: needs lang=en and exactly one h1")
tag = LANGUAGES[page_language(site, path)[0]].tag
if page.lang != tag or page.h1_count != 1:
errors.append(f"{label}: needs lang={tag} and exactly one h1")
if label in NOTES_FIXTURES and not page.noindex():
errors.append(
f"{label}: example application must have a head noindex directive"
@@ -280,15 +297,21 @@ def check_search_metadata(site, pages):
return len(indexable), errors
def check_site(site):
"""Check assets, HTML fragment destinations and SVG syntax."""
def parse_pages(site):
"""Parse every HTML page once for reference, metadata and translation checks."""
pages = {}
errors = []
references = 0
for path in sorted(site.rglob("*.html")):
parser = HtmlReferences()
parser.feed(path.read_text(encoding="utf-8"))
pages[path] = parser
return pages
def check_site(site):
"""Check assets, HTML fragment destinations and SVG syntax."""
pages = parse_pages(site)
errors = []
references = 0
if site / "index.html" not in pages:
errors.append("website/public/index.html is missing")
@@ -346,6 +369,190 @@ def check_site(site):
return len(pages), references, indexable, errors
# Han, kana and CJK punctuation; a space between two of them renders as a gap.
CJK_GAP = re.compile(
r"([、-〿぀-ヿ㐀-䶿一-鿿豈-﫿!-￯])"
r"\s+([、-〿぀-ヿ㐀-䶿一-鿿豈-﫿!-￯])"
)
CODE_TAGS = {"code", "kbd", "pre", "script", "style"}
class TranslationText(HTMLParser):
"""Collect prose, inline code and message attributes for comparing translations."""
def __init__(self, synced):
super().__init__()
self.synced = set(synced)
self.stack = []
self.prose = []
self.inline_code = Counter()
self.messages = Counter()
def handle_starttag(self, tag, attrs):
attributes = dict(attrs)
self.messages.update(name for name in attributes if name.startswith("data-msg-"))
if tag in CODE_TAGS:
self.stack.append([tag, attributes.get("id"), ""])
def handle_startendtag(self, tag, attrs):
attributes = dict(attrs)
self.messages.update(name for name in attributes if name.startswith("data-msg-"))
def handle_endtag(self, tag):
if self.stack and self.stack[-1][0] == tag:
name, identifier, text = self.stack.pop()
if name in ("code", "kbd") and identifier not in self.synced:
self.inline_code[" ".join(text.split())] += 1
if self.stack:
self.stack[-1][2] += text
def handle_data(self, data):
if self.stack:
self.stack[-1][2] += data
else:
self.prose.append(data)
def translation_warning(path, root, message):
"""Return a warning tied to a repository file."""
return (path.relative_to(root).as_posix(), message)
def check_translations(root, pages):
"""Check translation records, copied code, structure parity and CJK spacing.
A changed English source is a warning: its translation keeps working, shows a
notice after sync and is updated later. Errors cover what sync or the
translator controls directly.
"""
site = root / "website/public"
errors = []
warnings = []
try:
state = SiteState(root)
except ValueError as error:
return [str(error)], []
for path, page in pages.items():
language, english = page_language(site, path)
if language == "en":
continue
label = path.relative_to(site).as_posix()
source = path.read_text(encoding="utf-8")
record = translation_record(source)
english_path = site / english
if not english_path.is_file():
errors.append(f"{label}: translates {english}, which does not exist")
continue
english_source = english_path.read_text(encoding="utf-8")
if record is None:
errors.append(f"{label}: needs one <!-- website:translation source-sha256=... status=... --> record")
continue
status = record.get("status")
if status not in ("draft", "reviewed"):
errors.append(f"{label}: translation status must be draft or reviewed")
if status == "draft" and not page.noindex():
errors.append(f"{label}: a draft translation needs <meta name=\"robots\" content=\"noindex\" />")
if status == "reviewed":
if page.noindex():
errors.append(f"{label}: remove noindex from a reviewed translation")
if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", record.get("reviewed", "")):
errors.append(f"{label}: a reviewed translation needs reviewed=\"YYYY-MM-DD\"")
fresh = record.get("source-sha256") == source_digest(english_source)
if not fresh:
warnings.append(translation_warning(
path, root,
f"The English page {english} has changed since this translation. Update the translation, "
"then set source-sha256 to the new value from python3 scripts/website.py translations.",
))
synced = synced_code_ids(english_source)
for identifier in synced:
expected = code_block(identifier).search(english_source).group(2)
found = code_block(identifier).findall(source)
if len(found) != 1 or found[0][1] != expected:
errors.append(f"{label}: code block {identifier} must match the English page; run sync")
english_page = pages[english_path]
differences = []
if page.ids != english_page.ids:
differences.append(
f"element IDs differ (missing {sorted(english_page.ids - page.ids)}, extra {sorted(page.ids - english_page.ids)})"
)
if copy_targets(source) != copy_targets(english_source):
differences.append("copy buttons differ from the English page")
translated = TranslationText(synced)
translated.feed(source)
original = TranslationText(synced)
original.feed(english_source)
if translated.messages != original.messages:
differences.append("data-msg-* message attributes differ from the English page")
for difference in differences:
if fresh:
errors.append(f"{label}: {difference}")
else:
warnings.append(translation_warning(path, root, difference))
if fresh and translated.inline_code != original.inline_code:
changed = sorted((translated.inline_code - original.inline_code) + (original.inline_code - translated.inline_code))
warnings.append(translation_warning(
path, root, f"inline code differs from the English page: {', '.join(repr(text) for text in changed[:5])}",
))
if language in ("ja", "zh-hans", "zh-hant"):
for text in translated.prose:
for match in CJK_GAP.finditer(text):
errors.append(
f"{label}: whitespace between {match.group(1)!r} and {match.group(2)!r} renders as a gap; "
"keep CJK text on one line without spaces"
)
for reference in page.references:
url = urlsplit(reference)
if url.scheme or url.netloc or not url.path or reference.startswith("/"):
continue
target = (path.parent / unquote(url.path)).resolve()
if target.is_dir():
target /= "index.html"
if not target.is_relative_to(site.resolve()):
continue
target_language, target_english = page_language(site.resolve(), target)
if (
target_language == "en"
and target_english != english
and state.translations.get((language, target_english)) == "reviewed"
):
warnings.append(translation_warning(
path, root, f"links to the English {target_english}; link to its reviewed translation instead",
))
english_strings = state.locales["en"]["strings"]
for code in LANGUAGES:
if code == "en" or not any(key[0] == code for key in state.translations):
continue
locale = state.locales[code]
same = sorted(
key for key, value in locale["strings"].items()
if value == english_strings[key] and key not in locale.get("sameAsEnglish", [])
)
if same:
warnings.append((f"website/locales/{code}.json", f"UI strings still in English: {', '.join(same)}"))
return errors, warnings
def report_warnings(warnings):
"""Print warnings, as GitHub annotations and a job summary when running in Actions."""
for file, message in warnings:
print(f"Warning: {file}: {message}")
if os.environ.get("GITHUB_ACTIONS") != "true" or not warnings:
return
for file, message in warnings:
print(f"::warning file={file}::{message}")
summary = os.environ.get("GITHUB_STEP_SUMMARY")
if summary:
with open(summary, "a", encoding="utf-8") as output:
output.write("### Website warnings\n\n")
output.writelines(f"- `{file}`: {message}\n" for file, message in warnings)
def check_publisher(root):
"""Detect a second Pages action that could replace the website."""
expected = root / ".github/workflows/website-pages.yml"
@@ -404,10 +611,16 @@ def main():
errors.extend(check_example_archive(root / "website/public"))
errors.extend(check_publisher(root))
errors.extend(check_repository_links(root))
translation_errors, warnings = check_translations(root, parse_pages(root / "website/public"))
errors.extend(translation_errors)
try:
sync_pages(root, check=True)
for path in sync_pages(root, check=True):
warnings.append(translation_warning(
path, root, "the out-of-date notice needs refreshing; run python3 scripts/website.py sync",
))
except (OSError, ValueError, KeyError) as error:
errors.append(str(error))
report_warnings(warnings)
if errors:
print("Website checks failed:\n" + "\n".join(errors), file=sys.stderr)
return 1
+71 -1
View File
@@ -6,6 +6,7 @@ import { createHash } from "node:crypto";
import { once } from "node:events";
import {
access,
copyFile,
mkdir,
mkdtemp,
readFile,
@@ -16,7 +17,7 @@ import {
open,
} from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { dirname, join, parse, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import { verifyWindowsProtocolInput } from "./lib/windows-protocol-input.mjs";
@@ -32,6 +33,9 @@ const native = requireWindowsNativeAuthority();
const { WindowsPrivateRuntime, windowsPrivateRuntime } = await import(
pathToFileURL(join(packageRoot, "dist/windows/WindowsPrivateRuntime.js"))
);
const { ghidraSessionRoot } = await import(
pathToFileURL(join(packageRoot, "dist/ghidra/GhidraSessionRoot.js"))
);
const workspace = await mkdtemp(join(tmpdir(), "rea-native-conformance-"));
const environment = Object.entries(process.env)
.filter(([, value]) => value !== undefined)
@@ -77,6 +81,31 @@ const wait = async (child) => {
};
const report = { ok: false, artifact: "verified", controls: {}, processes: {} };
try {
// Exercise the loader in a process whose image is not named node.exe. The
// ordinary Windows lane otherwise cannot detect eager Node-API imports.
const renamedHost = join(workspace, "rea-renamed-node.exe");
const renamedHostFixture = join(workspace, "renamed-host.mjs");
const loaderUrl = pathToFileURL(
join(packageRoot, "dist/windows/WindowsNativeLoader.js"),
).href;
await copyFile(process.execPath, renamedHost);
await writeFile(
renamedHostFixture,
`import {basename} from 'node:path';
import {requireWindowsNativeAuthority} from ${JSON.stringify(loaderUrl)};
const {inspection}=requireWindowsNativeAuthority();
process.stdout.write(JSON.stringify({executable:basename(process.execPath),abiVersion:inspection.abiVersion}));\n`,
);
const renamedHostResult = execFileSync(renamedHost, [renamedHostFixture], {
encoding: "utf8",
timeout: 20_000,
windowsHide: true,
});
assert.deepEqual(JSON.parse(renamedHostResult), {
executable: "rea-renamed-node.exe",
abiVersion: 1,
});
report.controls.renamedHostNodeApi = true;
const sourceDirectory = join(workspace, "source");
await mkdir(sourceDirectory);
const source = join(sourceDirectory, "target.bin");
@@ -110,6 +139,47 @@ try {
])
assert.throws(() => native.call("open", [rejected]));
report.controls.ordinaryDriveSeparatorsAndRequestedIdentity = true;
const driveRoot = parse(workspace).root;
const runtimeParent = ghidraSessionRoot({
base: join(driveRoot, ".Tmp", "scratch"),
});
assert.equal(runtimeParent, driveRoot);
const driveRootRuntime = native.call("runtime_create", [
runtimeParent,
"rea-drive-root-",
]);
try {
assert.equal(driveRootRuntime.path.startsWith(`${driveRoot}\\`), false);
assert.equal(dirname(driveRootRuntime.path), driveRoot);
const snapshot = await native.call("runtime_snapshot", [
driveRootRuntime.handle,
source,
"probe.bin",
]);
const sourceBytes = await readFile(source);
assert.equal(
snapshot.sha256,
createHash("sha256").update(sourceBytes).digest("hex"),
);
assert.equal(snapshot.source.requestedPath, source);
const readback = native.call("runtime_open", [
driveRootRuntime.handle,
"probe.bin",
]);
try {
assert.equal(readback.size, sourceBytes.byteLength);
assert.deepEqual(
native.call("read", [readback.handle, 0, 65536]),
sourceBytes.subarray(0, 65536),
);
} finally {
native.call("close", [readback.handle]);
}
} finally {
native.call("runtime_close", [driveRootRuntime.handle]);
}
assert.equal(await exists(driveRootRuntime.path), false);
report.controls.driveRootRuntimeParent = true;
const snapshotOwner = WindowsPrivateRuntime.create(
workspace.replace(/\\/u, "/"),
"single-flight-",
+59 -6
View File
@@ -26,22 +26,49 @@ export async function verifyElectronToolContracts({
await mkdtemp(join(tmpdir(), "rea-electron-e2e-")),
);
const main = join(root, "main.cjs");
const preload = join(root, "preload.cjs");
const ready = join(root, "ready");
const entrypoint = join(repositoryRoot, "scripts", "rea.mjs");
const runId = randomUUID();
await writeFile(
main,
`const {app,BrowserWindow}=require("electron");
app.whenReady().then(()=>{const window=new BrowserWindow();
window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringify(ready)},"ready");});`,
`const electron = require("electron");
const path = require("node:path");
const { app, BrowserWindow, ipcMain } = electron;
const Window = BrowserWindow;
const mainIpc = ipcMain;
app.whenReady().then(() => {
const window = new Window({
webPreferences: { preload: path.join(__dirname, "preload.cjs") },
});
window.loadFile("renderer.html");
require("node:fs").writeFileSync(${JSON.stringify(ready)}, "ready");
});
mainIpc.handle("fixture:echo", (_event, value) => ({ value }));`,
);
await writeFile(
join(root, "renderer.html"),
'<!doctype html><button id="run">Run</button><script src="renderer.js"></script>',
`<!doctype html>
<button id="run">Run</button>
<output id="result"></output>
<script src="renderer.js"></script>`,
);
await writeFile(
preload,
`const electron = require("electron");
const { contextBridge, ipcRenderer } = electron;
const bridge = contextBridge;
const rendererIpc = ipcRenderer;
bridge.exposeInMainWorld("fixture", {
echo: (value) => rendererIpc.invoke("fixture:echo", value),
});`,
);
await writeFile(
join(root, "renderer.js"),
'document.querySelector("#run").addEventListener("click",()=>console.log("fixture-click"));',
`document.querySelector("#run").addEventListener("click", async () => {
const result = await window.fixture.echo("renderer");
document.querySelector("#result").textContent = result.value;
});`,
);
const transport = new StdioClientTransport({
command: process.execPath,
@@ -135,6 +162,25 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
const staticAnalysis = await call("analyze_javascript_application", {
input_path: root,
});
assert.equal(staticAnalysis.normalized_result.summary.browser_windows, 1);
assert.equal(
staticAnalysis.normalized_result.summary.preload_entrypoints,
1,
);
assert.equal(
staticAnalysis.normalized_result.summary.context_bridge_apis,
1,
);
assert.equal(staticAnalysis.normalized_result.summary.ipc.main_handlers, 1);
assert.equal(
staticAnalysis.normalized_result.summary.ipc.renderer_transmissions,
1,
);
assert.equal(
staticAnalysis.normalized_result.summary.ipc
.paired_renderer_transmissions,
1,
);
const reconciled = await call("reconcile_javascript_runtime", {
static_layers: [
{
@@ -169,7 +215,7 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
executable_path: executable,
application_path: main,
application_root: root,
actions: [{ step_id: "settle", kind: "wait", duration_ms: 50 }],
actions: [{ step_id: "ipc", kind: "click", selector: "#run" }],
};
const active = await client.callTool(
{ name: "capture_electron_scenario", arguments: input },
@@ -192,6 +238,13 @@ window.loadFile("renderer.html");require("node:fs").writeFileSync(${JSON.stringi
assert.equal(capture.application.cleanup, "terminated-owned-process");
}
assert.equal(capture.actions[0].status, "completed");
assert.ok(
capture.ipc.events.some(
({ kind, channel }) =>
kind === "main-handler-invocation" && channel === "fixture:echo",
),
JSON.stringify(capture.ipc),
);
assert.equal(
capture.coverage.status,
"partial_attach",

Some files were not shown because too many files have changed in this diff Show More