Compare commits

..
Author SHA1 Message Date
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>morluto
6ca98eaa82 chore(release/6.3.0): release rea-agents 6.3.0 (#1381)
* chore(release/6.3.0): release rea-agents 6.3.0

* docs(release): restore 6.3 capture migration notes

* docs(release): include omitted 6.3 ancestry entries

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-10 07:30:30 +08:00
142 changed files with 594 additions and 4777 deletions
-20
View File
@@ -64,14 +64,6 @@ jobs:
env:
REA_ANALYSIS_PROVIDER: ghidra
run: npm run verify:ghidra:windows -- --x86 | Tee-Object -FilePath windows-ghidra-x86-proof.log
- name: Verify native x64 DLL Windows Ghidra operations
env:
REA_ANALYSIS_PROVIDER: ghidra
run: npm run verify:ghidra:windows -- --dll | Tee-Object -FilePath windows-ghidra-dll-proof.log
- name: Verify native x86 DLL Windows Ghidra operations
env:
REA_ANALYSIS_PROVIDER: ghidra
run: npm run verify:ghidra:windows -- --x86 --dll | Tee-Object -FilePath windows-ghidra-x86-dll-proof.log
- name: Verify ordinary-user CLI and MCP operations
env:
REA_ANALYSIS_PROVIDER: ghidra
@@ -80,14 +72,6 @@ jobs:
env:
REA_ANALYSIS_PROVIDER: ghidra
run: npm run verify:ghidra:windows:package -- --x86 | Tee-Object -FilePath windows-ghidra-x86-package-proof.log
- name: Verify ordinary-user x64 DLL CLI and MCP operations
env:
REA_ANALYSIS_PROVIDER: ghidra
run: npm run verify:ghidra:windows:package -- --dll | Tee-Object -FilePath windows-ghidra-dll-package-proof.log
- name: Verify ordinary-user x86 DLL CLI and MCP operations
env:
REA_ANALYSIS_PROVIDER: ghidra
run: npm run verify:ghidra:windows:package -- --x86 --dll | Tee-Object -FilePath windows-ghidra-x86-dll-package-proof.log
- uses: actions/upload-artifact@v4
if: always()
with:
@@ -97,8 +81,4 @@ jobs:
windows-ghidra-package-proof.log
windows-ghidra-x86-proof.log
windows-ghidra-x86-package-proof.log
windows-ghidra-dll-proof.log
windows-ghidra-x86-dll-proof.log
windows-ghidra-dll-package-proof.log
windows-ghidra-x86-dll-package-proof.log
retention-days: 30
-1
View File
@@ -66,7 +66,6 @@
* **javascript:** resolve exported callables through lexical bindings ([#1351](https://github.com/morluto/rea/issues/1351)) ([a9f5dad](https://github.com/morluto/rea/commit/a9f5dad5a8f196c88f6305d8d958b355cc0b52c2))
* **javascript:** resolve query-only HTML references to their document ([55a46f7](https://github.com/morluto/rea/commit/55a46f7c7176075c9edda4fe61ed995f2ceadf6a))
* **javascript:** retain array identity after length mutations ([4bcea1a](https://github.com/morluto/rea/commit/4bcea1aa76e3e7ca665197181739c5e27a62c40a))
* **javascript:** retain conditional mutation ordering uncertainty ([b8a8190](https://github.com/morluto/rea/commit/b8a8190b2f30e377eee2d69d1ba31ea454fa70b9))
* **javascript:** retain distinct module provenance path components ([2bd7de6](https://github.com/morluto/rea/commit/2bd7de6f8ca62fc1d48334cfa7fdced461b7b153))
* **javascript:** retain unresolved value frontiers in provenance queries ([3e19875](https://github.com/morluto/rea/commit/3e19875c7fa9c61cff655bbe6046ddc4ab558533))
* **javascript:** store source map URLs once ([ed8fcae](https://github.com/morluto/rea/commit/ed8fcae94b699b62b44162f880faa077a36c455d))
+1 -1
View File
@@ -307,7 +307,7 @@ Report vulnerabilities through [SECURITY.md](SECURITY.md).
## Star history
🎉 **50,000 GitHub stars — thank you!**
🎉 **40,000 GitHub stars — thank you!**
Thanks to everyone using REA, reporting bugs, requesting features, testing builds, and contributing fixes.
+1 -1
View File
@@ -258,7 +258,7 @@ npx rea-agents@latest setup
## سجل النجوم
🎉 **50,000 نجمة على GitHub — شكرًا لكم!**
🎉 **40,000 نجمة على GitHub — شكرًا لكم!**
شكرًا لكل من يستخدم REA، ويبلّغ عن الأخطاء، ويقترح ميزات، ويختبر إصدارات البناء، ويساهم بالإصلاحات.
+1 -1
View File
@@ -256,7 +256,7 @@ Melde Sicherheitslücken gemäß [SECURITY.md](SECURITY.md).
## Sternverlauf
🎉 **50.000 GitHub-Sterne – vielen Dank!**
🎉 **40.000 GitHub-Sterne – vielen Dank!**
Danke an alle, die REA nutzen, Fehler melden, Funktionswünsche äußern, Builds testen und Fehlerbehebungen beitragen.
+1 -1
View File
@@ -256,7 +256,7 @@ Informa de vulnerabilidades siguiendo [SECURITY.md](SECURITY.md).
## Historial de estrellas
🎉 **50.000 estrellas en GitHub: ¡gracias!**
🎉 **40.000 estrellas en GitHub: ¡gracias!**
Gracias a quienes usan REA, informan de errores, solicitan funciones, prueban compilaciones y contribuyen con correcciones.
+1 -1
View File
@@ -241,7 +241,7 @@ npx rea-agents@latest setup
## تاریخچهٔ ستاره‌ها
🎉 **۵۰٬۰۰۰ ستارهٔ GitHub — سپاسگزاریم!**
🎉 **۴۰٬۰۰۰ ستارهٔ GitHub — سپاسگزاریم!**
از همهٔ کسانی که از REA استفاده می‌کنند، خطاها را گزارش می‌دهند، قابلیت‌های جدید پیشنهاد می‌کنند، buildها را آزمایش می‌کنند و در بهبود پروژه مشارکت دارند سپاسگزاریم.
+1 -1
View File
@@ -257,7 +257,7 @@ Signalez les vulnérabilités via [SECURITY.md](SECURITY.md).
## Historique des étoiles
🎉 **50 000 étoiles GitHub — merci !**
🎉 **40 000 étoiles GitHub — merci !**
Merci à toutes les personnes qui utilisent REA, signalent des bugs, proposent des fonctionnalités, testent les builds et contribuent des correctifs.
+1 -1
View File
@@ -258,7 +258,7 @@ npx rea-agents@latest setup
## スター履歴
🎉 **GitHub スター 50,000 件、ありがとうございます!**
🎉 **GitHub スター 40,000 件、ありがとうございます!**
REA の利用、バグ報告、機能の要望、ビルドのテスト、修正への貢献に感謝します。
+1 -1
View File
@@ -258,7 +258,7 @@ npx rea-agents@latest setup
## 스타 기록
🎉 **GitHub 스타 50,000개, 감사합니다!**
🎉 **GitHub 스타 40,000개, 감사합니다!**
REA를 사용하고, 버그를 보고하고, 기능을 요청하고, 빌드를 테스트하고, 수정에 기여해 주신 모든 분께 감사드립니다.
+1 -1
View File
@@ -256,7 +256,7 @@ Zgłaszaj podatności zgodnie z [SECURITY.md](SECURITY.md).
## Historia gwiazdek
🎉 **50 000 gwiazdek na GitHubie — dziękujemy!**
🎉 **40 000 gwiazdek na GitHubie — dziękujemy!**
Dziękujemy wszystkim, którzy używają REA, zgłaszają błędy, proponują nowe funkcje, testują kompilacje i przesyłają poprawki.
+1 -1
View File
@@ -256,7 +256,7 @@ Relate vulnerabilidades conforme [SECURITY.md](SECURITY.md).
## Histórico de estrelas
🎉 **50.000 estrelas no GitHub — muito obrigado!**
🎉 **40.000 estrelas no GitHub — muito obrigado!**
Obrigado a todos que usam o REA, relatam bugs, sugerem funcionalidades, testam builds e contribuem com correções.
+1 -1
View File
@@ -256,7 +256,7 @@ npx rea-agents@latest setup
## История звёзд
🎉 **50 000 звёзд на GitHub — спасибо!**
🎉 **40 000 звёзд на GitHub — спасибо!**
Спасибо всем, кто использует REA, сообщает об ошибках, предлагает новые функции, тестирует сборки и помогает с исправлениями.
+1 -1
View File
@@ -256,7 +256,7 @@ npx rea-agents@latest setup
## ประวัติดาว
🎉 **50,000 ดาวบน GitHub — ขอบคุณทุกคน!**
🎉 **40,000 ดาวบน GitHub — ขอบคุณทุกคน!**
ขอบคุณทุกคนที่ใช้ REA รายงานบั๊ก เสนอฟีเจอร์ ทดสอบบิลด์ และช่วยแก้ไข
+1 -1
View File
@@ -256,7 +256,7 @@ Güvenlik açıklarını [SECURITY.md](SECURITY.md) üzerinden bildirin.
## Yıldız geçmişi
🎉 **GitHub'da 50.000 yıldız — teşekkürler!**
🎉 **GitHub'da 40.000 yıldız — teşekkürler!**
REA'yı kullanan, hata bildiren, özellik talep eden, derlemeleri test eden ve düzeltmelere katkıda bulunan herkese teşekkürler.
+1 -1
View File
@@ -256,7 +256,7 @@ npx rea-agents@latest setup
## Історія зірок
🎉 **50 000 зірок на GitHub — дякуємо!**
🎉 **40 000 зірок на GitHub — дякуємо!**
Дякуємо всім, хто використовує REA, повідомляє про помилки, пропонує нові функції, тестує збірки й допомагає з виправленнями.
+1 -1
View File
@@ -256,7 +256,7 @@ Báo cáo lỗ hổng theo [SECURITY.md](SECURITY.md).
## Lịch sử sao
🎉 **50.000 sao trên GitHub — xin cảm ơn!**
🎉 **40.000 sao trên GitHub — xin cảm ơn!**
Cảm ơn mọi người đã sử dụng REA, báo lỗi, đề xuất tính năng, kiểm thử bản dựng và đóng góp bản sửa lỗi.
+1 -1
View File
@@ -258,7 +258,7 @@ npx rea-agents@latest setup
## Star 歷史
🎉 **GitHub Star 達到 50,000 個,感謝大家!**
🎉 **GitHub Star 達到 40,000 個,感謝大家!**
感謝每一位使用 REA、報告錯誤、提出功能需求、測試建置和貢獻修復的朋友。
+1 -1
View File
@@ -258,7 +258,7 @@ npx rea-agents@latest setup
## Star 历史
🎉 **GitHub Star 达到 50,000 个,感谢大家!**
🎉 **GitHub Star 达到 40,000 个,感谢大家!**
感谢每一位使用 REA、报告错误、提出功能需求、测试构建和贡献修复的朋友。
-1
View File
@@ -18,7 +18,6 @@ REA can attach to a user-owned Chrome-family browser through the Chrome DevTools
- `compare_web_captures` / `rea compare-web-captures` compares passive captures or exact-step-aligned browser scenarios. Scenario results commit reproducible literal normalization and expose alignment failures plus artifact-level action, screenshot, DOM, accessibility, URL, history, storage, and event differences. Missing or truncated evidence is never treated as equivalence.
- `capture_web_screenshot` / `rea capture-web-screenshot` returns a content-addressed visible-viewport PNG.
- `compare_web_screenshots` / `rea compare-web-screenshots` performs bounded local PNG pixel comparison without OCR or external services.
- The CLI comparison commands accept each capture, screenshot artifact or normalization policy as inline JSON or a JSON file path. Use files for screenshot artifacts and scenario captures, whose embedded PNG bytes can exceed the host's command-line length limit.
- Every surface has equivalent CLI and MCP contracts and returns Evidence provenance.
- MCP tools return complete results inline and include the Evidence ID. Session evidence can be exported with `export_evidence_bundle`.
+1 -21
View File
@@ -20,23 +20,6 @@ files when neither `--provider` nor `--snapshot` is supplied. See
[JavaScript artifact reconstruction](javascript-artifact-reconstruction.md)
for results, integrity checks and coverage.
The complete application Evidence can be hundreds of megabytes for a real
application. When an agent or a person reads the result, save it to a file and
project a summary, a module page or one module from the saved Evidence:
```bash
rea analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
rea inspect-analysis-view app-view.json
```
For example, a 334 MB Obsidian application Evidence file yields a summary view
of about 10 KB. Replace the view with
`{"kind": "page", "collection": "modules", "offset": 0, "limit": 32}` for a
module page. See
[JavaScript application workflows](javascript-application-workflows.md) for
views, feature traces and comparisons that take the same saved Evidence.
## Native analysis
Configure [Hopper or Ghidra](installation.md#hopper), or the
@@ -196,10 +179,7 @@ user permissions; consult the chosen guide for host requirements and effects.
## Output and exit status
The default terminal format is TOON. Use `--json` when saving results for a
JSON consumer. `--json` is indented on a terminal and compact when piped or
redirected, which keeps large results smaller for agents and scripts; pipe
through `jq .` for indented files. Output selection and formatting do not change
operation status.
JSON consumer. Output selection and formatting do not change operation status.
| Status | Meaning |
| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
+2 -5
View File
@@ -62,14 +62,11 @@ Mach-O target coverage. It requires `clang`, LLD, and `lld-link`; set
these tools before compilation. Keeping this matrix separate lets Linux
host/provider acceptance run with only the host compiler.
`npm run verify:ghidra:windows` checks source-owned native x86-64 PE applications;
`-- --x86` selects x86 and `-- --dll` selects DLLs, and the flags can be combined.
These lanes check all 25 read-only operations, target/snapshot/import SHA-256 linkage,
`npm run verify:ghidra:windows` and its `-- --x86` variant check the source-owned native x86 and x86-64 PE
fixture, all 25 read-only operations, target/snapshot/import SHA-256 linkage,
and project, endpoint, process, and runtime cleanup. The independent native
lane checks DACLs, handle admission, cancellation, and Job Object lifecycle.
`npm run verify:ghidra:windows:package` packs and installs REA into an isolated
prefix, then checks ordinary-user CLI/MCP operations against canonical contracts.
It accepts the same architecture and DLL flags and verifies the selected PE role
in the analysis profile.
All lanes require matching native controls; package startup alone establishes
only package compatibility.
+1 -1
View File
@@ -489,7 +489,7 @@ entry comments atomically and returns refreshed analysis. These session metadata
edits leave executable bytes unchanged and are discarded on close. GUI controls
require Hopper; Windows P0 remains read-only.
Windows P0 admits native x86 and x86-64 PE applications and DLLs on fixed local NTFS volumes.
Windows P0 admits native x86 and x86-64 PE applications on fixed local NTFS volumes.
The npm package bundles native Job Object ownership, protected private runtime
DACLs, and handle-based path admission; no separate addon installation is needed.
See the [Windows Ghidra P0 guide](windows-ghidra-p0.md) for verified scope and
+2 -42
View File
@@ -90,44 +90,6 @@ keep the result ambiguous. Runtime Evidence can later corroborate an exact
mapped candidate, but structural reachability, semantic influence, runtime
observation, and causal proof remain separate claims.
Semantic literal nodes keep their complete value in `properties.value`; literal
queries match that field. String labels use `string literal` when that is
shorter than the complete JSON value. A literal's `identity.role_key` commits
its canonical JSON value as `value-sha256:<digest>` when the digest key is
shorter; smaller values keep their existing JSON key. This keeps large values
out of identity and display metadata without expanding short values or
truncating their Evidence.
New analyses therefore produce different IDs for hashed literals from the
earlier payload-bearing role keys. Use IDs returned by the selected parent
graph and read literal values from `properties.value`, rather than decoding
role keys or labels. Previously stored graphs and their original IDs remain
valid inputs.
Endpoint nodes keep the exact endpoint in each observation's `properties.value`;
semantic request and response nodes keep it in `properties.endpoint`. Endpoint
identity keys use `value-sha256:<digest>` over the canonical JSON string when
that is shorter, or when the original value starts with the reserved digest
prefix. Other short endpoint keys remain unchanged. Endpoint labels use the
shorter of the exact value and `<kind> endpoint`; semantic request/response
labels use the shorter of the endpoint and operation method. These labels are
display metadata, not endpoint lookup values.
New analyses therefore change hashed endpoint IDs and longer endpoint labels.
Use IDs from the selected graph and read the complete endpoint properties;
previously stored graphs still support their original IDs. Literal endpoint
search, semantic tracing, and version comparison use the preserved values.
Event and listener nodes likewise preserve the complete event name in
`properties.event_name`. Event role keys use the same reserved-prefix string
digest rule, scoped to the emitter; repeated registrations, removals, and
dispatches keep one event per exact emitter/name pair. Long event labels use
`event`, and listener labels use the shorter of `<method>:<name>` and
`<method>:listener`. Dynamic names remain `null` with explicit uncertainty. New analyses
change hashed event IDs and longer event/listener labels; query the preserved
name or select IDs from the parent graph. Previously stored graphs keep their
original IDs and remain valid trace inputs.
## Version comparison
REA pairs entities only when a tier produces one unique candidate on each side.
@@ -183,10 +145,8 @@ Direct return expressions, including expression-bodied arrows, are evaluated
through an execution-free value lattice. Literal object fields and direct
return sites are represented in the result. Calls, dynamic spreads, computed
keys, and parser recovery remain partial or unknown.
Objects and arrays passed to calls, constructors, or tagged templates, used
as method receivers, or stored through property targets are not assumed
unchanged afterward. Property stores conservatively retain reference escape
uncertainty rather than pretending to execute later writes. Aliases
Objects and arrays passed to calls, constructors, or tagged templates, or used
as method receivers, are not assumed unchanged after the invocation. Aliases
and shared children in spread and rest copies retain that uncertainty; copied
primitive slots and unrelated containing properties remain known. Object rest
excludes consumed keys, array rest excludes the consumed prefix for known
-4
View File
@@ -82,10 +82,6 @@ x86-64 ELF and native Windows PE targets analyzed on Linux x64 with Ghidra
See [NativeAOT recovery](ghidra-nativeaot.md) for build/configuration, recovered
metadata, derived-memory provenance, and the real verification lane.
REA's experimental Windows Ghidra boundary admits native x86 and x86-64 PE
applications and DLLs for static analysis. This does not establish Windows
NativeAOT metadata-recovery coverage.
For NativeAOT analysis, preserve the selected native artifact's path and digest
and report recovered type data as provider observations. Linking those types
to native function addresses requires verified
+6 -37
View File
@@ -36,13 +36,6 @@ after SDK conversion and check advertised validation and actual calls.
Individual model APIs can impose additional nesting limits; complete producer
captures can exceed ten structural levels.
Self-contained output schemas advertise a content-bound `$id`, including their
declared dialect. SDK validators can reuse compiled schemas across complete
catalog refreshes and equivalent tool outputs; changing the schema changes its
identity. Explicit schema IDs and relative external reference bases are
preserved. This keeps every tool and validation rule in discovery and does not
change the SDK's catalog invalidation or availability checks.
`compare_web_captures` accepts exactly one of two input shapes:
- Passive: `before` and `after` each contain `inspection`, the complete
@@ -193,27 +186,10 @@ the current client, so it is not a fresh-start recovery. On the same connection:
This close/reopen flow was exercised on one real Linux stdio connection after a
controlled startup timeout, followed by a successful overview and function
analysis. A separate Windows x64 stdio run with Ghidra 12.1.4, JDK 21 and Node
22.19.0 exercised the same connection and caller-selected native PE: the default
60-second client request timed out, close succeeded, and reopening before a
240-second request completed the cold overview in about 163 seconds. The next
overview took about 19 ms. The cold and warm overviews reported two procedures
and two segments; the input digest was unchanged and no owned runtime root or
observed process-family member remained after final close. This was a small
1,024-byte fixture, so the longer request setting remains an example, not a
guarantee for larger targets. No macOS cold-start recovery was exercised.
With a supplied progress token, the Windows run received an operation-start
notification before timeout and start/completion notifications for the
successful retry. It received no intermediate import or auto-analysis progress.
These operation markers do not measure analysis work completed or extend the
client's deadline. Each new provider session still imports the target into a
fresh temporary project.
REA cleans only resources it owns and never switches to another provider
automatically. A provider timeout, installation failure, or host permission
denial needs its own reported recovery; increasing a client deadline alone does
not fix those failures.
analysis. It is not a Windows/macOS coverage claim. REA cleans only resources it
owns and never switches to another provider automatically. A provider timeout,
installation failure, or host permission denial needs its own reported recovery;
increasing a client deadline alone does not fix those failures.
## Tool results
@@ -222,9 +198,7 @@ Custom clients upgrading from 6.1 should follow the
Evidence-producing tools return the complete canonical Evidence record in both
text and structured content. Read `structuredContent.normalized_result` for the
operation result and `structuredContent.evidence_id` for its identity.
`raw_result` holds a provider representation distinct from that result, such
as an upstream tool's original report, and is `null` when there is none. The same
operation result and `structuredContent.evidence_id` for its identity. The same
record is retained in the session bundle. `analysis_profile` is always present:
a concrete profile object or `null`; either value participates in semantic
identity. Records omitting this field are rejected. Pass the returned Evidence
@@ -261,12 +235,7 @@ Ordinary responses keep their existing complete result contract.
## Retained application Evidence inputs
`inspect_analysis_view` projects a caller-selected view of already completed
`inspect_binary_layout`, `analyze_javascript_application`, or `analyze_function`
Evidence. Native views select procedure, pseudocode, assembly, callers/callees,
references, and high-pcode facets without starting a provider. Native offset
and limit default to 0 and 64; pseudocode uses UTF-16 code units and never
splits surrogate pairs, while other pages count rows. Unavailable facts and
provider limitations remain explicit. Source is
`inspect_binary_layout` or `analyze_javascript_application` Evidence. Source is
an exact same-session retained reference or portable inline Evidence. Views are
a summary, a layout mitigations or linkage facet, one section/symbol/module, or
a stable page with a caller-selected positive `limit`. Module pages include
+1 -1
View File
@@ -379,7 +379,7 @@ the tool fails with a `debugger-attach-denied` reason.
Install a Ghidra 12.1.x release and the 64-bit full JDK it declares, then configure REA to
use them. Ghidra analysis supports Linux x64/arm64 and macOS x64/arm64; each installation requires
the matching native decompiler. Experimental Windows x64 P0 admits native
x86 and x86-64 PE applications and DLLs on local NTFS using bundled Job Object ownership,
x86-64 PE applications on local NTFS using bundled Job Object ownership,
protected runtime DACLs, and handle-based path admission. See [Windows Ghidra P0](windows-ghidra-p0.md) and
[issue #527](https://github.com/morluto/rea/issues/527).
On Linux and macOS, `annotate_native_function` atomically edits a function name
+2 -52
View File
@@ -110,14 +110,6 @@ while comparisons and trace assertions use the normalized text. Retaining the
original adds at most `limits.output_bytes` of text, because it comes from the
same admitted PTY chunks rather than a second unbounded stream.
Rendered frame `lines` hold the normalized visible rows with trailing U+0020
spaces removed. `columns` records the original terminal geometry. Normalization
can change text length, so restoring an earlier normalized row's padding requires
information beyond `columns`. `serialized_state` retains the normalized terminal
serialization. Captures commit this line format in their comparison contract, so
a capture whose lines were padded to full width does not compare as the same
contract.
Port normalization recognizes explicit `port`, `tcp_port`, `udp_port`, and
`listen` fields, URL authorities, IP endpoints, and `localhost` endpoints, with
ports from 0 through 65535. A port may end a sentence, as in `port: 8080.`;
@@ -167,11 +159,8 @@ Trace assertions use coverage for the sources they select, so an assertion
about complete raw terminal output need not fail because rendered snapshots
were omitted. Captures missing `truncation_details` are rejected on import;
preserve them as historical files and recapture for current comparisons.
`limits.file_bytes` is already the cumulative whole-file hashing budget,
independent of `limits.output_bytes`, which bounds retained terminal content.
Each filesystem checkpoint starts with a fresh `file_bytes` budget across all
selected roots; it is not a per-file allowance. File contents are read in
64 KiB chunks for hashing and are not embedded in capture Evidence.
These diagnostics use the existing scenario
budgets; they do not introduce a separate file-hashing budget.
When the host withholds an unrelated process’s ownership token, REA leaves that
process untouched and records its PID and reason in `cleanup.unverified_processes`
@@ -187,45 +176,6 @@ as well. A newly started unreadable process can prevent verified cleanup even
after the selected command exits. REA preserves this uncertainty and leaves
that process untouched.
## Hash a report independently of terminal output
Select the producer's report directory and give `file_bytes` enough room for
the complete files you need, including other files in those roots. For example,
save this scenario and run `rea capture-process ./scenario.json --json > capture.json`:
```json
{
"executable": "node",
"arguments": [
"-e",
"const fs=require('node:fs');fs.mkdirSync('reports',{recursive:true});fs.writeFileSync('reports/final.json',JSON.stringify({status:'complete',data:'X'.repeat(131072)}));console.log('x'.repeat(8192));"
],
"working_directory": ".",
"filesystem_observation_paths": ["./reports"],
"limits": { "output_bytes": 1024, "file_bytes": 262144 }
}
```
The report exceeds the terminal budget but fits the independent 256 KiB hash
budget. Its complete SHA-256 appears in `normalized_result.files_after` at
`root_0:final.json`; check `truncation_details.filesystem_after` for omissions
and actual bytes hashed. The verbose terminal output is truncated without
preventing the report digest. The same scenario object is the MCP
`capture_process_scenario` input. REA never returns a prefix hash as a full-file
digest: an insufficient remaining budget produces a null digest and an
explicit omission reason.
A digest binds the complete bytes observed at that checkpoint, not the
producer's notion of a finished report. A stable periodic or running report
can also have a complete digest. Choose the final file and verify the
producer's documented completion marker, exit outcome and settlement
observations before calling it final. `after_settlement` names a filesystem
checkpoint, not a guarantee that an application finished successfully or
that an unobserved writer will never change the file. REA checks opened-file
identity/state around hashing and reports observed changes or unavailable
files; cancellation stops the hash. If an external sealer produces a manifest,
keep its completion claims and digest separate from REA's checkpoint facts.
## Compare two captures
Compare saved capture Evidence with:
+2 -13
View File
@@ -17,7 +17,7 @@ additional permission flag or degraded mode is required.
- Node.js 22.x (>=22.19), 24.x (>=24.11), or 26+;
- an operator-installed Ghidra 12.1.x distribution (verified with 12.1.4);
- a 64-bit full JDK inside that installation's declared Java range (JDK 21 or newer, with no maximum, for current 12.1 releases);
- an explicit native, non-managed x86 or x86-64 PE application or DLL; and
- an explicit native, non-managed, non-DLL x86 or x86-64 PE application; and
- the 25 read-only Ghidra inventory, memory, and function-analysis operations.
Loaded memory reads and file offsets preserve Ghidra's observed source mappings.
@@ -38,13 +38,6 @@ REA selects its packaged bridge scripts by explicit path. Same-named files or
directories in the caller's working directory do not select a different
script or require users to clean that directory before analysis.
Native DLL analysis preserves the PE shared-library role in the analysis profile.
REA imports an immutable copy for static analysis; it does not load the library
into the analyst process or execute its entry routine. The source-owned DLL
fixtures have no entry routine or imports and expose two exported functions.
The same native snapshot, path-admission and cleanup requirements apply to EXEs
and DLLs.
## Configuration
```powershell
@@ -107,7 +100,7 @@ and artifact SHA-256 before loading the package-owned addon. Native failure
reasons remain distinct from a valid installation and an unsupported target.
PE header classification and SHA-256 come from the same open file. The Windows
provider admits native x86 and x86-64 PE applications and DLLs and rejects unsupported roles,
provider admits native x86 and x86-64 PE applications and rejects unsupported roles,
architectures, managed images, and malformed headers. The original selected
source coordinate is preserved for native admission.
@@ -166,11 +159,7 @@ artifact and manifest.
npm run verify:windows-native
npm run verify:ghidra:windows
npm run verify:ghidra:windows -- --x86
npm run verify:ghidra:windows -- --dll
npm run verify:ghidra:windows -- --x86 --dll
npm run verify:ghidra:windows:package -- --x86
npm run verify:ghidra:windows:package -- --dll
npm run verify:ghidra:windows:package -- --x86 --dll
npm run verify:ghidra:windows:package -- C:\fixtures\installed-rea C:\fixtures\sample.exe
```
+60 -95
View File
@@ -3,103 +3,68 @@ import { dirname, resolve } from "node:path";
import { fileURLToPath } from "node:url";
const root = resolve(dirname(fileURLToPath(import.meta.url)), "..");
// Source-owned static fixtures: no compiler, target loading or execution.
for (const architecture of ["x86_64", "x86"]) {
for (const dll of [false, true]) {
const x86 = architecture === "x86";
const output = resolve(
root,
"build",
"fixtures",
`rea-ghidra-windows${x86 ? "-x86" : ""}.${dll ? "dll" : "exe"}`,
);
const bytes = Buffer.alloc(1024);
const peOffset = 0x80;
const optionalHeader = peOffset + 24;
const optionalHeaderSize = x86 ? 0xe0 : 0xf0;
const sectionTable = optionalHeader + optionalHeaderSize;
const directories = optionalHeader + (x86 ? 96 : 112);
const x86 = architecture === "x86";
const output = resolve(
root,
"build",
"fixtures",
x86 ? "rea-ghidra-windows-x86.exe" : "rea-ghidra-windows.exe",
);
const bytes = Buffer.alloc(1024);
const peOffset = 0x80;
const optionalHeader = peOffset + 24;
const optionalHeaderSize = x86 ? 0xe0 : 0xf0;
const sectionTable = optionalHeader + optionalHeaderSize;
bytes.write("MZ", 0, "ascii");
bytes.writeUInt32LE(peOffset, 0x3c);
bytes.write("PE\0\0", peOffset, "binary");
bytes.writeUInt16LE(x86 ? 0x14c : 0x8664, peOffset + 4);
bytes.writeUInt16LE(1, peOffset + 6);
bytes.writeUInt16LE(optionalHeaderSize, peOffset + 20);
bytes.writeUInt16LE(
(x86 ? 0x102 : 0x22) | (dll ? 0x2000 : 0),
peOffset + 22,
);
bytes.write("MZ", 0, "ascii");
bytes.writeUInt32LE(peOffset, 0x3c);
bytes.write("PE\0\0", peOffset, "binary");
bytes.writeUInt16LE(x86 ? 0x14c : 0x8664, peOffset + 4);
bytes.writeUInt16LE(1, peOffset + 6);
bytes.writeUInt16LE(optionalHeaderSize, peOffset + 20);
bytes.writeUInt16LE(x86 ? 0x102 : 0x22, peOffset + 22);
bytes.writeUInt16LE(x86 ? 0x10b : 0x20b, optionalHeader);
bytes.writeUInt32LE(0x200, optionalHeader + 4);
bytes.writeUInt32LE(dll ? 0 : 0x1000, optionalHeader + 16);
bytes.writeUInt32LE(0x1000, optionalHeader + 20);
if (x86) bytes.writeUInt32LE(0x400000, optionalHeader + 28);
else bytes.writeBigUInt64LE(0x1_4000_0000n, optionalHeader + 24);
bytes.writeUInt32LE(0x1000, optionalHeader + 32);
bytes.writeUInt32LE(0x200, optionalHeader + 36);
bytes.writeUInt16LE(6, optionalHeader + 40);
bytes.writeUInt16LE(6, optionalHeader + 48);
bytes.writeUInt32LE(0x2000, optionalHeader + 56);
bytes.writeUInt32LE(0x200, optionalHeader + 60);
bytes.writeUInt16LE(3, optionalHeader + 68);
bytes.writeUInt16LE(x86 ? 0x8140 : 0x8160, optionalHeader + 70);
if (x86) {
bytes.writeUInt32LE(0x10_0000, optionalHeader + 72);
bytes.writeUInt32LE(0x1000, optionalHeader + 76);
bytes.writeUInt32LE(0x10_0000, optionalHeader + 80);
bytes.writeUInt32LE(0x1000, optionalHeader + 84);
bytes.writeUInt32LE(16, optionalHeader + 92);
} else {
bytes.writeBigUInt64LE(0x10_0000n, optionalHeader + 72);
bytes.writeBigUInt64LE(0x1000n, optionalHeader + 80);
bytes.writeBigUInt64LE(0x10_0000n, optionalHeader + 88);
bytes.writeBigUInt64LE(0x1000n, optionalHeader + 96);
bytes.writeUInt32LE(16, optionalHeader + 108);
}
bytes.write(".text", sectionTable, "ascii");
bytes.writeUInt32LE(dll ? 0x180 : 0x16, sectionTable + 8);
bytes.writeUInt32LE(0x1000, sectionTable + 12);
bytes.writeUInt32LE(0x200, sectionTable + 16);
bytes.writeUInt32LE(0x200, sectionTable + 20);
bytes.writeUInt32LE(0x6000_0020, sectionTable + 36);
// EXE entry returns zero. DLL caller adds one to the callee's 42.
bytes.set(
dll
? [0xe8, 0x0b, 0, 0, 0, 0x83, 0xc0, 1, 0xc3]
: [0xe8, 0x0b, 0, 0, 0, 0x31, 0xc0, 0xc3],
0x200,
);
bytes.set([0xb8, 0x2a, 0, 0, 0, 0xc3], 0x210);
if (dll) {
// Export both functions; an entry-point-free library is discovered by exports.
bytes.writeUInt32LE(0x1040, directories);
bytes.writeUInt32LE(0x130, directories + 4);
const exports = 0x240;
bytes.writeUInt32LE(0x1100, exports + 12);
bytes.writeUInt32LE(1, exports + 16);
bytes.writeUInt32LE(2, exports + 20);
bytes.writeUInt32LE(2, exports + 24);
bytes.writeUInt32LE(0x1080, exports + 28);
bytes.writeUInt32LE(0x1088, exports + 32);
bytes.writeUInt32LE(0x1090, exports + 36);
bytes.writeUInt32LE(0x1000, 0x280);
bytes.writeUInt32LE(0x1010, 0x284);
bytes.writeUInt32LE(0x1120, 0x288);
bytes.writeUInt32LE(0x1140, 0x28c);
bytes.writeUInt16LE(0, 0x290);
bytes.writeUInt16LE(1, 0x292);
bytes.write("rea-fixture.dll\0", 0x300, "binary");
bytes.write("fixture_caller\0", 0x320, "binary");
bytes.write("fixture_return42\0", 0x340, "binary");
}
await mkdir(dirname(output), { recursive: true });
await writeFile(output, bytes);
if (!x86 && !dll) process.stdout.write(`${output}\n`);
bytes.writeUInt16LE(x86 ? 0x10b : 0x20b, optionalHeader);
bytes.writeUInt32LE(0x200, optionalHeader + 4);
bytes.writeUInt32LE(0x1000, optionalHeader + 16);
bytes.writeUInt32LE(0x1000, optionalHeader + 20);
if (x86) bytes.writeUInt32LE(0x400000, optionalHeader + 28);
else bytes.writeBigUInt64LE(0x1_4000_0000n, optionalHeader + 24);
bytes.writeUInt32LE(0x1000, optionalHeader + 32);
bytes.writeUInt32LE(0x200, optionalHeader + 36);
bytes.writeUInt16LE(6, optionalHeader + 40);
bytes.writeUInt16LE(6, optionalHeader + 48);
bytes.writeUInt32LE(0x2000, optionalHeader + 56);
bytes.writeUInt32LE(0x200, optionalHeader + 60);
bytes.writeUInt16LE(3, optionalHeader + 68);
bytes.writeUInt16LE(x86 ? 0x8140 : 0x8160, optionalHeader + 70);
if (x86) {
bytes.writeUInt32LE(0x10_0000, optionalHeader + 72);
bytes.writeUInt32LE(0x1000, optionalHeader + 76);
bytes.writeUInt32LE(0x10_0000, optionalHeader + 80);
bytes.writeUInt32LE(0x1000, optionalHeader + 84);
bytes.writeUInt32LE(16, optionalHeader + 92);
} else {
bytes.writeBigUInt64LE(0x10_0000n, optionalHeader + 72);
bytes.writeBigUInt64LE(0x1000n, optionalHeader + 80);
bytes.writeBigUInt64LE(0x10_0000n, optionalHeader + 88);
bytes.writeBigUInt64LE(0x1000n, optionalHeader + 96);
bytes.writeUInt32LE(16, optionalHeader + 108);
}
bytes.write(".text", sectionTable, "ascii");
bytes.writeUInt32LE(0x16, sectionTable + 8);
bytes.writeUInt32LE(0x1000, sectionTable + 12);
bytes.writeUInt32LE(0x200, sectionTable + 16);
bytes.writeUInt32LE(0x200, sectionTable + 20);
bytes.writeUInt32LE(0x6000_0020, sectionTable + 36);
// entry calls the second function, returns zero; second returns 42.
bytes.set([0xe8, 0x0b, 0, 0, 0, 0x31, 0xc0, 0xc3], 0x200);
bytes.set([0xb8, 0x2a, 0, 0, 0, 0xc3], 0x210);
await mkdir(dirname(output), { recursive: true });
await writeFile(output, bytes);
if (!x86) process.stdout.write(`${output}\n`);
}
+1 -1
View File
@@ -1283,7 +1283,7 @@ async function inspect(
evidence.provider.version,
"pwntools@4.15.0;pyelftools@0.33;unicorn@2.1.2",
);
assert.equal(evidence.raw_result, null);
assert.deepEqual(evidence.normalized_result, evidence.raw_result);
assert.equal(evidence.normalized_result.diagnostics.truncated, false);
return evidence.normalized_result;
}
+21 -102
View File
@@ -15,7 +15,7 @@ import {
import { tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path";
import { promisify } from "node:util";
import { fileURLToPath, pathToFileURL } from "node:url";
import { pathToFileURL } from "node:url";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
@@ -24,50 +24,10 @@ if (process.platform !== "win32" || process.arch !== "x64")
throw new Error(
"Packaged Windows Ghidra acceptance requires a Windows x64 host.",
);
const workspace = await mkdtemp(join(tmpdir(), "rea-packaged-ghidra-"));
const exec = promisify(execFile);
const arguments_ = process.argv.slice(2);
const internalWorkspace = arguments_[0] === "--internal-workspace";
const fixtureArguments = internalWorkspace ? arguments_.slice(2) : arguments_;
const x86Fixture = fixtureArguments.includes("--x86");
const dllFixture = fixtureArguments.includes("--dll");
const [packageRootArgument, targetArgument, reportArgument] =
fixtureArguments.filter(
(argument) => argument !== "--x86" && argument !== "--dll",
);
// Windows locks the loaded package-owned addon until the verifier exits.
// Keep installation cleanup in a parent that never loads that addon.
if (!internalWorkspace) {
const ownedWorkspace = await mkdtemp(join(tmpdir(), "rea-packaged-ghidra-"));
let result;
try {
result = await exec(
process.execPath,
[
fileURLToPath(import.meta.url),
"--internal-workspace",
ownedWorkspace,
...arguments_,
],
{ timeout: 900_000, maxBuffer: 8 * 1024 * 1024 },
);
} catch (cause) {
if (typeof cause.stdout === "string") process.stdout.write(cause.stdout);
if (typeof cause.stderr === "string") process.stderr.write(cause.stderr);
throw cause;
} finally {
await rm(ownedWorkspace, { recursive: true, force: true });
}
const report = JSON.parse(result.stdout);
report.workspaceCleanup = true;
report.ok = true;
if (reportArgument !== undefined)
await writeFile(reportArgument, `${JSON.stringify(report)}\n`);
process.stdout.write(`${JSON.stringify(report)}\n`);
process.stderr.write(result.stderr);
process.exit(0);
}
assert.ok(arguments_[1] !== undefined, "Missing verifier workspace.");
const workspace = resolve(arguments_[1]);
const x86Fixture = process.argv[2] === "--x86";
const packageRootArgument = x86Fixture ? undefined : process.argv[2];
let packageRoot = resolve(packageRootArgument ?? ".");
// The default lane verifies the npm artifact in an isolated prefix. Explicit
// package roots support an already-installed artifact without a second install.
@@ -115,17 +75,17 @@ if (packageRootArgument === undefined) {
);
packageRoot = join(prefix, "node_modules", "rea-agents");
} catch (cause) {
throw new Error("Packaged Windows verifier installation failed.", {
cause,
});
await rm(workspace, { recursive: true, force: true });
throw cause;
}
}
const target = resolve(
targetArgument ??
`build/fixtures/rea-ghidra-windows${x86Fixture ? "-x86" : ""}.${dllFixture ? "dll" : "exe"}`,
process.argv[3] ??
(x86Fixture
? "build/fixtures/rea-ghidra-windows-x86.exe"
: "build/fixtures/rea-ghidra-windows.exe"),
);
let TOOL_CONTRACTS, native, token, sha256;
let parsedTarget;
try {
({ TOOL_CONTRACTS } = await import(
pathToFileURL(join(packageRoot, "dist/contracts/toolContracts.js"))
@@ -147,25 +107,9 @@ try {
);
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
sha256 = digest(await readFile(target));
const { parseBinaryTarget } = await import(
pathToFileURL(join(packageRoot, "dist/application/BinaryTargetResolver.js"))
);
parsedTarget = await parseBinaryTarget(target);
assert.equal(parsedTarget.ok, true);
assert.equal(parsedTarget.value.format, "pe");
assert.equal(parsedTarget.value.managed, false);
if (targetArgument === undefined) {
assert.equal(
parsedTarget.value.architecture,
x86Fixture ? "x86" : "x86_64",
);
assert.equal(
parsedTarget.value.executableRole,
dllFixture ? "shared-library" : "application",
);
}
} catch (cause) {
throw new Error("Packaged Windows verifier prerequisites failed.", { cause });
await rm(workspace, { recursive: true, force: true });
throw cause;
}
const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
const runtimeParent = join(workspace, "runtime with spaces");
@@ -201,8 +145,6 @@ const report = {
architecture: "x64",
nonAdmin: true,
targetSha256: sha256,
targetArchitecture: parsedTarget.value.architecture,
targetRole: parsedTarget.value.executableRole,
cli: {},
mcpOperations: [],
nativeControls: "available",
@@ -266,8 +208,8 @@ try {
});
const call = async (name, arguments_) => {
report.lastOperation = name;
if (reportArgument !== undefined)
await writeFile(reportArgument, `${JSON.stringify(report)}\n`);
if (process.argv[4] !== undefined)
await writeFile(process.argv[4], `${JSON.stringify(report)}\n`);
const contract = TOOL_CONTRACTS.find((value) => value.name === name);
assert.ok(contract !== undefined, `Missing canonical contract: ${name}`);
const argumentsParsed = contract.inputSchema.parse(arguments_);
@@ -288,25 +230,6 @@ try {
await call("list_names", {});
const procedures = await call("list_procedures", {});
assert.ok(procedures.length > 0);
if (targetArgument === undefined) {
const calleeAddress = x86Fixture ? "0x401010" : "0x140001010";
const calleeCode = await call("procedure_pseudo_code", {
procedure: calleeAddress,
});
assert.match(calleeCode, /\b(?:42|0x2a)\b/iu);
if (dllFixture) {
const callerAddress = x86Fixture ? "0x401000" : "0x140001000";
assert.ok(
(
await call("procedure_callees", { procedure: callerAddress })
).includes(calleeAddress),
);
assert.match(
await call("procedure_pseudo_code", { procedure: callerAddress }),
/\+\s*1\b/u,
);
}
}
assert.ok((await call("list_segments", {})).length > 0);
await call("list_strings", {});
const procedure =
@@ -373,12 +296,6 @@ try {
"Packaged verification omitted an admitted operation.",
);
const session = await call("binary_session", {});
assert.equal(
session.analysis_provider_binding?.analysis_profile?.parameters
?.executable_role,
parsedTarget.value.executableRole,
"MCP profile must preserve the selected PE role.",
);
assert.equal(
session.analysis_provider_binding?.selection_source,
"auto-single-candidate",
@@ -409,9 +326,9 @@ try {
assert.deepEqual(await readdir(join(callerDirectory, name)), []);
report.callerScriptCollisionsPreserved = true;
report.runtimeCleanup = true;
// Only the parent can attest success after removing the installation.
if (reportArgument !== undefined)
await writeFile(reportArgument, `${JSON.stringify(report)}\n`);
report.ok = true;
if (process.argv[4] !== undefined)
await writeFile(process.argv[4], `${JSON.stringify(report)}\n`);
process.stdout.write(`${JSON.stringify(report)}\n`);
} catch (cause) {
report.failure = {
@@ -421,7 +338,9 @@ try {
.replaceAll(target, "<target>")
.replaceAll(workspace, "<workspace>"),
};
if (reportArgument !== undefined)
await writeFile(reportArgument, `${JSON.stringify(report)}\n`);
if (process.argv[4] !== undefined)
await writeFile(process.argv[4], `${JSON.stringify(report)}\n`);
throw cause;
} finally {
await rm(workspace, { recursive: true, force: true });
}
+6 -21
View File
@@ -62,21 +62,21 @@ if (
);
const root = resolve(fileURLToPath(new URL("..", import.meta.url)));
const architecture = process.argv.includes("--x86") ? "x86" : "x86_64";
const dllFixture = process.argv.includes("--dll");
const role = dllFixture ? "shared-library" : "application";
const architecture = process.argv[2] === "--x86" ? "x86" : "x86_64";
const targetPath = resolve(
root,
"build",
"fixtures",
`rea-ghidra-windows${architecture === "x86" ? "-x86" : ""}.${dllFixture ? "dll" : "exe"}`,
architecture === "x86"
? "rea-ghidra-windows-x86.exe"
: "rea-ghidra-windows.exe",
);
const target = await parseBinaryTarget(targetPath);
if (
!target.ok ||
target.value.format !== "pe" ||
target.value.architecture !== architecture ||
target.value.executableRole !== role ||
target.value.executableRole !== "application" ||
target.value.managed !== false
)
throw new Error(
@@ -89,8 +89,6 @@ const profile = await resolveGhidraAnalysisProfile(
);
if (!profile.ok || profile.value.profile === null)
throw new Error("Windows Ghidra profile could not be committed");
if (profile.value.profile.parameters.executable_role !== role)
throw new Error("Windows Ghidra profile lost the observed PE role");
const client = new GhidraClient({
platform: installation.platform,
@@ -164,19 +162,6 @@ try {
throw new Error(
"Windows fixture decompilation lost its known return value",
);
if (dllFixture) {
const callerAddress = architecture === "x86" ? "0x401000" : "0x140001000";
const callees = await functionOperation("procedure_callees", {
procedure: callerAddress,
});
if (!callees.includes(callee.address))
throw new Error("DLL export caller lost its controlled call reference");
const callerCode = await functionOperation("procedure_pseudo_code", {
procedure: callerAddress,
});
if (typeof callerCode !== "string" || !/\+\s*1\b/u.test(callerCode))
throw new Error("DLL export caller lost its addition of one");
}
await inventory("inspect_native_load_image", {});
const memory = await inventory("read_bytes", {
@@ -269,7 +254,7 @@ try {
limitations: [
"approved-non-sensitive-fixtures-only",
"windows-x64-local-ntfs-only",
"native-x86-and-x86-64-pe-applications-and-dlls-only",
"native-x86-and-x86-64-pe-applications-only",
"no-gui-or-mutation-authority",
],
};
-3
View File
@@ -17,7 +17,6 @@ import {
import { verifyDyldEnvironment } from "./dyld-environment.mjs";
import { verifyDylibResolution } from "./macos-dylib-e2e.mjs";
import { verifyMachOOccurrences } from "./macho-occurrences.mjs";
import { verifyNativeProvenance } from "./native-provenance.mjs";
const exec = promisify(execFile);
@@ -70,7 +69,6 @@ const EXPECTED_BUNDLES = [
let report;
try {
const { app, executable } = await buildMacosBundleFixture(root);
const nativeProvenance = await verifyNativeProvenance(app, executable);
const machoOccurrences = await verifyMachOOccurrences(app, executable);
const zip = join(root, "MacFixture.zip");
await exec("/usr/bin/ditto", ["-c", "-k", "--keepParent", app, zip]);
@@ -168,7 +166,6 @@ try {
containers: containers.map(({ format }) => format),
bundles: EXPECTED_BUNDLES.length,
dmg_detached: true,
native_provenance: nativeProvenance,
macho_occurrences: machoOccurrences,
dylib_resolution: dylibs,
dyld_environment: dyldEnvironment,
@@ -1,71 +0,0 @@
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { toolContract } from "../../../dist/contracts/toolContracts.js";
import {
artifactCliEvidence,
withArtifactMcp,
} from "../../lib/artifact-e2e.mjs";
/** Inspect the compiled bundle with real native utilities through both adapters. */
export async function verifyNativeProvenance(app, executable) {
const original = await readFile(executable);
const sha256 = createHash("sha256").update(original).digest("hex");
const symbols = ["$s4main3FooV", "plain_symbol"];
const operations = [
["inspect_macho", []],
["inspect_signature", []],
["inspect_plist", []],
["list_architectures", []],
["demangle_swift", symbols],
];
const observations = [];
await withArtifactMcp(app, async (client) => {
const definitions = new Map(
(await client.listTools()).tools.map((tool) => [tool.name, tool]),
);
for (const [operation, arguments_] of operations) {
const cli = toolContract(operation).outputSchema.parse(
await artifactCliEvidence(
operation.replaceAll("_", "-"),
app,
arguments_,
),
);
const response = await client.callTool(
{
name: operation,
arguments: operation === "demangle_swift" ? { symbols } : {},
},
{ toolDefinition: definitions.get(operation) },
);
assert.notEqual(response.isError, true, JSON.stringify(response));
const mcp = toolContract(operation).outputSchema.parse(
response.structuredContent,
);
assert.equal(cli.raw_result, null);
assert.equal(mcp.raw_result, null);
assert.ok(cli.normalized_result.provenance.length > 0);
assert.deepEqual(mcp.normalized_result, cli.normalized_result);
assert.deepEqual(
JSON.parse(response.content.find((item) => item.type === "text").text),
response.structuredContent,
);
await client.ping();
observations.push({
operation,
commands: cli.normalized_result.provenance.length,
cli_bytes: Buffer.byteLength(JSON.stringify(cli)),
mcp_bytes: Buffer.byteLength(JSON.stringify(response)),
});
}
});
assert.deepEqual(await readFile(executable), original);
return {
real_native_utilities: true,
target_sha256: sha256,
bytes_unchanged: true,
observations,
};
}
+3 -3
View File
@@ -54,15 +54,15 @@ try {
data: { ...evidence, normalized_result: { payload: [marker] } },
meta: metadata,
};
// The destination is a redirected file, so both formats are compact.
const skeleton = JSON.stringify(template);
const skeleton = JSON.stringify(template, null, format === "json" ? 2 : 0);
const encodedMarker = JSON.stringify(marker);
const markerIndex = skeleton.indexOf(encodedMarker);
assert.ok(markerIndex >= 0);
assert.equal(markerIndex, skeleton.lastIndexOf(encodedMarker));
const prefix = skeleton.slice(0, markerIndex);
const suffix = `${skeleton.slice(markerIndex + encodedMarker.length)}\n`;
const separator = ",";
const indentation = prefix.slice(prefix.lastIndexOf("\n") + 1);
const separator = format === "json" ? `,\n${indentation}` : ",";
const expectedHash = createHash("sha256");
expectedHash.update(prefix);
for (let index = 0; index < count; index += 1) {
+1 -1
View File
@@ -80,7 +80,7 @@ export async function connectRecordedCrash({ entrypoint, environment }) {
evidence.confidence,
debuggerContext ? "derived" : "observed",
);
assert.equal(evidence.raw_result, null);
assert.deepEqual(evidence.raw_result, report);
assert.equal(evidence.subject.local_path, path);
assert.equal(report.artifact.path, path);
assert.equal(
+6 -13
View File
@@ -2,7 +2,7 @@
name: reverse-engineer-anything
description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.
metadata:
version: "35"
version: "34"
---
# REA
@@ -62,20 +62,13 @@ with a version-pinned `rea-agents` package or continue through the CLI.
A concrete CLI fallback for an operator-supplied JavaScript tree or ASAR is:
```bash
npx -y rea-agents@latest analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
npx -y rea-agents@latest inspect-analysis-view app-view.json
npx -y rea-agents@latest analyze-javascript-application /absolute/path/to/app --json
```
No MCP registration or native engine is required. The CLI returns the complete
Evidence record directly, and a real application can produce hundreds of
megabytes, so save it to a file rather than reading it into context. Start from
the summary view, then request a module page
(`{"kind": "page", "collection": "modules", "offset": 0, "limit": 32}`) or one
module the same way, or pass the saved Evidence inline to
`trace-application-feature`. Read graph, limitations, and unknowns with the
same care as an MCP result. This fallback does not establish that MCP is
configured. Native CLI tasks still require their selected engine.
No MCP registration or native engine is required. Read the returned Evidence,
graph, limitations, and unknowns with the same care as an MCP result; the CLI
returns the Evidence record directly. This fallback does not establish that
MCP is configured. Native CLI tasks still require their selected engine.
## Route the target first
@@ -34,14 +34,6 @@ cross-references. Addresses and recovered pseudocode are analysis observations,
not original source. Provider unavailability and unsupported metadata remain
unknown rather than false.
For a completed `analyze_function` dossier, use `inspect_analysis_view` with
`view: {kind: "native", facet: "value_flow_operations", offset: 0, limit: 32}`
and the exact retained Evidence ID to read selected facts without reanalysis.
Native facets also include procedure, pseudocode, assembly, callers/callees,
references, and value-flow summaries. Pseudocode offsets and limits count UTF-16
code units; use the returned `next_offset` for continuation. Provider limitations
and unavailable facts remain explicit; the complete parent Evidence is retained.
To see which functions or Objective-C methods a Mach-O actually calls in one
run, use `observe_native_calls` with explicit breakpoints and a bounded
`duration_ms`/`max_events`. It launches a new process under LLDB, so the target
-22
View File
@@ -1,22 +0,0 @@
import { expect, it } from "vitest";
import { createAnalysisExecution } from "./AnalysisProvider.js";
const PROVIDER = { id: "fixture", name: "Fixture", version: "1" };
it("records no raw result unless the provider supplies a distinct one", () => {
const result = { entries: [{ value: "normalized" }] };
expect(createAnalysisExecution(result, PROVIDER)).toMatchObject({
result,
rawResult: null,
});
expect(
createAnalysisExecution(result, PROVIDER, {
rawResult: { stdout: "raw" },
}).rawResult,
).toEqual({ stdout: "raw" });
expect(
createAnalysisExecution(result, PROVIDER, { rawResult: null }).rawResult,
).toBeNull();
});
+2 -8
View File
@@ -41,13 +41,7 @@ export interface AnalysisExecution {
readonly subject: EvidenceSubjectTarget | null;
}
/**
* Build a validated atomic provider observation at an adapter boundary.
*
* Pass `rawResult` only for a provider representation distinct from
* `result`. Omitting it records `null` rather than a second complete copy of
* the normalized result in every Evidence record and tool response.
*/
/** Build a validated atomic provider observation at an adapter boundary. */
export const createAnalysisExecution = (
result: unknown,
provider: ProviderIdentity,
@@ -62,7 +56,7 @@ export const createAnalysisExecution = (
result: jsonValueSchema.parse(result),
rawResult:
options.rawResult === undefined
? null
? jsonValueSchema.parse(result)
: jsonValueSchema.parse(options.rawResult),
provider,
...(options.analysisProfile === undefined
+2 -2
View File
@@ -2,6 +2,8 @@ import fs from "node:fs";
import { lstat, readFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { resolveRef } from "isomorphic-git";
export type ReferenceSourceVcsInfo =
| {
readonly kind: "git";
@@ -40,8 +42,6 @@ export const readReferenceSourceVcs = async (
};
const resolveSourceHead = async (root: string): Promise<string> => {
// isomorphic-git is loaded on first use so CLI and MCP startup skip it.
const { resolveRef } = await import("isomorphic-git");
const dotgit = join(root, ".git");
if (!(await lstat(dotgit)).isFile())
return resolveRef({ fs, dir: root, ref: "HEAD" });
@@ -2,8 +2,6 @@ import { expect, it } from "vitest";
import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../../contracts/javascript/javascriptRuntimeReconciliationExample.js";
import { createEvidence, parseEvidence } from "../../domain/evidence.js";
import { functionDossierSchema } from "../../domain/hopperValues.js";
import { ghidraFunctionDossier } from "../../domain/ghidraValues.fixture.js";
import {
analysisViewJavaScriptAnalysisWithSource,
analysisViewJavaScriptEvidence,
@@ -11,96 +9,6 @@ import {
} from "../../../tests/fixtures/analysisView.js";
import { inspectAnalysisView } from "./AnalysisViewService.js";
it.each(["/fixtures/unknown.exe", ""])(
"preserves recorded empty native identity fields: path=%j",
(path) => {
const original = functionDossierSchema.parse(ghidraFunctionDossier());
const parent = createEvidence(
{ path, format: "pe", sha256: "c".repeat(64) },
{ id: "ghidra", name: "Ghidra", version: "12.1.4" },
{
operation: "analyze_function",
parameters: {},
result: {
...original,
procedure: {
...original.procedure,
address: "",
body: { available: false, reason: "Not recorded" },
},
},
},
);
const result = inspectAnalysisView({
source: { kind: "inline", evidence: parent },
view: {
kind: "native",
facet: "procedure",
offset: 0,
limit: 64,
},
});
if (!result.ok) throw result.error;
expect(result.value.normalized_result).toMatchObject({
procedure_address: "",
artifact: { path },
item: { address: "" },
});
expect(result.value.locations).toEqual(
path.length > 0 ? [{ kind: "artifact-path", path }] : [],
);
},
);
it("projects a small native view from an authenticated >10 MiB retained dossier", () => {
const original = functionDossierSchema.parse(ghidraFunctionDossier());
const parent = createEvidence(
{ path: "/fixtures/large.exe", format: "pe", sha256: "c".repeat(64) },
{ id: "ghidra", name: "Ghidra", version: "12.1.4" },
{
operation: "analyze_function",
parameters: { address: "0x401000" },
result: { ...original, pseudocode: "X".repeat(11 * 1024 * 1024) },
limitations: ["Synthetic fixture"],
},
);
const input = {
source: {
kind: "retained-evidence" as const,
evidence_id: parent.evidence_id,
},
view: {
kind: "native" as const,
facet: "pseudocode" as const,
offset: 0,
limit: 128,
},
};
const view = inspectAnalysisView(input, (id) =>
id === parent.evidence_id ? parent : undefined,
);
if (!view.ok) throw view.error;
expect(view.value.evidence_links).toEqual([parent.evidence_id]);
expect(view.value.normalized_result).toMatchObject({
kind: "native",
parent_operation: "analyze_function",
parent_evidence_id: parent.evidence_id,
procedure_address: "0x401000",
artifact: { path: "/fixtures/large.exe", sha256: "c".repeat(64) },
coverage: { total: 11 * 1024 * 1024, examined: 128, next_offset: 128 },
});
expect(view.value.locations).toEqual([
{ kind: "artifact-path", path: "/fixtures/large.exe" },
{ kind: "address", address: "0x401000" },
]);
expect(JSON.stringify(view.value).length).toBeLessThan(12000);
expect(JSON.stringify(parent).length).toBeGreaterThan(10 * 1024 * 1024);
expect(inspectAnalysisView(input, () => undefined)).toMatchObject({
ok: false,
error: { _tag: "EvidenceIntegrityError" },
});
});
it("projects inline layout Evidence and retains a derived view record", () => {
const parent = analysisViewLayoutEvidence();
const result = inspectAnalysisView({
@@ -53,13 +53,6 @@ export const inspectAnalysisViewValidated = (
operation: parent.operation,
normalizedResult: parent.normalized_result,
limitations: parent.limitations,
artifact:
parent.subject === null
? null
: {
path: parent.subject.local_path,
sha256: parent.subject.digest.sha256,
},
},
input.view,
);
@@ -133,16 +126,7 @@ const createAnalysisViewEvidence = (
confidence: "derived",
authority: parent.authority,
limitations: result.limitations,
locations: [
...(result.artifact.path.length > 0
? [{ kind: "artifact-path" as const, path: result.artifact.path }]
: []),
...(result.kind === "native" &&
result.procedure_address !== null &&
result.procedure_address.length > 0
? [{ kind: "address" as const, address: result.procedure_address }]
: []),
],
locations: [{ kind: "artifact-path", path: result.artifact.path }],
evidenceLinks: [parent.evidence_id],
});
};
@@ -5,7 +5,6 @@ import {
BINARY_LAYOUT_TEST_PROVIDER,
} from "../../../tests/fixtures/binaryDiagnostics/layout.js";
import { AnalysisCapabilityUnavailableError } from "../../domain/analysisErrorCore.js";
import { parseEvidence } from "../../domain/evidence.js";
import { err, ok } from "../../domain/result.js";
it.each([
@@ -29,19 +28,6 @@ it.each([
},
);
it("records the decoded layout once in its Evidence", async () => {
const layout = binaryLayoutFixture();
const service = new BinaryLayoutService({
identity: BINARY_LAYOUT_TEST_PROVIDER,
inspect: () => Promise.resolve(ok(layout)),
});
const result = await service.inspect({ path: layout.artifact.path });
if (!result.ok) throw result.error;
const evidence = parseEvidence(result.value);
expect(evidence.normalized_result).toEqual(layout);
expect(evidence.raw_result).toBeNull();
});
it("preserves a provider's actionable unsupported reason", async () => {
const failure = new AnalysisCapabilityUnavailableError(
"fixture-layout",
@@ -78,8 +78,7 @@ export class BinaryLayoutService {
operation: OPERATION,
parameters: input.data,
result: value,
// The decoded report is the only representation; do not repeat it.
rawResult: null,
rawResult: value,
confidence: "observed",
limitations: value.limitations,
locations: [{ kind: "artifact-path", path: value.artifact.path }],
@@ -24,7 +24,6 @@ it("labels an envelope containing debugger-derived maps as derived", async () =>
const evidence = parseEvidence(result.value);
expect(evidence.confidence).toBe("derived");
expect(evidence.normalized_result).toEqual(fixture);
expect(evidence.raw_result).toBeNull();
});
it.each([
@@ -83,8 +83,7 @@ export class RecordedCrashService {
operation: OPERATION,
parameters: input.data,
result: value,
// The decoded report is the only representation; do not repeat it.
rawResult: null,
rawResult: value,
confidence:
value.debugger.status === "available" ? "derived" : "observed",
limitations: value.limitations,
@@ -1,6 +1,5 @@
import { createHash } from "node:crypto";
import { compactStringIdentityKey } from "../../domain/compactStringIdentity.js";
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
import type { JavaScriptStaticAnalysis } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
@@ -163,15 +162,16 @@ const addEndpoint = (
const source =
sourceNodeFor(context, input.file.path, input.value.module_key) ??
input.asset;
const key = compactStringIdentityKey(input.value.value);
const label = `${input.value.kind} endpoint`;
const node = context.accumulator.addNode({
kind: "endpoint",
identity: artifactLocalIdentity(input.file.sha256, input.value.kind, key),
identity: artifactLocalIdentity(
input.file.sha256,
input.value.kind,
input.value.value,
),
observations: [
{
label:
input.value.value.length <= label.length ? input.value.value : label,
label: input.value.value,
properties: {
endpoint_kind: input.value.kind,
value: input.value.value,
@@ -1,4 +1,3 @@
import { compactStringIdentityKey } from "../../domain/compactStringIdentity.js";
import { createJavaScriptSemanticGraphUnknown } from "../../domain/javascript/javascriptSemanticGraph.js";
import type { JavaScriptSemanticGraphNode } from "../../domain/javascript/javascriptSemanticGraphSchemas.js";
import type {
@@ -22,7 +21,7 @@ export const projectSemanticEvents = (
const eventKey =
operation.eventName === null
? operation.eventId
: `${operation.emitterKey}\u0000${compactStringIdentityKey(operation.eventName)}`;
: `${operation.emitterKey}\u0000${operation.eventName}`;
const event =
eventNodes.get(eventKey) ?? addEventNode(context, operation, eventKey);
if (event === null) continue;
@@ -60,11 +59,7 @@ const addEventNode = (
kind: "event",
roleKey: `event:${eventKey}`,
location: operation.eventName === null ? operation.location : null,
label:
operation.eventName === null ||
operation.eventName.length <= "event".length
? operation.eventName
: "event",
label: operation.eventName,
functionNodeId: null,
properties: {
emitter_key: operation.emitterKey,
@@ -83,9 +78,7 @@ const addListenerNode = (
label:
operation.eventName === null
? operation.method
: operation.eventName.length <= "listener".length
? `${operation.method}:${operation.eventName}`
: `${operation.method}:listener`,
: `${operation.method}:${operation.eventName}`,
functionNodeId:
operation.ownerCallableId === null
? null
@@ -175,21 +175,7 @@ export const createJavaScriptSemanticGraphProjection =
): Value => {
bindSemanticGraphApplicationNodes(state, applicationGraph);
if (state.roots.size === 0) addFallbackRoot(rootArtifactSha256, state);
const relations = [...state.relations.values()];
const unknowns = [...state.unknowns.values()];
type Family = (typeof JAVASCRIPT_SEMANTIC_RELATION_FAMILIES)[number];
const retainedByFamily = new Map<Family, number>();
const unknownIdsByFamily = new Map<Family, string[]>();
for (const relation of relations) {
const family = JAVASCRIPT_SEMANTIC_RELATION_FAMILY[relation.relation];
retainedByFamily.set(family, (retainedByFamily.get(family) ?? 0) + 1);
}
for (const unknown of unknowns) {
const identifiers = unknownIdsByFamily.get(unknown.family);
if (identifiers === undefined)
unknownIdsByFamily.set(unknown.family, [unknown.unknown_id]);
else identifiers.push(unknown.unknown_id);
}
const graph = factory({
schema: "JavaScriptSemanticRelationGraph",
root_artifact_sha256: rootArtifactSha256,
@@ -197,7 +183,7 @@ export const createJavaScriptSemanticGraphProjection =
root_node_ids: [...state.roots],
evidence_contexts: state.evidenceContexts.contexts,
nodes: [...state.nodes.values()],
relations,
relations: [...state.relations.values()],
fingerprints,
unknowns,
coverage: {
@@ -218,9 +204,15 @@ export const createJavaScriptSemanticGraphProjection =
families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({
family,
status: semanticFamilyStatus(family),
retained_relations: retainedByFamily.get(family) ?? 0,
retained_relations: [...state.relations.values()].filter(
(relation) =>
JAVASCRIPT_SEMANTIC_RELATION_FAMILY[relation.relation] ===
family,
).length,
omitted_relations: truncatedFiles > 0 ? null : 0,
unknown_ids: unknownIdsByFamily.get(family) ?? [],
unknown_ids: unknowns
.filter((unknown) => unknown.family === family)
.map(({ unknown_id: identifier }) => identifier),
})),
},
limitations: [
@@ -137,11 +137,7 @@ const addRequestNode = (
kind: operation.kind === "request" ? "request" : "response",
roleKey: operation.requestId,
location: operation.location,
label:
operation.endpoint !== null &&
operation.endpoint.length <= operation.method.length
? operation.endpoint
: operation.method,
label: operation.endpoint ?? operation.method,
functionNodeId:
operation.ownerCallableId === null
? null
@@ -2,7 +2,6 @@ import {
semanticContainer,
semanticPropertyPointer,
} from "../../domain/javascript/javascriptSemanticSlots.js";
import { digestCanonicalValue } from "../../domain/canonicalDigest.js";
import type { JavaScriptSemanticBinding } from "../../domain/javascript/javascriptSemanticIr.js";
import type {
JavaScriptSemanticProperty,
@@ -154,27 +153,16 @@ const addLiteralNode = (
binding: JavaScriptSemanticBinding,
value: string | number | boolean | null,
role: string,
) => {
const encodedValue = JSON.stringify(value);
const valueDigest = `value-sha256:${digestCanonicalValue(value, "Semantic literal")}`;
const valueKey =
encodedValue.length <= valueDigest.length ? encodedValue : valueDigest;
const stringLabel = "string literal";
return retainSemanticGraphNode(context.state, context.file, {
) =>
retainSemanticGraphNode(context.state, context.file, {
kind: "literal",
// Identity needs an exact value commitment, not another payload copy.
// Keep the complete literal in properties for queries and Evidence export.
roleKey: `literal:${binding.bindingId}:${role}:${valueKey}`,
roleKey: `literal:${binding.bindingId}:${role}:${JSON.stringify(value)}`,
location: binding.definitions[0]?.location ?? null,
label:
typeof value === "string" && encodedValue.length > stringLabel.length
? stringLabel
: encodedValue,
label: JSON.stringify(value),
functionNodeId:
context.bindingNodes.get(binding.bindingId)?.function_node_id ?? null,
properties: { value },
});
};
/** Create one canonical slot for a root binding and exact static property path. */
export const semanticPropertySlot = (
@@ -25,7 +25,6 @@ export class LocalWebScriptArtifacts implements WebScriptArtifactPort {
async load(input: SelectedWebScriptInput, options?: ExecutionOptions) {
let field: readonly (string | number)[] = ["manifest_path"];
let targetPath = input.manifest_path;
let callerSelected = true;
try {
const manifestBytes = await readWebArtifactJson(
input.manifest_path,
@@ -59,7 +58,6 @@ export class LocalWebScriptArtifacts implements WebScriptArtifactPort {
const root = resolve(dirname(input.manifest_path));
const sourcePath = join(root, "files", portable);
targetPath = sourcePath;
callerSelected = false;
await assertContained(root, sourcePath);
const sourceBytes = await readStableArtifact(
sourcePath,
@@ -101,7 +99,7 @@ export class LocalWebScriptArtifacts implements WebScriptArtifactPort {
return err(
webArtifactReadError(
cause,
{ operation: this.operation, field, targetPath, callerSelected },
{ operation: this.operation, field, targetPath },
options?.signal,
),
);
-37
View File
@@ -3,7 +3,6 @@ import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import { readStableArtifact } from "../../artifacts/readStableArtifact.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import {
AnalysisAccessDeniedError,
AnalysisCancelledError,
AnalysisInputError,
} from "../../domain/analysisErrorCore.js";
@@ -16,8 +15,6 @@ export interface WebArtifactReadContext {
readonly operation: "trace_web_module_imports" | "trace_web_source_location";
readonly field: readonly (string | number)[];
readonly targetPath: string;
/** Whether the caller named this file, rather than a manifest deriving it. */
readonly callerSelected: boolean;
}
/** A malformed producer document, distinct from host I/O failures. */
@@ -57,10 +54,6 @@ export const webArtifactReadError = (
): AnalysisError => {
if (signal?.aborted === true)
return new AnalysisCancelledError(context.operation);
const selection = context.callerSelected
? selectedFileError(cause, context)
: undefined;
if (selection !== undefined) return selection;
if (cause instanceof ArtifactReaderFailure)
return new ArtifactOperationError(
context.operation,
@@ -98,33 +91,3 @@ export const webArtifactReadError = (
},
});
};
/**
* Report a missing, non-regular or unreadable caller-selected file as the
* caller's selection or a host permission denial, not an artifact fault.
*/
const selectedFileError = (
cause: unknown,
context: WebArtifactReadContext,
): AnalysisError | undefined => {
const selected = (message: string) =>
new AnalysisInputError(context.operation, { cause }, [
{ path: context.field, reason: "invalid_value", message },
]);
if (cause instanceof ArtifactReaderFailure)
return cause.reason === "path" ? selected(cause.message) : undefined;
const code =
cause instanceof Error && "code" in cause ? String(cause.code) : "";
if (code === "EACCES" || code === "EPERM")
return new AnalysisAccessDeniedError(
context.operation,
context.targetPath,
code,
{ cause },
);
if (code === "ENOENT" || code === "ENOTDIR")
return selected(
`Selected file could not be read (${code}): ${context.targetPath}`,
);
return undefined;
};
@@ -48,7 +48,6 @@ export class LocalWebSourceLocationArtifacts implements WebSourceLocationArtifac
operation: "trace_web_source_location",
field: ["source_map", "path"],
targetPath: input.source_map.path,
callerSelected: true,
},
options?.signal,
),
@@ -52,7 +52,6 @@ export class LocalWebModuleArtifacts implements WebModuleArtifactPort {
operation: "trace_web_module_imports",
field: ["import_map", "path"],
targetPath: selected.path,
callerSelected: true,
},
options?.signal,
),
+1 -1
View File
@@ -18,7 +18,7 @@ export const registerAnalysisViewCommands = (
): void => {
cli.command(CLI_COMMANDS.inspectAnalysisView, {
description:
"Project a selected view of completed layout, JavaScript application or native function Evidence JSON",
"Project a selected view of completed layout or JavaScript application Evidence JSON",
args: z.object({
inputJson: z.string().describe("Inline workflow JSON or JSON file path"),
}),
+9
View File
@@ -123,6 +123,7 @@ const registerAssetCatalogCommand = (
.default(1_000)
.describe("Maximum records to return"),
}),
alias: {},
run: ({ args, options }) =>
logCliCommand(logger, CLI_COMMANDS.inspectAssetCatalog, () =>
runProviderAnalysis(
@@ -169,6 +170,11 @@ const registerInterfaceBuilderCommand = (
.default(40_000)
.describe("Maximum decoded connections"),
}),
alias: {
maxDocuments: "max-documents",
maxObjects: "max-objects",
maxConnections: "max-connections",
},
run: ({ args, options }) =>
logCliCommand(logger, "decode-interface-builder", () =>
runProviderAnalysis(
@@ -227,6 +233,9 @@ const registerInspectionCommand = (
.default("fail")
.describe("Behavior when declared artifact integrity does not match"),
}),
alias: {
integrityPolicy: "integrity-policy",
},
run: ({ args, options }) =>
logCliCommand(logger, "inspect-artifact", () =>
runProviderAnalysis(
+36 -78
View File
@@ -15,14 +15,13 @@ import {
type AnalysisInputIssue,
} from "../domain/analysisErrorCore.js";
import { browserCaptureComparisonInputSchema } from "../domain/browserCaptureComparison.js";
import { projectInputIssues } from "../domain/inputIssueProjection.js";
import { discoverWebMcpToolsInputSchema } from "../domain/webMcpDiscovery.js";
import {
captureWebScreenshotInputSchema,
compareWebScreenshotsInputSchema,
} from "../domain/webScreenshot.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { parseCliJsonInput } from "../cliJsonInput.js";
import { safeParseJson } from "../domain/safeJson.js";
import type { Logger } from "pino";
import { CLI_COMMANDS } from "../cliCommandNames.js";
@@ -80,39 +79,46 @@ const registerCaptureDiff = (
description:
"Compare two normalized passive captures or recorded browser scenarios",
args: z.object({
beforeJson: z
.string()
.describe("Earlier normalized web capture JSON or JSON file path"),
afterJson: z
.string()
.describe("Later normalized web capture JSON or JSON file path"),
beforeJson: z.string().describe("Earlier normalized web capture JSON"),
afterJson: z.string().describe("Later normalized web capture JSON"),
}),
options: z.object({
normalizationJson: z
.string()
.default('{"rules":[]}')
.describe(
"Recorded literal normalization policy JSON or JSON file path for browser scenarios",
"Recorded literal normalization policy for browser scenarios",
),
}),
run: ({ args, options }) =>
logCliCommand(logger, "compare-web-captures", async () => {
const inputs = await parseJsonArguments("compare_web_captures", {
before: args.beforeJson,
after: args.afterJson,
normalization: options.normalizationJson,
});
if (!inputs.ok) return inputs.error;
const { before, after, normalization } = inputs.value;
const before = safeParseJson(args.beforeJson);
const after = safeParseJson(args.afterJson);
const normalization = safeParseJson(options.normalizationJson);
if (!before.ok)
return inputError("compare_web_captures", [
invalidJsonIssue("before"),
]);
if (!after.ok)
return inputError("compare_web_captures", [
invalidJsonIssue("after"),
]);
if (!normalization.ok)
return inputError("compare_web_captures", [
invalidJsonIssue("normalization"),
]);
const scenarioComparison =
browserCaptureComparisonInputSchema.safeParse({
before_scenario: before,
after_scenario: after,
normalization,
before_scenario: before.value,
after_scenario: after.value,
normalization: normalization.value,
});
const parsed = scenarioComparison.success
? scenarioComparison
: browserCaptureComparisonInputSchema.safeParse({ before, after });
: browserCaptureComparisonInputSchema.safeParse({
before: before.value,
after: after.value,
});
if (!parsed.success) return inputError("compare_web_captures");
const result = await compareWebCaptureEvidence(
new CdpBrowserProvider(),
@@ -162,12 +168,8 @@ const registerScreenshotDiff = (
cli.command(CLI_COMMANDS.compareWebScreenshots, {
description: "Compare two self-verifying PNG artifact JSON values",
args: z.object({
beforeJson: z
.string()
.describe("Earlier screenshot artifact JSON or JSON file path"),
afterJson: z
.string()
.describe("Later screenshot artifact JSON or JSON file path"),
beforeJson: z.string().describe("Earlier screenshot artifact JSON"),
afterJson: z.string().describe("Later screenshot artifact JSON"),
}),
options: z.object({
channelThreshold: z
@@ -180,27 +182,12 @@ const registerScreenshotDiff = (
}),
run: ({ args, options }) =>
logCliCommand(logger, "compare-web-screenshots", async () => {
const operation = "compare_web_screenshots";
const inputs = await parseJsonArguments(operation, {
before: args.beforeJson,
after: args.afterJson,
});
if (!inputs.ok) return inputs.error;
const { before, after } = inputs.value;
const input = {
before,
after,
const parsed = compareWebScreenshotsInputSchema.safeParse({
before: parseJson(args.beforeJson),
after: parseJson(args.afterJson),
channel_threshold: options.channelThreshold,
};
const parsed = compareWebScreenshotsInputSchema.safeParse(input);
if (!parsed.success)
return browserCliError(
new AnalysisInputError(
operation,
{ cause: parsed.error },
projectInputIssues(parsed.error.issues, input),
),
);
});
if (!parsed.success) return inputError("compare_web_screenshots");
const result = await compareWebScreenshotEvidence(
new CdpBrowserProvider(),
parsed.data,
@@ -210,40 +197,11 @@ const registerScreenshotDiff = (
});
};
/**
* Parse each named inline JSON argument or local JSON file, like other
* JSON-input commands. Screenshot artifacts and scenario captures embed PNG
* bytes that can exceed the host's command-line length limit.
*/
const parseJsonArguments = async (
operation: string,
fields: Readonly<Record<string, string>>,
): Promise<
| { readonly ok: true; readonly value: Readonly<Record<string, unknown>> }
| { readonly ok: false; readonly error: JsonValue }
> => {
const values: Record<string, unknown> = {};
for (const [field, value] of Object.entries(fields)) {
const parsed = await parseCliJsonInput(value, operation);
if (!parsed.ok)
return isFileInputFailure(parsed.error)
? parsed
: {
ok: false,
error: inputError(operation, [invalidJsonIssue(field)]),
};
values[field] = parsed.value;
}
return { ok: true, value: values };
const parseJson = (value: string): unknown => {
const parsed = safeParseJson(value);
return parsed.ok ? parsed.value : undefined;
};
/** File read and decode failures name the selected path; inline JSON does not. */
const isFileInputFailure = (error: JsonValue): boolean =>
typeof error === "object" &&
error !== null &&
!Array.isArray(error) &&
"input_path" in error;
const invalidJsonIssue = (field: string): AnalysisInputIssue => ({
path: [field],
reason: "invalid_format",
+5
View File
@@ -54,6 +54,7 @@ const registerOverviewCommands = (
options: overviewOptions.extend({
integrityPolicy: javascriptApplicationOptions.shape.integrityPolicy,
}),
alias: { integrityPolicy: "integrity-policy" },
run: async ({ args, options }) => {
// Route JavaScript targets exactly like analyze-javascript-application,
// including typed cancellation; cancellation wraps logging so its exit
@@ -273,6 +274,9 @@ const registerTraceCommand = (
"target-format": formatSelectionOption,
provider: providerSelectionOption,
}),
alias: {
caseSensitive: "case-sensitive",
},
run: ({ args, options }) =>
logCliCommand(logger, "trace", () =>
runDirectAnalysis(
@@ -451,6 +455,7 @@ const registerSearchCommand = (
"target-format": formatSelectionOption,
provider: providerSelectionOption,
}),
alias: { caseSensitive: "case-sensitive" },
run: ({ args, options }) =>
logCliCommand(logger, "search", () =>
runDirectAnalysis(
+13
View File
@@ -157,6 +157,13 @@ export const registerCoreNativeCommands = (
"target-format": formatSelectionOption,
provider: providerSelectionOption,
}),
alias: {
maxDepth: "max-depth",
maxFunctions: "max-functions",
maxCallSites: "max-call-sites",
maxNodes: "max-nodes",
maxEdges: "max-edges",
},
run: ({ args, options }) =>
logCliCommand(logger, CLI_COMMANDS.traceNativeValues, () =>
runDirectAnalysis(
@@ -276,6 +283,7 @@ const registerNativeDispatchMetadataCommand = (
"target-format": formatSelectionOption,
provider: providerSelectionOption,
}),
alias: { maxRecords: "max-records" },
run: async ({ args, options }) =>
logCliCommand(logger, "inspect-native-dispatch-metadata", () =>
runDirectAnalysis(
@@ -335,6 +343,11 @@ const registerNativeUiActionCommand = (
"target-format": formatSelectionOption,
provider: providerSelectionOption,
}),
alias: {
maxDepth: "max-depth",
maxNodes: "max-nodes",
maxEdges: "max-edges",
},
run: async ({ args, options }) =>
logCliCommand(logger, "trace-native-ui-action", () =>
runDirectAnalysis(
+1
View File
@@ -190,6 +190,7 @@ const registerJavaScriptApplicationCommand = (
path: z.string().describe("ASAR or extracted application path"),
}),
options: javascriptApplicationOptions,
alias: { integrityPolicy: "integrity-policy" },
run: ({ args, options }) =>
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.analyzeJavaScriptApplication, () =>
+3 -15
View File
@@ -4,27 +4,16 @@ import { bufferedJsonParts, jsonParts } from "../domain/jsonSerialization.js";
type JsonOutputFormat = "json" | "jsonl";
/**
* Write ordinary JSON without constructing a document-sized string.
*
* `json` is indented for a terminal reader. Piped or redirected output, which
* agents and scripts consume, is compact: indentation adds no information and
* measured 16-45% more model tokens on real results. The value is identical.
*/
/** Write ordinary JSON without constructing a document-sized string. */
export const writeJsonOutput = async (
value: unknown,
destination: Writable,
format: JsonOutputFormat = "json",
): Promise<void> => {
for (const part of bufferedJsonParts(
jsonOutputParts(value, format, isTerminal(destination)),
))
for (const part of bufferedJsonParts(jsonOutputParts(value, format)))
await writePart(destination, part);
};
const isTerminal = (destination: Writable): boolean =>
"isTTY" in destination && destination.isTTY === true;
const writePart = (destination: Writable, part: string): Promise<void> =>
new Promise((resolve, reject) => {
const onError = (cause: unknown): void => reject(cause);
@@ -50,7 +39,6 @@ const writePart = (destination: Writable, part: string): Promise<void> =>
export function* jsonOutputParts(
value: unknown,
format: JsonOutputFormat,
pretty = true,
): Generator<string> {
if (format === "jsonl" && Array.isArray(value)) {
for (const item of value) {
@@ -59,6 +47,6 @@ export function* jsonOutputParts(
}
return;
}
yield* jsonParts(value, pretty && format === "json");
yield* jsonParts(value, format === "json");
yield "\n";
}
+1
View File
@@ -148,6 +148,7 @@ const registerMaintenanceCommands = (
.default(false)
.describe("Also remove REA caches and state"),
}),
alias: { purgeData: "purge-data" },
run: ({ options }) =>
logCliCommand(
logger,
+5
View File
@@ -65,6 +65,10 @@ export const registerUtilityCommands = (
.default(500)
.describe("Maximum accessibility nodes per capture"),
}),
alias: {
windowId: "window-id",
maxNodes: "max-nodes",
},
run: ({ args, options }) =>
logCliCommand(logger, command, async () => {
const parameters = {
@@ -168,6 +172,7 @@ const registerNativeCommands = (
"Plist path relative to the app root (default: Contents/Info.plist)",
),
}),
alias: { relativePath: "relative-path" },
run: ({ args, options }) =>
logCliCommand(logger, "inspect-plist", () =>
runProviderAnalysis(
+1 -3
View File
@@ -2,6 +2,7 @@ import { Cli } from "incur";
import { z } from "zod";
import { captureBrowserScenario } from "./application/BrowserScenarioCaptureService.js";
import { createBrowserScenarioProvider } from "./composition/browserScenario.js";
import { CLI_COMMANDS } from "./cliCommandNames.js";
import { parseCliJsonInput, resolveCliJsonPaths } from "./cliJsonInput.js";
import { logCliCommand } from "./cliLogging.js";
@@ -43,9 +44,6 @@ export const registerBrowserScenarioCommands = (
projectInputIssues(scenario.error.issues, input.value),
),
);
// Load Playwright only for this command, not during CLI startup.
const { createBrowserScenarioProvider } =
await import("./composition/browserScenario.js");
const result = await captureBrowserScenario(
createBrowserScenarioProvider(environment),
scenario.data,
+2 -17
View File
@@ -8,14 +8,14 @@ import { evidenceResultOf } from "./toolOutputSchemaPrimitives.js";
const RETAINED_EXAMPLE_ID = `ev_${"a".repeat(64)}`;
/** Selected views of already completed layout, JavaScript and native function Evidence. */
/** Selected views of already completed layout and JavaScript analysis Evidence. */
export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
{
name: "inspect_analysis_view",
...toolContractMetadata("inspect_analysis_view"),
kind: "application",
description:
"Inspect selected views of completed binary-layout, JavaScript application or native analyze_function Evidence without repeating analysis. Use an exact same-session retained evidence_id or portable inline Evidence. Select a summary, layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Native views select procedure, pseudocode, assembly, callers/callees, references and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.",
"Inspect a selected view of completed binary-layout or JavaScript application Evidence without repeating analysis. Use an exact same-session retained evidence_id or portable inline Evidence. Select a summary, layout mitigations/linkage facet, one section/symbol/module, or a stable page with a positive limit. Module pages contain JavaScript assets, bundled modules and source modules; use node_id when paths are ambiguous or unavailable. Returns selected facts inline with artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the retained Evidence when needed.",
inputSchema: inspectAnalysisViewInputSchema,
outputSchema: evidenceResultOf(analysisViewResultSchema),
examples: [
@@ -43,21 +43,6 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
},
},
},
{
title: "Page native high-pcode operations from retained Evidence",
input: {
source: {
kind: "retained-evidence",
evidence_id: RETAINED_EXAMPLE_ID,
},
view: {
kind: "native",
facet: "value_flow_operations",
offset: 0,
limit: 32,
},
},
},
{
title: "Page module identities from retained application Evidence",
input: {
@@ -1,69 +1,11 @@
import { Ajv } from "ajv";
import { Ajv2020 } from "ajv/dist/2020.js";
import { AjvJsonSchemaValidator } from "@modelcontextprotocol/client/validators/ajv";
import { expect, it } from "vitest";
import { z } from "zod";
import { toolContract } from "./toolContracts.js";
import { toolOutputSchemaWithMetadata } from "./toolSchemaMetadata.js";
it("keeps dialect-specific validators distinct after JSON schema serialization", () => {
const contract = {
...toolContract("binary_session"),
outputSchema: z.strictObject({ value: z.number().int().min(1).max(8) }),
};
const output = toolOutputSchemaWithMetadata(contract);
const provider = new AjvJsonSchemaValidator();
for (const target of ["draft-2020-12", "draft-07"] as const) {
const projected = output["~standard"].jsonSchema.output({ target });
const parsed = z
.record(z.string(), z.unknown())
.parse(JSON.parse(JSON.stringify(projected)));
const validate = provider.getValidator(parsed);
expect(validate({ value: 8 }).valid).toBe(true);
expect(validate({ value: 9 }).valid).toBe(false);
expect(validate({ value: 1.5 }).valid).toBe(false);
}
});
it("preserves explicit schema identity and relative reference resolution", () => {
const value = z.string();
const canonical = z.strictObject({ value });
const contract = {
...toolContract("binary_session"),
outputSchema: canonical,
};
const projected = toolOutputSchemaWithMetadata(contract)[
"~standard"
].jsonSchema.output({
target: "draft-2020-12",
libraryOptions: {
override: (context: {
readonly zodSchema: unknown;
readonly jsonSchema: Record<string, unknown>;
}) => {
if (context.zodSchema !== value) return;
delete context.jsonSchema.type;
context.jsonSchema.$ref = "child.json";
},
},
});
const ajv = new Ajv2020({ strict: false });
ajv.addSchema({ $id: "child.json", type: "string", minLength: 3 });
const validate = ajv.compile(projected);
expect(validate({ value: "abc" })).toBe(true);
expect(validate({ value: "ab" })).toBe(false);
const identified = toolOutputSchemaWithMetadata({
...contract,
outputSchema: canonical.meta({ $id: "https://example.test/result" }),
})["~standard"].jsonSchema.output({ target: "draft-2020-12" });
expect(identified.$id).toBe("https://example.test/result");
const explicit = ajv.compile(identified);
expect(explicit({ value: "abc" })).toBe(true);
expect(explicit({ value: 1 })).toBe(false);
});
it.each(["draft-2020-12", "draft-07"] as const)(
"preserves refined numeric bounds and strict repeated objects through %s references",
(target) => {
+4 -40
View File
@@ -2,7 +2,6 @@ import { z } from "zod";
import type { ToolContract } from "./toolContractTypes.js";
import { presentInputJsonSchema } from "./inputSchemaPresentation.js";
import { digestCanonicalValue } from "../domain/canonicalDigest.js";
const PROPERTY_DESCRIPTIONS: Readonly<Record<string, string>> = {
addresses: "Ordered provider-normalized procedure addresses to analyze.",
@@ -70,55 +69,20 @@ export const toolInputSchemaWithMetadata = <Contract extends ToolContract>(
},
);
/** Share output definitions and content-bound validator identities on the wire. */
/** Share repeated output definitions without changing canonical validation. */
export const toolOutputSchemaWithMetadata = <Contract extends ToolContract>(
contract: Contract,
): Contract["outputSchema"] =>
withAdvertisedJsonSchema(
contract.outputSchema,
"output",
(project) => (options) => {
const shared = project({
(project) => (options) =>
project({
...options,
libraryOptions: { reused: "ref", ...options.libraryOptions },
});
// An explicit ID owns the reference base. Relative external references
// also depend on that base, so leave those projections unchanged.
if ("$id" in shared || hasRelativeSchemaReference(shared)) return shared;
return {
...shared,
$id: `urn:rea:tool-output:sha256:${digestCanonicalValue(
{ target: options.target, schema: shared },
"Tool output schema",
)}`,
};
},
}),
);
const hasRelativeSchemaReference = (
schema: Record<string, unknown>,
): boolean => {
const pending: unknown[] = [schema];
const visited = new WeakSet<object>();
while (pending.length > 0) {
const current = pending.pop();
if (current === null || typeof current !== "object" || visited.has(current))
continue;
visited.add(current);
for (const [key, value] of Object.entries(current)) {
if (
["$id", "$ref", "$dynamicRef", "$recursiveRef"].includes(key) &&
typeof value === "string" &&
!value.startsWith("#") &&
!URL.canParse(value)
)
return true;
if (value !== null && typeof value === "object") pending.push(value);
}
}
return false;
};
const withAdvertisedJsonSchema = <Schema extends z.ZodType>(
canonical: Schema,
io: "input" | "output",
+13 -184
View File
@@ -1,8 +1,6 @@
import { expect, it } from "vitest";
import { resolveJavaScriptSourceMapReference } from "../javascript/javascriptSourceMapPaths.js";
import { functionDossierSchema } from "../hopperValues.js";
import { ghidraFunctionDossier } from "../ghidraValues.fixture.js";
import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../../contracts/javascript/javascriptRuntimeReconciliationExample.js";
import {
analysisViewBindJavaScriptGraphs,
@@ -50,190 +48,21 @@ const javascriptParent = (
limitations: JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE.limitations,
});
const nativeParent = (
result: unknown = ghidraFunctionDossier(),
): AnalysisViewParent => ({
evidenceId: "ev_" + "b".repeat(64),
operation: "analyze_function",
normalizedResult: result,
limitations: ["Retained provider limitation"],
artifact: { path: "/fixtures/example.exe", sha256: "a".repeat(64) },
it("accepts caller-selected page sizes and rejects malformed bounds", () => {
const input = {
source: { kind: "retained-evidence", evidence_id: `ev_${"a".repeat(64)}` },
view: { kind: "page", collection: "sections", offset: 0, limit: 500 },
};
expect(inspectAnalysisViewInputSchema.safeParse(input).success).toBe(true);
for (const limit of [0, -1, 0.5, Infinity, Number.MAX_SAFE_INTEGER + 1])
expect(
inspectAnalysisViewInputSchema.safeParse({
...input,
view: { ...input.view, limit },
}).success,
).toBe(false);
});
it("projects native procedure and bounded assembly and high-pcode with exact parent identity", () => {
const parent = nativeParent();
const procedure = projectAnalysisView(parent, {
kind: "native",
facet: "procedure",
offset: 0,
limit: 64,
});
if (!procedure.ok) throw procedure.error;
expect(procedure.value).toMatchObject({
kind: "native",
parent_operation: "analyze_function",
artifact: parent.artifact,
item: { address: "0x401000" },
});
expect(procedure.value.view_digest).not.toBe(parent.evidenceId.slice(3));
const assembly = projectAnalysisView(parent, {
kind: "native",
facet: "assembly",
offset: 0,
limit: 1,
});
if (!assembly.ok) throw assembly.error;
expect(assembly.value).toMatchObject({
coverage: { examined: 1, total: 2, next_offset: 1 },
item: ["0x401000: CALL 0x401020"],
});
const flow = projectAnalysisView(parent, {
kind: "native",
facet: "value_flow_operations",
offset: 0,
limit: 1,
});
if (!flow.ok) throw flow.error;
expect(flow.value).toMatchObject({ item: [{ opcode: "COPY" }] });
});
it("rejects invalid native parents, unsupported source operations and singleton offsets", () => {
const view = {
kind: "native",
facet: "assembly",
offset: 0,
limit: 4,
} as const;
expect(projectAnalysisView(nativeParent({}), view)).toMatchObject({
ok: false,
error: { _tag: "AnalysisInputError" },
});
expect(
projectAnalysisView({ ...nativeParent(), artifact: null }, view),
).toMatchObject({
ok: false,
error: { _tag: "AnalysisInputError" },
});
expect(projectAnalysisView(layoutParent(), view)).toMatchObject({
ok: false,
error: { _tag: "AnalysisInputError" },
});
expect(projectAnalysisView(javascriptParent(), view)).toMatchObject({
ok: false,
error: { _tag: "AnalysisInputError" },
});
expect(
projectAnalysisView(nativeParent(), {
kind: "native",
facet: "procedure",
offset: 1,
limit: 2,
}),
).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" } });
});
it("preserves native API limitations and unavailable residual unknowns", () => {
const original = functionDossierSchema.parse(ghidraFunctionDossier());
const view = {
kind: "native",
facet: "procedure",
offset: 0,
limit: 64,
} as const;
const unavailable = projectAnalysisView(
nativeParent({
...original,
native_api: {
available: false,
reason: "No decompiler",
residual_unknowns: ["Calling convention unresolved"],
},
}),
view,
);
if (!unavailable.ok) throw unavailable.error;
expect(unavailable.value.unknowns).toContain("Calling convention unresolved");
const available = projectAnalysisView(nativeParent(), view);
if (!available.ok) throw available.error;
if (!original.native_api?.available) throw new Error("expected API fixture");
expect(available.value.limitations).toEqual(
expect.arrayContaining(original.native_api.limitations),
);
});
it("reports the malformed native dossier field", () => {
const result = projectAnalysisView(
nativeParent({
...functionDossierSchema.parse(ghidraFunctionDossier()),
pseudocode: 12,
}),
{
kind: "native",
facet: "pseudocode",
offset: 0,
limit: 64,
},
);
expect(result).toMatchObject({
ok: false,
error: {
issues: expect.arrayContaining([
expect.objectContaining({
path: ["source", "normalized_result", "pseudocode"],
}),
]),
},
});
});
it("native pseudocode pages do not split surrogate pairs", () => {
const original = functionDossierSchema.parse(ghidraFunctionDossier());
const parent = nativeParent({ ...original, pseudocode: "ab😀cd" });
expect(
projectAnalysisView(parent, {
kind: "native",
facet: "pseudocode",
offset: 3,
limit: 2,
}),
).toMatchObject({ ok: false, error: { _tag: "AnalysisInputError" } });
const safe = projectAnalysisView(parent, {
kind: "native",
facet: "pseudocode",
offset: 2,
limit: 3,
});
if (!safe.ok) throw safe.error;
expect(safe.value).toMatchObject({
item: { text: "😀c", unit: "utf16-code-units" },
coverage: { examined: 3, next_offset: 5 },
});
});
it.each([
{ kind: "page", collection: "sections", offset: 0, limit: 500 },
{ kind: "native", facet: "assembly", offset: 0, limit: 500 },
])(
"accepts caller-selected page sizes and rejects malformed bounds: $kind",
(view) => {
const input = {
source: {
kind: "retained-evidence",
evidence_id: `ev_${"a".repeat(64)}`,
},
view,
};
expect(inspectAnalysisViewInputSchema.safeParse(input).success).toBe(true);
for (const limit of [0, -1, 0.5, Infinity, Number.MAX_SAFE_INTEGER + 1])
expect(
inspectAnalysisViewInputSchema.safeParse({
...input,
view: { ...input.view, limit },
}).success,
).toBe(false);
},
);
it("projects layout summary, facet, item, and stable pages", () => {
const parent = layoutParent();
const layout = analysisViewLayoutFixture();
+3 -60
View File
@@ -12,7 +12,6 @@ import { jsonObjectSchema, jsonValueSchema } from "../jsonValue.js";
import { err, ok, type Result } from "../result.js";
import type { AnalysisError } from "../analysisErrorBase.js";
import { binaryLayoutSchema } from "../native/binaryLayout.js";
import { functionDossierSchema } from "../hopperValues.js";
import {
parseOwnedJavaScriptApplicationAnalysisSteps,
type JavaScriptApplicationAnalysisResult,
@@ -20,7 +19,6 @@ import {
import { analysisInputErrorFromIssues } from "../inputIssueProjection.js";
import { projectBinaryLayoutView } from "./binaryLayoutView.js";
import { projectJavaScriptApplicationView } from "./javascriptApplicationView.js";
import { projectNativeFunctionView } from "./nativeFunctionView.js";
/** Public name of the selected-view workflow. */
export const INSPECT_ANALYSIS_VIEW_OPERATION = "inspect_analysis_view" as const;
@@ -75,35 +73,12 @@ const pageViewSchema = z.strictObject({
limit: z.number().int().positive(),
});
/** Bounded native function facets of already authenticated analyze_function Evidence. */
const nativeViewSchema = z.strictObject({
kind: z.literal("native"),
facet: z.enum([
"procedure",
"pseudocode",
"assembly",
"callers",
"callees",
"incoming_references",
"outgoing_references",
"value_flow_summary",
"value_flow_operations",
"value_flow_def_use",
"value_flow_effects",
"value_flow_parameters",
"value_flow_parameter_uses",
]),
offset: z.number().int().nonnegative().default(0),
limit: z.number().int().positive().default(64),
});
/** Caller-selected projection of already completed analysis Evidence. */
export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
summaryViewSchema,
facetViewSchema,
itemViewSchema,
pageViewSchema,
nativeViewSchema,
]);
/** Inline Evidence or an exact same-session retained reference. */
@@ -133,9 +108,7 @@ const coverageSchema = z.strictObject({
});
const artifactSchema = z.strictObject({
path: z
.string()
.describe("Exact recorded artifact path, including an empty string."),
path: z.string().min(1),
sha256: digestSchema,
});
@@ -177,7 +150,6 @@ const viewResultBase = {
parent_operation: z.enum([
"inspect_binary_layout",
"analyze_javascript_application",
"analyze_function",
]),
parent_digest: digestSchema,
view_digest: digestSchema,
@@ -213,13 +185,6 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
items: z.array(jsonValueSchema),
...viewResultBase,
}),
z.strictObject({
kind: z.literal("native"),
view: nativeViewSchema,
procedure_address: z.string().nullable(),
item: jsonValueSchema,
...viewResultBase,
}),
]);
export type AnalysisViewRequest = z.output<typeof analysisViewRequestSchema>;
@@ -235,7 +200,6 @@ export interface AnalysisViewParent {
readonly operation: string;
readonly normalizedResult: unknown;
readonly limitations: readonly string[];
readonly artifact?: { readonly path: string; readonly sha256: string } | null;
}
/** Construct a typed input failure for one selected-view constraint. */
@@ -252,11 +216,7 @@ export type UnsignedAnalysisView =
>
| Omit<Extract<AnalysisViewResult, { readonly kind: "facet" }>, "view_digest">
| Omit<Extract<AnalysisViewResult, { readonly kind: "item" }>, "view_digest">
| Omit<Extract<AnalysisViewResult, { readonly kind: "page" }>, "view_digest">
| Omit<
Extract<AnalysisViewResult, { readonly kind: "native" }>,
"view_digest"
>;
| Omit<Extract<AnalysisViewResult, { readonly kind: "page" }>, "view_digest">;
/** Attach the content digest that must change when projected bytes change. */
export const sealAnalysisView = (
@@ -301,7 +261,7 @@ const unsupportedParent = (
new AnalysisUnsupportedTargetError(
INSPECT_ANALYSIS_VIEW_OPERATION,
parentArtifactPath(parent),
`Parent Evidence operation ${JSON.stringify(parent.operation)} is not inspect_binary_layout, analyze_javascript_application or analyze_function.`,
`Parent Evidence operation ${JSON.stringify(parent.operation)} is not inspect_binary_layout or analyze_javascript_application.`,
);
const parentArtifactPath = (parent: AnalysisViewParent): string => {
@@ -367,22 +327,5 @@ export const projectAnalysisView = (
);
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
}
if (parent.operation === "analyze_function") {
const native = functionDossierSchema.safeParse(parent.normalizedResult);
if (!native.success)
return err(
analysisInputErrorFromIssues(
INSPECT_ANALYSIS_VIEW_OPERATION,
native.error.issues.map((issue) => ({
...issue,
path: ["source", "normalized_result", ...issue.path],
})),
{ source: { normalized_result: parent.normalizedResult } },
{ cause: native.error },
),
);
const projected = projectNativeFunctionView(parent, native.data, view);
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
}
return err(unsupportedParent(parent));
};
@@ -152,10 +152,6 @@ export const projectBinaryLayoutView = (
view: AnalysisViewRequest,
): Result<UnsignedAnalysisView, AnalysisError> => {
const shared = parentFields(parent, layout);
if (view.kind === "native")
return err(
layoutIncompatible("Native views require analyze_function Evidence."),
);
if (view.kind === "summary")
return ok({
kind: "summary",
@@ -243,10 +243,6 @@ export const projectJavaScriptApplicationView = (
): Result<UnsignedAnalysisView, AnalysisError> => {
const shared = parentFields(parent, analysis);
const modules = moduleEntries(analysis.graph.nodes);
if (view.kind === "native")
return err(
jsIncompatible("Native views require analyze_function Evidence."),
);
if (view.kind === "summary")
return ok({
kind: "summary",
@@ -1,202 +0,0 @@
import type { AnalysisError } from "../analysisErrorBase.js";
import { uniqueSorted } from "../canonicalOrdering.js";
import type { FunctionDossier } from "../hopperValues.js";
import { jsonValueSchema } from "../jsonValue.js";
import { err, ok, type Result } from "../result.js";
import {
analysisViewInputError,
type AnalysisViewCoverage,
type AnalysisViewParent,
type AnalysisViewRequest,
type UnsignedAnalysisView,
} from "./analysisView.js";
import {
completeWithinViewCoverage,
pageViewCoverage,
} from "./analysisViewCoverage.js";
type Artifact = { readonly path: string; readonly sha256: string };
const viewError = (message: string): AnalysisError =>
analysisViewInputError([
{ path: ["view"], reason: "invalid_value", message },
]);
const parentFields = (
parent: AnalysisViewParent,
dossier: FunctionDossier,
artifact: Artifact,
) => {
const flow = dossier.native_value_flow;
const api = dossier.native_api;
const unknowns: string[] = [];
if (flow === null) unknowns.push("native_value_flow was not recorded");
else if (!flow.available)
unknowns.push("native_value_flow unavailable: " + flow.reason);
else if (flow.truncated)
unknowns.push(
"native_value_flow is truncated; omitted observations remain unknown",
);
if (api === null) unknowns.push("native_api was not recorded");
else if (!api.available)
unknowns.push(
"native_api unavailable: " + api.reason,
...api.residual_unknowns,
);
return {
parent_evidence_id: parent.evidenceId,
parent_operation: "analyze_function" as const,
parent_digest: parent.evidenceId.slice(3),
procedure_address: dossier.procedure.address,
artifact,
limitations: uniqueSorted([
...parent.limitations,
...dossier.limitations,
...(flow?.limitations ?? []),
...(api?.available ? api.limitations : []),
]),
unknowns: uniqueSorted(unknowns),
};
};
const pseudocodePage = (
body: string,
offset: number,
limit: number,
): Result<
{
readonly item: { readonly text: string; readonly unit: "utf16-code-units" };
readonly coverage: AnalysisViewCoverage;
},
AnalysisError
> => {
const start = Math.min(offset, body.length);
const high = (n: number) => n >= 0xd800 && n <= 0xdbff;
const low = (n: number) => n >= 0xdc00 && n <= 0xdfff;
if (
start > 0 &&
start < body.length &&
high(body.charCodeAt(start - 1)) &&
low(body.charCodeAt(start))
)
return err(viewError("Pseudocode offset splits a UTF-16 surrogate pair."));
let end = Math.min(start + limit, body.length);
if (
end > start &&
end < body.length &&
high(body.charCodeAt(end - 1)) &&
low(body.charCodeAt(end))
)
end--;
if (end === start && start < body.length)
return err(
viewError(
"Increase pseudocode limit to include the next complete Unicode character.",
),
);
return ok({
item: { text: body.slice(start, end), unit: "utf16-code-units" },
coverage: pageViewCoverage(start, end - start, body.length),
});
};
const nativeRows = (
dossier: FunctionDossier,
facet: Exclude<
Extract<AnalysisViewRequest, { readonly kind: "native" }>["facet"],
"procedure" | "pseudocode" | "value_flow_summary"
>,
) => {
const sourceFlow = dossier.native_value_flow;
const flow = sourceFlow?.available ? sourceFlow : undefined;
return {
assembly: dossier.assembly,
callers: dossier.callers,
callees: dossier.callees,
incoming_references: dossier.incoming_references,
outgoing_references: dossier.outgoing_references,
value_flow_operations: flow?.operations,
value_flow_def_use: flow?.def_use,
value_flow_effects: flow?.effects,
value_flow_parameters: flow?.parameters,
value_flow_parameter_uses: flow?.parameter_uses,
}[facet];
};
/** Project bounded native function facets from complete retained Evidence without reanalysis. */
export const projectNativeFunctionView = (
parent: AnalysisViewParent,
dossier: FunctionDossier,
view: AnalysisViewRequest,
): Result<UnsignedAnalysisView, AnalysisError> => {
if (view.kind !== "native")
return err(viewError("analyze_function Evidence requires a native view."));
const artifact = parent.artifact;
if (!artifact)
return err(
analysisViewInputError([
{
path: ["source", "subject"],
reason: "invalid_value",
message:
"Native function Evidence requires an artifact-bound subject.",
},
]),
);
const common = parentFields(parent, dossier, artifact);
const singleton = (item: unknown): UnsignedAnalysisView => ({
kind: "native",
view,
item: jsonValueSchema.parse(item),
coverage: completeWithinViewCoverage(1, 1),
...common,
});
const flow = dossier.native_value_flow;
if (view.facet === "procedure" || view.facet === "value_flow_summary") {
if (view.offset !== 0)
return err(viewError("Singleton native facets require offset 0."));
if (view.facet === "procedure") return ok(singleton(dossier.procedure));
const status =
flow === null
? { available: false, reason: "not recorded" }
: !flow.available
? { available: false, reason: flow.reason }
: {
available: true,
provenance: flow.provenance,
truncated: flow.truncated,
omitted_operations_lower_bound:
flow.omitted_operations_lower_bound,
known_omitted_inputs: flow.known_omitted_inputs,
known_omitted_edges: flow.known_omitted_edges,
operations: flow.operations.length,
def_use: flow.def_use.length,
effects: flow.effects.length,
parameters: flow.parameters.length,
parameter_uses: flow.parameter_uses.length,
};
return ok(singleton(status));
}
if (view.facet === "pseudocode") {
const page = pseudocodePage(dossier.pseudocode, view.offset, view.limit);
return page.ok
? ok({ kind: "native", view, ...page.value, ...common })
: page;
}
const rows = nativeRows(dossier, view.facet);
if (rows === undefined)
return ok(
singleton({
available: false,
reason: flow?.available === false ? flow.reason : "not recorded",
}),
);
const items = rows.slice(view.offset, view.offset + view.limit);
return ok({
kind: "native",
view,
item: jsonValueSchema.parse(items),
coverage: pageViewCoverage(view.offset, items.length, rows.length),
...common,
});
};
@@ -509,46 +509,3 @@ describe("AppKit keyed archive connectors", () => {
expect(to).toEqual(expect.arrayContaining(["objc_selector", "unknown"]));
});
});
it("names the archive once per evidence record", () => {
const relativePath = "Views/Main.storyboardc/scene.nib/objects.nib";
const result = buildInterfaceBuilderAnalysis({
targetSha256: hash,
toolVersion: "test",
documents: [
{
relativePath,
archiveSha256: hash,
documentKind: "storyboard_scene",
raw: {
"com.apple.ibtool.document.objects": {
controller: { customClass: "StoreViewController" },
next: { customClass: "BuildViewController" },
},
"com.apple.ibtool.document.connections": {
controller: [
{ type: "segue", identifier: "showBuild", destinationId: "next" },
],
},
},
},
],
limits: interfaceBuilderLimitsSchema.parse({}),
});
const evidence = [...result.graph.nodes, ...result.graph.edges].flatMap(
(item) => item.evidence,
);
expect(new Set(evidence.map(({ artifact_path }) => artifact_path))).toEqual(
new Set([relativePath]),
);
expect(evidence.map(({ description }) => description)).toEqual(
expect.arrayContaining([
"compiled document",
"object controller",
"segue showBuild",
]),
);
expect(
evidence.filter(({ description }) => description.includes(relativePath)),
).toEqual([]);
});
+1 -3
View File
@@ -72,9 +72,7 @@ export const buildInterfaceBuilderAnalysis = (input: {
const evidenceFor = (description: string) => [
{
kind: "interface_builder_resource" as const,
// artifact_path names the archive; repeating it here cost a path per
// node and edge.
description,
description: `${document.relativePath}: ${description}`,
location: { address: null, file_offset: null },
artifact_path: document.relativePath,
artifact_sha256: document.archiveSha256,
-10
View File
@@ -1,10 +0,0 @@
import { digestCanonicalValue } from "./canonicalDigest.js";
/** Retain short exact string keys or commit their value in a reserved digest namespace. */
export const compactStringIdentityKey = (value: string): string => {
const prefix = "value-sha256:";
const digestKey = `${prefix}${digestCanonicalValue(value, "String identity")}`;
return value.length <= digestKey.length && !value.startsWith(prefix)
? value
: digestKey;
};
+1 -18
View File
@@ -35,23 +35,7 @@ const projectIssue = (
typeof part === "string" || typeof part === "number" ? [part] : [],
);
// Branch issue paths are relative to the union; branches often agree.
if (issue.code === "invalid_union") {
// Unmatched discriminators and ambiguous unions have no branch issues.
// Keep their outer failure instead of flattening it into an empty list.
if (issue.errors.length === 0) {
const expected =
"options" in issue ? issue.options?.filter(isSafeExpected) : undefined;
return [
{
path,
reason:
valueAtPath(input, path) === undefined
? "missing_argument"
: "invalid_value",
...(expected === undefined ? {} : { expected }),
},
];
}
if (issue.code === "invalid_union")
return uniqueIssues(
issue.errors.flatMap((branch) =>
branch.flatMap((branchIssue) =>
@@ -62,7 +46,6 @@ const projectIssue = (
),
),
);
}
if (issue.code === "invalid_key")
return issue.issues.flatMap((keyIssue) =>
projectIssue(
@@ -12,7 +12,6 @@ import type {
import {
boundedExpression,
collectContextBridgeMembers,
electronCalleeName,
electronStaticValue,
objectProperty,
} from "./electronStaticAnalysisValues.js";
@@ -20,6 +19,7 @@ import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
import { addLocatedFinding } from "./javascriptStaticAnalysisFindings.js";
import {
argumentNode,
calleeName,
range,
staticPath,
staticPathResolutionContext,
@@ -42,7 +42,7 @@ const inspectBrowserWindow = (
node: t.NewExpression,
context: JavaScriptFindingContext,
): void => {
const name = electronCalleeName(node.callee, context);
const name = calleeName(node.callee);
if (name !== "BrowserWindow" && !name.endsWith(".BrowserWindow")) return;
const options = argumentNode(node.arguments[0]);
const collected = collectWindowOptions(context.source, options);
@@ -165,7 +165,7 @@ const inspectContextBridge = (
node: t.CallExpression,
context: JavaScriptFindingContext,
): void => {
const name = electronCalleeName(node.callee, context);
const name = calleeName(node.callee);
const main =
name === "contextBridge.exposeInMainWorld" ||
name.endsWith(".contextBridge.exposeInMainWorld");
@@ -213,7 +213,7 @@ const inspectUtilityProcess = (
node: t.CallExpression,
context: JavaScriptFindingContext,
): void => {
const name = electronCalleeName(node.callee, context);
const name = calleeName(node.callee);
if (name !== "utilityProcess.fork" && !name.endsWith(".utilityProcess.fork"))
return;
const moduleNode = argumentNode(node.arguments[0]);
@@ -7,7 +7,6 @@ import type {
} from "./electronStaticAnalysisTypes.js";
import {
boundedExpression,
electronCalleeName,
electronStaticValue,
handlerKind,
} from "./electronStaticAnalysisValues.js";
@@ -37,7 +36,7 @@ const inspectIpcCall = (
node: t.CallExpression,
context: JavaScriptFindingContext,
): void => {
const name = electronCalleeName(node.callee, context);
const name = calleeName(node.callee);
const descriptor = ipcDescriptor(name);
if (descriptor === undefined) return;
const channelNode = argumentNode(node.arguments[0]);
@@ -6,26 +6,7 @@ import {
readExactJavaScriptLiteral,
semanticStaticPropertyName,
} from "./javascriptAstValues.js";
import { calleeName, sourceSlice } from "./javascriptStaticAnalysisHelpers.js";
import type { JavaScriptFindingContext } from "./javascriptStaticAnalysisState.js";
/** Name a callee, spelling a root renamed from Electron as its export. */
export const electronCalleeName = (
callee: t.Node,
context: JavaScriptFindingContext,
): string => {
const name = calleeName(callee);
let root = callee;
while (t.isMemberExpression(root) || t.isOptionalMemberExpression(root)) {
if (!t.isNode(root.object)) return name;
root = root.object;
}
if (!t.isIdentifier(root)) return name;
const exported = context.electronBindings?.get(root.start ?? -1);
return exported === undefined
? name
: `${exported}${name.slice(root.name.length)}`;
};
import { sourceSlice } from "./javascriptStaticAnalysisHelpers.js";
/** Preserve one literal value or the exact inert expression. */
export const electronStaticValue = (
@@ -1,103 +0,0 @@
import { describe, expect, it } from "vitest";
import {
createJavaScriptApplicationEdge,
createJavaScriptApplicationGraph,
type JavaScriptApplicationGraph,
} from "./javascriptApplicationGraph.js";
import {
artifactEvidence,
completeCoverage,
fixtureNode,
} from "./javascriptApplicationGraph.fixture.js";
import { compareJavaScriptApplicationVersions } from "./javascriptApplicationVersionComparison.js";
const LEFT_SHA = "a".repeat(64);
const RIGHT_SHA = "b".repeat(64);
const digest = (value: number): string => value.toString(16).padStart(64, "0");
const buildGraph = (
artifactSha: string,
changed: number,
): JavaScriptApplicationGraph => {
const nodes = ["a", "b", "c", "d"].map((name, index) =>
fixtureNode(
"javascript-module",
{
strategy: "canonical-path",
stability: "artifact-version",
artifact_sha256: artifactSha,
path: `${name}.js`,
},
`${name}.js`,
{
structural_fingerprint_sha256: digest(index + 1),
structural_fingerprint_algorithm: "fixture",
source_sha256: digest(index === 1 || index === 3 ? changed : 0),
},
artifactEvidence(artifactSha, `${name}.js`),
),
);
const [a, b] = nodes;
if (a === undefined || b === undefined)
throw new Error("Missing fixture node");
return createJavaScriptApplicationGraph({
schema: "JavaScriptApplicationGraph",
root_node_ids: [a.node_id],
nodes,
edges: [
createJavaScriptApplicationEdge({
source_node_id: a.node_id,
target_node_id: b.node_id,
relation: "imports",
properties: {},
evidence: artifactEvidence(artifactSha, "a.js"),
}),
],
coverage: completeCoverage,
limitations: [],
});
};
describe("application change graph", () => {
it("anchors each changed_from edge to its own right-hand observation", () => {
const left = buildGraph(LEFT_SHA, 1);
const right = buildGraph(RIGHT_SHA, 2);
const result = compareJavaScriptApplicationVersions({
left: {
evidenceId: `ev_${"1".repeat(64)}`,
rootArtifactSha256: LEFT_SHA,
graph: left,
},
right: {
evidenceId: `ev_${"2".repeat(64)}`,
rootArtifactSha256: RIGHT_SHA,
graph: right,
},
leftNativeEvidence: [],
rightNativeEvidence: [],
});
const changedFrom = result.graph.edges.filter(
({ relation }) => relation === "changed_from",
);
expect(changedFrom).toHaveLength(2);
for (const edge of changedFrom) {
const source = right.nodes.find(
({ node_id: id }) => id === edge.source_node_id,
);
const target = left.nodes.find(
({ node_id: id }) => id === edge.target_node_id,
);
expect(source?.observations[0]?.label).toBe(
target?.observations[0]?.label,
);
expect(edge.evidence.artifact).toEqual(
source?.observations[0]?.evidence.artifact,
);
expect(edge.evidence.location).toEqual(
source?.observations[0]?.evidence.location,
);
}
});
});
@@ -53,14 +53,8 @@ export const buildJavaScriptApplicationChangeGraph = (
({ source_node_id: source, target_node_id: target }) =>
retained.has(source) && retained.has(target),
);
const leftById = new Map(
input.left.nodes.map((node) => [node.node_id, node]),
);
const rightById = new Map(
input.right.nodes.map((node) => [node.node_id, node]),
);
const comparisonEdges = input.items.flatMap((item) =>
changedFromEdge(item, input, retained, leftById, rightById),
changedFromEdge(item, input, retained),
);
const candidateEdges = uniqueEdges([...sourceEdges, ...comparisonEdges]);
const rootNodeIds = preferredRoots.filter((nodeId) => retained.has(nodeId));
@@ -110,8 +104,6 @@ const changedFromEdge = (
item: ApplicationVersionComparisonItem,
input: ChangeGraphInput,
retained: ReadonlySet<string>,
leftById: ReadonlyMap<string, ApplicationNode>,
rightById: ReadonlyMap<string, ApplicationNode>,
): ApplicationEdge[] => {
const left = item.left_node_id;
const right = item.right_node_id;
@@ -125,12 +117,7 @@ const changedFromEdge = (
!retained.has(right)
)
return [];
const evidence = comparisonEvidence(
item,
input,
leftById.get(left),
rightById.get(right),
);
const evidence = comparisonEvidence(item, input);
return [
createJavaScriptApplicationEdge({
source_node_id: right,
@@ -151,9 +138,13 @@ const changedFromEdge = (
const comparisonEvidence = (
item: ApplicationVersionComparisonItem,
input: ChangeGraphInput,
left: ApplicationNode | undefined,
right: ApplicationNode | undefined,
): ApplicationGraphEvidence => {
const right = input.right.nodes.find(
({ node_id: id }) => id === item.right_node_id,
);
const left = input.left.nodes.find(
({ node_id: id }) => id === item.left_node_id,
);
const source =
right?.observations[0]?.evidence ?? left?.observations[0]?.evidence;
const complete =
@@ -109,7 +109,7 @@ type FixtureNodeArguments = [
evidence: ApplicationGraphEvidence,
];
export const fixtureNode = (
const fixtureNode = (
...[kind, identity, label, properties, evidence]: FixtureNodeArguments
): FixtureNode =>
createJavaScriptApplicationNode({
@@ -76,16 +76,6 @@ export const semanticStaticPropertyKey = (
return !computed && t.isIdentifier(property) ? property.name : null;
};
/** Read exact consumed object-pattern keys, leaving dynamic keys unknown. */
export const semanticObjectPatternKeys = (
pattern: t.ObjectPattern,
): readonly string[] =>
pattern.properties.flatMap((property) => {
if (t.isRestElement(property)) return [];
const key = semanticStaticPropertyKey(property.key, property.computed);
return key === null ? [] : [key];
});
/**
* Display an exact JavaScript string as nonempty label text. Graph labels and
* artifact-local keys are nonempty, so the legal empty value is shown as `""`;
@@ -590,24 +590,3 @@ describe("property mutations through TypeScript satisfies aliases", () => {
).toEqual({ status: "literal", value: "TOKEN" });
});
});
describe("property mutation collection on minified bundles", () => {
it("records each write to one object without replaying earlier writes", () => {
const writes = Array.from(
{ length: 2000 },
(_, index) => `source.p${index} = ${index};`,
).join(" ");
const start = performance.now();
const value = resultValue(
`const source = { token: "TOKEN" }; ${writes} return [source.token, source.p1999];`,
);
expect(performance.now() - start).toBeLessThan(2000);
expect(value).toMatchObject({
status: "array",
items: [
{ value: { status: "literal", value: "TOKEN" } },
{ value: { status: "unknown" } },
],
});
}, 30000);
});
@@ -9,163 +9,6 @@ const resultValue = (body: string) =>
"result",
).returnSites[0]?.value;
describe("bounded member path propagation", () => {
it("bounds cyclic member paths with no observed container origin", () => {
const branches = Array.from(
{ length: 16 },
(_, index) => `if (flag === ${index}) node = node.m${index};`,
).join(" else ");
const start = performance.now();
const callable = onlyCallable(
analyzeJavaScriptSemantics(
`export function result(seed) { let node=seed; while(node) { ${branches} use(node.state); } return seed.keep; }`,
),
"result",
);
expect(callable.returnSites[0]?.value.status).toBe("unknown");
expect(performance.now() - start).toBeLessThan(2000);
}, 30000);
it("preserves Array prototype effects through an otherwise unknown alias", () => {
expect(
resultValue(
"const values=[1]; const prototype=Array.prototype; prototype[Symbol.iterator]=custom; for(const item of values) { mutate(item); } return values[0];",
)?.status,
).toBe("unknown");
});
it("bounds loops derived from a scalar member while preserving sibling facts", () => {
const branches = Array.from(
{ length: 16 },
(_, index) => `if (flag === ${index}) node = node.m${index};`,
).join(" else ");
const start = performance.now();
expect(
resultValue(
`const source={child:1,keep:{value:3}}; let node=source.child; while(node) { ${branches} use(node.state); } return source.keep.value;`,
),
).toEqual({ status: "literal", value: 3 });
expect(performance.now() - start).toBeLessThan(2000);
}, 30000);
it.each([
["child:external,", "child", "present"],
["", "missing", "unknown-coverage"],
])(
"retains slot presence below an unobserved container: %s",
(property, name, presence) => {
expect(
resultValue(
`const source={${property}keep:3}; const alias=source.${name}; alias.deep.value=2; return source;`,
),
).toMatchObject({
status: "object",
properties: expect.arrayContaining([
{
name,
presence,
value: {
status: "unknown",
reason: "This property may have been mutated.",
},
},
{
name: "keep",
presence: "present",
value: { status: "literal", value: 3 },
},
]),
});
},
);
it.each([
["use(node);", "m0.value", "unknown"],
["node.value = 2;", "m0.value", "unknown"],
["node.value = 2;", "mode", "literal"],
])(
"bounds a loop that reassigns one reference through many members: %s seed.%s",
(effect, read, status) => {
const branches = Array.from(
{ length: 9 },
(_, index) => `if (flag === ${index}) node = node.m${index};`,
).join(" else ");
const start = performance.now();
const value = resultValue(
`const seed = { mode: "initial", m0: { value: 1 } }; let node = seed; while (node) { ${branches} ${effect} } return seed.${read};`,
);
expect(performance.now() - start).toBeLessThan(2000);
expect(value?.status).toBe(status);
},
30000,
);
it.each([
["const alias = flag ? owner.child : owner; alias.value = 2;", "value"],
["const alias = flag ? owner : owner.child; alias.value = 2;", "value"],
["const alias = flag ? owner.child : owner; alias[key] = 2;", "value"],
[
"const alias = flag ? owner.child : owner; alias.child = 2;",
"child.value",
],
[
"let alias = owner; let i = 0; while (i++ < 1) { alias = flag ? alias.child : alias; alias.value = 2; }",
"value",
],
])(
"keeps a write that reaches a child through a longer path: %s",
(body, read) => {
expect(
resultValue(
`const shared = { value: 1, child: { value: 1 } }; const owner = { child: shared }; ${body} return shared.${read};`,
)?.status,
).toBe("unknown");
},
);
it.each([
"mutate(source.left, source.right);",
"const alias = flag ? source.left : source.right; mutate(alias);",
"const copy = flag ? [source.left] : [source.right]; mutate(copy);",
"const alias = flag ? source.left : source.right; alias.value = 2;",
])("keeps members outside the mutated paths: %s", (body) => {
expect(
resultValue(
`const source = { left: { value: 1 }, right: { value: 2 }, keep: { value: 3 } }; ${body} return source.keep.value;`,
),
).toEqual({ status: "literal", value: 3 });
});
});
describe("distinct observable mutation paths", () => {
it.each([
["alias.value = 2;", "left"],
["alias.value = 2;", "right"],
["mutate(alias);", "left"],
["mutate(alias);", "right"],
])(
"propagates both conditional paths to shared bindings: %s %s",
(effect, target) => {
expect(
resultValue(
`const left={value:1}; const right={value:2}; const source={left,right}; const alias=flag?source.left:source.right; ${effect} return ${target}.value;`,
)?.status,
).toBe("unknown");
},
);
it.each(["mutate(alias);", "alias.value = 2;"])(
"preserves sibling facts below a shared prefix: %s",
(effect) => {
expect(
resultValue(
`const shared={left:{value:1},right:{value:2},keep:{value:3}}; const source={nested:shared}; const alias=flag?source.nested.left:source.nested.right; ${effect} return shared.keep.value;`,
),
).toEqual({ status: "literal", value: 3 });
},
);
});
describe("reference lifetimes across calls and shallow copies", () => {
it("bounds dynamic destructuring across shared alias branches", () => {
const declarations = Array.from(
@@ -229,8 +72,6 @@ describe("reference lifetimes across calls and shallow copies", () => {
"mutate(alias); alias={};",
"if(flag) alias={}; mutate(alias);",
"alias={}; if(flag) mutate(alias);",
"alias={}; flag && mutate(alias);",
"alias={}; flag ? mutate(alias) : noop();",
"function invoke(){mutate(alias);} alias={}; invoke();",
"alias={}; mutate(alias); alias=source;",
])("retains uncertain call ordering: %s", (body) => {
@@ -14,10 +14,7 @@ import {
immutableSemanticBindings,
immutableSemanticScopes,
} from "./javascriptSemanticProjection.js";
import {
semanticObjectPatternKeys,
semanticStaticPropertyKey,
} from "./javascriptAstValues.js";
import { semanticStaticPropertyKey } from "./javascriptAstValues.js";
import type {
JavaScriptSemanticAnalysisState,
JavaScriptSemanticBindingState,
@@ -32,7 +29,6 @@ import {
semanticVariableScope,
} from "./javascriptSemanticState.js";
import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js";
import { semanticRequireOrigin } from "./javascriptSemanticRequireOrigin.js";
import { collectSemanticMemberMutations } from "./javascriptSemanticMemberMutations.js";
import {
collectSemanticReturns,
@@ -286,92 +282,6 @@ const resolveGlobalAliasFact = (
return openGlobalFact(name);
};
const ELECTRON_MODULE = /^electron(?:\/(?:common|main|renderer|utility))?$/u;
/**
* Electron export paths for call and construction roots bound to Electron
* under another name, such as esbuild's `import { ipcMain as ipcMain2 }`.
* Keys are root identifier offsets; roots spelled as their export are omitted.
*/
export const classifyParsedJavaScriptElectronBindings = (
file: ParsedJavaScriptSource,
): ReadonlyMap<number, string> => {
const facts = new Map<number, string>();
const state = createState(file.program);
collectDefinitions(file.program, state);
const exportsByBinding = new Map<string, string | null>();
traverseJavaScriptAst(file.program, {
enter: (node) => {
if (
!t.isCallExpression(node) &&
!t.isOptionalCallExpression(node) &&
!t.isNewExpression(node)
)
return;
const root = calleeRoot(node.callee);
if (root === undefined) return;
const binding = resolveSemanticBindingState(state, root, root.name);
if (binding === undefined) return;
let exported = exportsByBinding.get(binding.bindingId);
if (exported === undefined) {
exported = electronExport(binding, state);
exportsByBinding.set(binding.bindingId, exported);
}
if (exported !== null && exported !== root.name)
facts.set(root.start ?? -1, exported);
},
});
return facts;
};
const calleeRoot = (node: t.Node): t.Identifier | undefined => {
while (t.isMemberExpression(node) || t.isOptionalMemberExpression(node)) {
if (!t.isNode(node.object)) return undefined;
node = node.object;
}
return t.isIdentifier(node) ? node : undefined;
};
// Read import and require origins directly; general value evaluation is
// unnecessary here and costly across large vendor bundles.
const electronExport = (
binding: JavaScriptSemanticBindingState,
state: JavaScriptSemanticAnalysisState,
): string | null => {
const origin = bindingOrigin(binding, state);
return origin !== undefined &&
ELECTRON_MODULE.test(origin.specifier) &&
origin.importedPath.length > 0
? origin.importedPath.join(".")
: null;
};
const bindingOrigin = (
binding: JavaScriptSemanticBindingState,
state: JavaScriptSemanticAnalysisState,
): JavaScriptModuleOrigin | undefined => {
if (binding.directOrigins.length > 0)
return binding.directOrigins.length === 1
? binding.directOrigins[0]
: undefined;
const [initializer] = binding.initializers;
// A var/let declaration permits writes without proving one occurred.
// Require one unconditional initializer and no actual binding writes.
if (
initializer === undefined ||
binding.initializers.length !== 1 ||
binding.definitions.some(({ kind }) => kind === "assignment") ||
state.conditionalInitializers.has(initializer.node)
)
return undefined;
const origin = semanticRequireOrigin(initializer.node, state);
const projection = initializer.projection;
return origin === undefined ||
!projection.every((key): key is string => typeof key === "string")
? undefined
: { ...origin, importedPath: [...origin.importedPath, ...projection] };
};
const createState = (program: t.Program): JavaScriptSemanticAnalysisState => {
const root: JavaScriptSemanticScopeState = {
scopeId: semanticScopeId("program", program),
@@ -655,21 +565,9 @@ const bindAssignment = (
else if (t.isForOfStatement(node) || t.isForInStatement(node))
for (const pattern of t.isVariableDeclaration(node.left)
? node.left.declarations.map(({ id }) => id)
: [node.left]) {
: [node.left])
for (const identifier of assignedPatternIdentifiers(pattern))
addAssignment(identifier, node, scope, state);
if (t.isForOfStatement(node))
// Keep the loop value unknown, but preserve references yielded into
// each bound slot so writes/escapes can reach their iterable origins.
bindPattern({
pattern,
initializer: node,
scope,
state,
mutable: true,
referenceOnly: true,
});
}
};
const assignedPatternIdentifiers = (
@@ -882,11 +780,7 @@ const bindPattern = (input: BindPatternInput): void => {
}
if (t.isIdentifier(pattern)) {
if (referenceOnly) {
const binding = resolveSemanticBindingFromScope(
scope,
pattern.name,
state,
);
const binding = scope.bindings.get(pattern.name);
if (binding !== undefined && initializer !== null)
binding.referenceInitializers.push({
node: initializer,
@@ -970,7 +864,14 @@ const bindPattern = (input: BindPatternInput): void => {
referenceOnly: true,
copyKind: "object-rest",
copyProjectionOffset: projection.length,
copyExcludedKeys: semanticObjectPatternKeys(pattern),
copyExcludedKeys: pattern.properties.flatMap((property) => {
if (t.isRestElement(property)) return [];
const key = semanticStaticPropertyKey(
property.key,
property.computed,
);
return key === null ? [] : [key];
}),
});
} else {
const name = semanticStaticPropertyKey(property.key, property.computed);
@@ -56,9 +56,6 @@ const evidenceContextId = (context: ApplicationGraphEvidenceContext): string =>
/** Graph-scoped owner for canonical provenance shared by semantic facts. */
export class JavaScriptSemanticEvidenceContextRegistry {
readonly #contexts = new Map<string, JavaScriptSemanticEvidenceContext>();
// Facts share a few contexts, so key them by canonical text and hash each
// distinct context once instead of on every fact.
readonly #idsByCanonicalContext = new Map<string, string>();
/** Intern normalized non-location provenance at fact construction. */
intern(
@@ -66,21 +63,18 @@ export class JavaScriptSemanticEvidenceContextRegistry {
): JavaScriptSemanticEvidenceReference {
const normalized = normalizeEvidence(evidence);
const { location, ...context } = normalized;
const canonicalContext = canonicalJson(
context,
"JavaScript semantic graph",
);
const internedId = this.#idsByCanonicalContext.get(canonicalContext);
if (internedId !== undefined) return { context_id: internedId, location };
const contextId = evidenceContextId(context);
// This canonical context is new, so a row already stored under its
// digest holds different provenance.
if (this.#contexts.has(contextId))
const row = { ...context, context_id: contextId };
const existing = this.#contexts.get(contextId);
if (
existing !== undefined &&
canonicalJson(existing, "JavaScript semantic graph") !==
canonicalJson(row, "JavaScript semantic graph")
)
throw new TypeError(
"JavaScript semantic evidence context digest collision",
);
this.#contexts.set(contextId, { ...context, context_id: contextId });
this.#idsByCanonicalContext.set(canonicalContext, contextId);
this.#contexts.set(contextId, row);
return { context_id: contextId, location };
}
@@ -107,7 +101,6 @@ export class JavaScriptSemanticEvidenceContextRegistry {
/** Release interned contexts after the owning graph is finalized. */
clear(): void {
this.#contexts.clear();
this.#idsByCanonicalContext.clear();
}
}
@@ -1,7 +1,6 @@
import { expect, it } from "vitest";
import { z } from "zod";
import { digestCanonicalValue } from "../canonicalDigest.js";
import type { ApplicationGraphEvidence } from "./javascriptApplicationEvidenceSchemas.js";
import {
JAVASCRIPT_SEMANTIC_NODE_KINDS,
@@ -616,42 +615,3 @@ it.each(invalidGraphCases)(
expect(issues(() => complete(steps))).toEqual(expected);
},
);
it("interns each distinct evidence context once under its content digest", () => {
const registry = new JavaScriptSemanticEvidenceContextRegistry();
const elsewhere: ApplicationGraphEvidence = {
...evidence(),
location: {
available: true,
value: {
kind: "source-range",
source: "other.js",
start: { line: 7, column: 2 },
end: { line: 7, column: 9 },
},
},
};
const first = registry.intern(evidence());
const repeated = registry.intern(elsewhere);
const reordered = registry.intern({
...evidence("inferred"),
limitations: [
"Static reachability does not prove runtime execution.",
"Static reachability does not prove runtime execution.",
],
});
const inferred = registry.intern(evidence("inferred"));
const { location: _location, ...context } = evidence();
expect(repeated.context_id).toBe(first.context_id);
expect(repeated.location).toEqual(elsewhere.location);
expect(reordered.context_id).toBe(inferred.context_id);
expect(inferred.context_id).not.toBe(first.context_id);
expect(first.context_id).toBe(
`jsrg_evidence_${digestCanonicalValue(context, "test")}`,
);
expect(registry.contexts.map(({ context_id }) => context_id)).toEqual(
[first.context_id, inferred.context_id].toSorted(),
);
expect(registry.resolve(repeated)).toEqual(elsewhere);
});
@@ -1,23 +1,15 @@
import * as t from "@babel/types";
import {
semanticObjectPatternKeys,
semanticStaticPropertyKey,
} from "./javascriptAstValues.js";
import {
semanticArrayIndex,
type JavaScriptSemanticPropertyPath,
} from "./javascriptSemanticPropertyPaths.js";
import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
import { semanticStaticPropertyKey } from "./javascriptAstValues.js";
import { semanticArrayIndex } from "./javascriptSemanticPropertyPaths.js";
interface IterationSourceValue {
readonly node: t.Node;
readonly projection: JavaScriptSemanticPropertyPath;
readonly projection: readonly (string | number | null)[];
}
/** An iterator receiver or property reference store, with default guards. */
/** An implicit iterator receiver, with guards for destructuring defaults. */
export interface SemanticIterationSource extends IterationSourceValue {
readonly kind: "iteration" | "reference-store";
readonly mutation: t.Node;
readonly fallbackSources?: readonly IterationSourceValue[];
readonly requiredSources?: readonly IterationSourceValue[];
@@ -38,10 +30,8 @@ const add = (
source: IterationSourceValue,
mutation: t.Node,
guards: SourceGuards,
kind: SemanticIterationSource["kind"] = "iteration",
): void => {
sources.push({
kind,
node: source.node,
projection: source.projection,
mutation,
@@ -76,27 +66,6 @@ const visit = (
guards: SourceGuards = {},
): void => {
const { sources, mutation } = context;
if (t.isMemberExpression(pattern)) {
// Property stores have no lexical alias binding. Preserve the selected
// reference's escape rather than claiming later writes cannot reach it.
if (source !== undefined)
add(
sources,
source.restStartIndex === undefined
? source
: {
node: source.node,
projection: [
...source.projection,
{ excludedKeys: [], startIndex: source.restStartIndex },
],
},
mutation,
guards,
"reference-store",
);
return;
}
if (t.isTSParameterProperty(pattern)) {
visit(context, pattern.parameter, source, guards);
return;
@@ -125,29 +94,8 @@ const visit = (
}
if (t.isObjectPattern(pattern)) {
for (const property of pattern.properties) {
if (t.isRestElement(property)) {
// A rest copy owns its scalar slots; only selected child references
// escape when that fresh copy is stored through a member target.
visit(
context,
property.argument,
source === undefined
? undefined
: {
node: source.node,
projection: [
...source.projection,
{
excludedKeys: [
...new Set(semanticObjectPatternKeys(pattern)),
].sort(compareUnicodeCodePoints),
},
],
},
guards,
);
continue;
}
// Object rest creates a fresh object and has no iterable binding pattern.
if (t.isRestElement(property)) continue;
visit(
context,
property.value,
@@ -186,7 +134,7 @@ const visit = (
visit(context, pattern.argument, undefined, guards);
};
/** Describe iterator receivers and property reference stores without evaluation. */
/** Describe implicit iteration sources without deciding their runtime values. */
export const semanticIterationSources = (
node: t.Node,
parent: t.Node | null,
@@ -195,10 +143,7 @@ export const semanticIterationSources = (
const context = { sources, mutation: node };
if (t.isVariableDeclarator(node) && node.init != null)
visit(context, node.id, { node: node.init, projection: [] });
else if (
t.isAssignmentExpression(node) &&
["=", "||=", "&&=", "??="].includes(node.operator)
)
else if (t.isAssignmentExpression(node) && node.operator === "=")
visit(context, node.left, { node: node.right, projection: [] });
else if (t.isForOfStatement(node)) {
add(sources, { node: node.right, projection: [] }, node, {});
@@ -18,7 +18,6 @@ import { semanticIterationSources } from "./javascriptSemanticIterationSources.j
import { semanticMutationInitializers } from "./javascriptSemanticMutationInitializers.js";
import {
semanticArrayIndex,
SemanticPropertyPathCoverage,
semanticPropertyPathKeyMatches,
type JavaScriptSemanticPropertyPath,
} from "./javascriptSemanticPropertyPaths.js";
@@ -43,10 +42,8 @@ export const collectSemanticMemberMutations = (
if (parent !== null) parents.set(node, parent);
},
});
const normalizePath = observableMutationPaths(state);
const recordedEffects = new Set<string>();
const selections = new Map<string, readonly string[]>();
const coveringPaths = new Map<string, SemanticPropertyPathCoverage>();
let arrayIterationUnknown = false;
const pendingReferences: {
readonly initializer: JavaScriptSemanticBindingState["referenceInitializers"][number];
@@ -96,10 +93,8 @@ export const collectSemanticMemberMutations = (
originAt: t.Node | undefined = mutation,
): void => {
const pending = [{ node, path, bindings, originAt }];
// Visit shallower alias traversals first so their exact prefixes suppress
// longer loop paths before those paths enumerate every assignment order.
for (let cursor = 0; cursor < pending.length; cursor++) {
const current = pending[cursor];
while (pending.length > 0) {
const current = pending.pop();
if (current === undefined) break;
const expression = unwrapJavaScriptExpression(current.node).node;
if (t.isIdentifier(expression)) {
@@ -131,13 +126,10 @@ export const collectSemanticMemberMutations = (
if (current.bindings.has(lifetime)) continue;
// Union compatible copied-slot selections at each binding. Carrying every
// combination of exclusions through branch diamonds is exponential.
// Preserve paths that can change observed slots or Array prototype
// effects; canonicalize suffixes that cannot distinguish lattice facts.
let path = normalizePath(binding.bindingId, current.path, effect);
if (path === null) continue;
const selectorIndex = path.findIndex(
const selectorIndex = current.path.findIndex(
(key) => key !== null && typeof key === "object",
);
let path = current.path;
const selector = path[selectorIndex];
if (selector !== null && typeof selector === "object") {
const selectionIdentity = JSON.stringify([
@@ -165,25 +157,6 @@ export const collectSemanticMemberMutations = (
index === selectorIndex ? { ...selector, excludedKeys } : key,
);
}
// Keep an antichain of exact paths. Widening distinct member keys would
// also invalidate unrelated siblings that never reach this effect.
const pathState = JSON.stringify([
binding.bindingId,
effect,
// Short writes stop at owned literal slots, whereas longer writes
// reach shared children. Only equal-depth writes cover one another.
effect === "write" ? path.length : null,
mutation?.start,
mutation?.end,
current.originAt?.start,
current.originAt?.end,
]);
let covering = coveringPaths.get(pathState);
if (covering === undefined) {
covering = new SemanticPropertyPathCoverage();
coveringPaths.set(pathState, covering);
}
if (!covering.retain(path)) continue;
const identity = JSON.stringify([
binding.bindingId,
effect,
@@ -194,11 +167,8 @@ export const collectSemanticMemberMutations = (
current.originAt?.end,
]);
if (recordedEffects.has(identity)) continue;
// A recorded write invalidates the binding to a non-primitive value, so
// only its first write can target a primitive. Re-evaluating it would
// replay every earlier write and make repeated writes superlinear.
const value =
effect === "write" && binding.mutatedPaths.length === 0
effect === "write"
? evaluateSemanticBinding(binding, state)
: undefined;
const primitiveWrite =
@@ -327,7 +297,7 @@ export const collectSemanticMemberMutations = (
bindings: new Set(),
effect: "escape",
originAt: source.mutation,
iterationOnly: source.kind === "iteration",
iterationOnly: true,
});
if (t.isAssignmentExpression(node)) markTarget(node.left, node);
else if (t.isUpdateExpression(node)) markTarget(node.argument, node);
@@ -372,19 +342,6 @@ export const collectSemanticMemberMutations = (
const { initializer } = reference;
if (referenceSourcesInactive(initializer, state))
pendingReferences.push(reference);
else if (t.isForOfStatement(initializer.node))
deferIterable({
node: initializer.node.right,
projection: [],
path: [null, ...reference.path],
fallbackPath: [],
bindings: reference.bindings,
effect: reference.effect,
originAt: initializer.node,
...(reference.mutation === undefined
? {}
: { mutation: reference.mutation }),
});
else if (
initializer.copyKind === "array-rest" &&
initializer.copyProjectionOffset !== undefined
@@ -587,241 +544,6 @@ const referencedValues = (
return [];
};
type MutationObservableOrigin =
| t.ObjectExpression
| t.ArrayExpression
| "ambient"
| "array"
| "array-prototype"
| "opaque";
interface MutationObserverNode {
readonly origins: Set<MutationObservableOrigin>;
readonly sources: Set<MutationObserverNode>;
readonly dependents: Set<MutationObserverNode>;
readonly projections: Map<string, MutationObserverNode>;
readonly read?: {
readonly source: MutationObserverNode;
readonly key: PropertyPath[number];
};
}
const observableMutationPaths = (
state: JavaScriptSemanticAnalysisState,
): ((
bindingId: string,
path: PropertyPath,
effect: ValueEffect,
) => PropertyPath | null) => {
const bindingNodes = new Map<string, MutationObserverNode>();
const expressions = new WeakMap<t.Node, MutationObserverNode>();
const pendingExpressions: {
readonly expression: t.Node;
readonly observer: MutationObserverNode;
}[] = [];
const pending: MutationObserverNode[] = [];
const queued = new Set<MutationObserverNode>();
const schedule = (node: MutationObserverNode): void => {
if (queued.has(node)) return;
queued.add(node);
pending.push(node);
};
const create = (
read?: MutationObserverNode["read"],
): MutationObserverNode => ({
origins: new Set(),
sources: new Set(),
dependents: new Set(),
projections: new Map(),
...(read === undefined ? {} : { read }),
});
const link = (
source: MutationObserverNode,
target: MutationObserverNode,
): void => {
if (target.sources.has(source)) return;
target.sources.add(source);
source.dependents.add(target);
schedule(target);
};
const expressionNode = (node: t.Node): MutationObserverNode => {
const expression = unwrapJavaScriptExpression(node).node;
const existing = expressions.get(expression);
if (existing !== undefined) return existing;
const observer = create();
expressions.set(expression, observer);
pendingExpressions.push({ expression, observer });
return observer;
};
const project = (
source: MutationObserverNode,
path: PropertyPath,
): MutationObserverNode => {
let observer = source;
for (const key of path) {
const identity = JSON.stringify(
key === null || typeof key === "object" ? key : String(key),
);
let selected = observer.projections.get(identity);
if (selected === undefined) {
selected = create({ source: observer, key });
observer.projections.set(identity, selected);
observer.dependents.add(selected);
schedule(selected);
}
observer = selected;
}
return observer;
};
for (const binding of state.bindingsById.values())
bindingNodes.set(binding.bindingId, create());
for (const binding of state.bindingsById.values()) {
const observer = bindingNodes.get(binding.bindingId);
if (observer === undefined) continue;
for (const initializer of binding.initializers)
link(
project(expressionNode(initializer.node), initializer.projection),
observer,
);
// Preserve exact paths where copies, defaults or iteration need their
// reference-specific projections; an opaque origin prevents pruning.
if (binding.referenceInitializers.length > 0)
observer.origins.add("opaque");
for (const initializer of binding.referenceInitializers) {
for (const source of [
initializer,
...(initializer.requiredSources ?? []),
...(initializer.fallbackSources ?? []),
])
link(expressionNode(source.node), observer);
}
}
const settle = (): void => {
while (pendingExpressions.length > 0 || pending.length > 0) {
while (pendingExpressions.length > 0) {
const selected = pendingExpressions.pop();
if (selected === undefined) break;
const { expression, observer } = selected;
if (t.isObjectExpression(expression) || t.isArrayExpression(expression))
observer.origins.add(expression);
else if (t.isIdentifier(expression)) {
const binding = resolveSemanticBindingState(
state,
expression,
expression.name,
);
if (binding !== undefined) {
const origin = bindingNodes.get(binding.bindingId);
if (origin !== undefined) link(origin, observer);
} else if (expression.name === "Array") observer.origins.add("array");
else if (
["globalThis", "global", "window", "self"].includes(expression.name)
)
observer.origins.add("ambient");
} else if (
t.isMemberExpression(expression) ||
t.isOptionalMemberExpression(expression)
)
link(
project(expressionNode(expression.object), [
semanticStaticPropertyKey(
expression.property,
expression.computed,
),
]),
observer,
);
else
for (const reference of referencedValues(expression, [], "escape"))
link(
project(expressionNode(reference.node), reference.path),
observer,
);
schedule(observer);
}
const observer = pending.pop();
if (observer === undefined) continue;
queued.delete(observer);
let changed = false;
const add = (origin: MutationObservableOrigin): void => {
if (observer.origins.has(origin)) return;
observer.origins.add(origin);
changed = true;
};
for (const source of observer.sources)
for (const origin of source.origins) add(origin);
if (observer.read !== undefined) {
const { source, key } = observer.read;
if (key === null || typeof key === "object") {
// Dynamic keys and copied-slot masks retain their original paths.
// Enumerating their exclusion combinations would recreate the growth
// that the mutation collector's selection union already prevents.
if (source.origins.size > 0) add("opaque");
} else
for (const origin of source.origins) {
if (typeof origin === "string") {
if (origin === "array-prototype" || origin === "opaque")
add(origin);
else if (
origin === "ambient" &&
semanticPropertyPathKeyMatches(key, "Array")
)
add("array");
else if (
origin === "array" &&
semanticPropertyPathKeyMatches(key, "prototype")
)
add("array-prototype");
} else {
for (const reference of referencedValues(
origin,
[key],
"escape",
)) {
link(
project(expressionNode(reference.node), reference.path),
observer,
);
if (reference.iterableFallbackPath !== undefined)
link(
project(
expressionNode(reference.node),
reference.iterableFallbackPath,
),
observer,
);
}
}
}
}
if (changed)
for (const dependent of observer.dependents) schedule(dependent);
}
};
settle();
return (bindingId, path, effect) => {
let observer = bindingNodes.get(bindingId);
if (observer === undefined) return path;
if (path.some((key) => key !== null && typeof key === "object"))
return path;
if (effect === "escape") {
observer = project(observer, path);
settle();
return observer.origins.size === 0 ? null : path;
}
if (observer.origins.size === 0) return path.length === 0 ? path : [null];
for (let offset = 0; offset < path.length; offset++) {
const key = path[offset];
if (key === undefined) break;
observer = project(observer, [key]);
settle();
if (observer.origins.size === 0 && offset + 1 < path.length)
return [...path.slice(0, offset + 1), null];
}
return path;
};
};
const objectReferencedValues = (
node: t.ObjectExpression,
path: PropertyPath,
@@ -94,25 +94,19 @@ const directMutationPosition = (
!t.isTaggedTemplateExpression(node)
)
return null;
let current: t.Node = node;
while (true) {
const parent = parents.get(current);
if (
parent === undefined ||
t.isFunction(parent) ||
(t.isLogicalExpression(parent) && parent.right === current) ||
(t.isConditionalExpression(parent) && parent.test !== current)
)
return null;
if (t.isStatement(parent)) return statementPosition(parent, parents);
if (t.isVariableDeclarator(parent)) {
const declaration = parents.get(parent);
let current = parents.get(node);
while (current !== undefined) {
if (t.isFunction(current)) return null;
if (t.isStatement(current)) return statementPosition(current, parents);
if (t.isVariableDeclarator(current)) {
const declaration = parents.get(current);
return t.isVariableDeclaration(declaration)
? statementPosition(declaration, parents, parent)
? statementPosition(declaration, parents, current)
: null;
}
current = parent;
current = parents.get(current);
}
return null;
};
const referenceInitializerPosition = (
@@ -33,89 +33,49 @@ export const invalidateSemanticEscapedPath = (
: { ...value, items: value.items.map(invalidateChild) };
};
/**
* Invalidate only slots that explicit mutations can affect, as if applying each
* path in order. Paths are grouped per slot so that many writes to one object
* cost one pass over its slots rather than one pass per write.
*/
export const invalidateSemanticMutationPaths = (
/** Invalidate only slots that an explicit mutation can affect. */
export const invalidateSemanticMutationPath = (
value: JavaScriptSemanticValue,
paths: readonly JavaScriptSemanticPropertyPath[],
path: JavaScriptSemanticPropertyPath,
): JavaScriptSemanticValue => {
if (paths.length === 0) return value;
// An unknown key or a replaced value leaves nothing that a later path can
// restore, wherever it occurs in the sequence.
if (
(value.status !== "object" && value.status !== "array") ||
paths.some(([key]) => key === undefined || key === null)
)
return {
status: "unknown",
reason: "This property may have been mutated.",
};
// Writing an array length discards every item known before it.
const reset =
value.status === "array"
? paths.findLastIndex(([key]) => key === "length")
: -1;
const base: typeof value =
reset < 0
? value
: { status: "array", items: [], unknownItems: true, omittedItems: null };
const slots = base.status === "object" ? base.properties : base.items;
const slotsByName = new Map<string, number[]>();
slots.forEach((property, index) => {
const indexes = slotsByName.get(property.name) ?? [];
indexes.push(index);
slotsByName.set(property.name, indexes);
});
const slotPaths = new Map<number, JavaScriptSemanticPropertyPath[]>();
const replacedSlots = new Set<number>();
const select = (index: number, remaining: JavaScriptSemanticPropertyPath) => {
const selected = slotPaths.get(index) ?? [];
selected.push(remaining);
slotPaths.set(index, selected);
if (remaining.length === 0) replacedSlots.add(index);
const [key, ...remaining] = path;
const unknown: JavaScriptSemanticValue = {
status: "unknown",
reason: "This property may have been mutated.",
};
// An added slot is already unknown, so later paths that select it are inert.
const added = new Set<string>();
for (const [key, ...remaining] of paths.slice(reset + 1)) {
if (key === undefined || key === null) continue;
if (typeof key === "object") {
slots.forEach((property, index) => {
if (semanticPropertyPathKeyMatches(key, property.name))
select(index, remaining);
});
continue;
}
const indexes = slotsByName.get(String(key));
if (indexes === undefined) added.add(String(key));
else for (const index of indexes) select(index, remaining);
}
const properties: JavaScriptSemanticProperty[] = slots.map(
(property, index) => {
const selected = slotPaths.get(index);
return selected === undefined
? property
: {
if (key === undefined || key === null) return unknown;
if (value.status === "object" || value.status === "array") {
if (value.status === "array" && key === "length")
return {
status: "array",
items: [],
unknownItems: true,
omittedItems: null,
};
const slots = value.status === "object" ? value.properties : value.items;
const observed = slots.some((property) =>
semanticPropertyPathKeyMatches(key, property.name),
);
const properties = slots.map((property) =>
semanticPropertyPathKeyMatches(key, property.name)
? {
...property,
...(replacedSlots.has(index)
...(remaining.length === 0
? { presence: "unknown-coverage" as const }
: {}),
value: invalidateSemanticMutationPaths(property.value, selected),
};
},
);
for (const name of added)
properties.push({
name,
value: {
status: "unknown",
reason: "This property may have been mutated.",
},
presence: "unknown-coverage",
});
return base.status === "object"
? { ...base, properties }
: { ...base, items: properties };
value: invalidateSemanticMutationPath(property.value, remaining),
}
: property,
);
if (!observed && typeof key !== "object")
properties.push({
name: String(key),
value: unknown,
presence: "unknown-coverage",
});
return value.status === "object"
? { ...value, properties }
: { ...value, items: properties };
}
return unknown;
};
@@ -35,89 +35,3 @@ export const semanticPropertyPathKeyMatches = (
const index = semanticArrayIndex(name);
return index !== null && index >= key.startIndex;
};
interface PropertyPathCoverageNode {
terminal: boolean;
readonly exact: Map<string, PropertyPathCoverageNode>;
readonly selectors: Map<
string,
{
readonly key: JavaScriptSemanticPropertyPath[number];
readonly node: PropertyPathCoverageNode;
}
>;
}
const coverageNode = (): PropertyPathCoverageNode => ({
terminal: false,
exact: new Map(),
selectors: new Map(),
});
/** Index exact effect prefixes without widening distinct member paths. */
export class SemanticPropertyPathCoverage {
readonly #root = coverageNode();
/** Retain a path only when no previously retained prefix covers it. */
retain(path: JavaScriptSemanticPropertyPath): boolean {
const pending = [{ node: this.#root, offset: 0 }];
while (pending.length > 0) {
const current = pending.pop();
if (current === undefined) break;
if (current.node.terminal) return false;
const key = path[current.offset];
if (key === undefined) continue;
if (key !== null && typeof key !== "object") {
const exact = current.node.exact.get(String(key));
if (exact !== undefined)
pending.push({ node: exact, offset: current.offset + 1 });
}
for (const selected of current.node.selectors.values()) {
if (keyCovers(selected.key, key))
pending.push({ node: selected.node, offset: current.offset + 1 });
}
}
let node = this.#root;
for (const key of path) {
if (key !== null && typeof key !== "object") {
const name = String(key);
let child = node.exact.get(name);
if (child === undefined) {
child = coverageNode();
node.exact.set(name, child);
}
node = child;
} else {
const identity = JSON.stringify(key);
let selected = node.selectors.get(identity);
if (selected === undefined) {
selected = { key, node: coverageNode() };
node.selectors.set(identity, selected);
}
node = selected.node;
}
}
node.terminal = true;
node.exact.clear();
node.selectors.clear();
return true;
}
}
const keyCovers = (
covering: JavaScriptSemanticPropertyPath[number],
key: JavaScriptSemanticPropertyPath[number],
): boolean => {
if (covering === null) return true;
if (key === null) return false;
if (typeof key !== "object")
return typeof covering === "object"
? semanticPropertyPathKeyMatches(covering, String(key))
: String(covering) === String(key);
return (
typeof covering === "object" &&
covering.excludedKeys.every((name) => key.excludedKeys.includes(name)) &&
(covering.startIndex === undefined ||
(key.startIndex !== undefined && covering.startIndex <= key.startIndex))
);
};
@@ -131,17 +131,15 @@ const directReturnExpressions = (callable: t.Node): ReturnExpression[] => {
return [{ node: callable.body, location: range(callable.body) }];
if (!t.isFunction(callable)) return [];
const output: ReturnExpression[] = [];
const pending = childNodes(callable.body).reverse();
while (pending.length > 0) {
const node = pending.pop();
if (node === undefined || t.isFunction(node) || t.isClass(node)) continue;
const visit = (node: t.Node): void => {
if (t.isFunction(node) || t.isClass(node)) return;
if (t.isReturnStatement(node)) {
output.push({ node: node.argument ?? null, location: range(node) });
continue;
return;
}
// Reverse children so the explicit stack retains source-tree order.
for (const child of childNodes(node).reverse()) pending.push(child);
}
for (const child of childNodes(node)) visit(child);
};
for (const child of childNodes(callable.body)) visit(child);
return output;
};
@@ -10,7 +10,7 @@ import { semanticSlotAtPath } from "./javascriptSemanticSlots.js";
import {
invalidateSemanticEscapedPath,
invalidateSemanticMutationPaths,
invalidateSemanticMutationPath,
} from "./javascriptSemanticMutationValues.js";
import type { JavaScriptBindingProvenance } from "./javascriptSemanticIr.js";
@@ -211,11 +211,12 @@ const evaluateBinding = (
if (isPrimitive(captured)) value = captured;
}
}
let projected = invalidateSemanticMutationPaths(value, binding.mutatedPaths);
for (const escape of binding.escapedPaths) {
// Escapes only invalidate object references; other lattice values survive.
if (projected.status !== "object" && projected.status !== "array") break;
if (
const mutated = binding.mutatedPaths.reduce(
invalidateSemanticMutationPath,
value,
);
const projected = binding.escapedPaths.reduce(
(value, escape) =>
context.capturePoint !== undefined &&
semanticEscapeFollowsCapture(
binding,
@@ -223,10 +224,10 @@ const evaluateBinding = (
escape.node,
context.state.parentsByNode,
)
)
continue;
projected = invalidateSemanticEscapedPath(projected, escape.path);
}
? value
: invalidateSemanticEscapedPath(value, escape.path),
mutated,
);
// Point-local memoization also bounds failed speculative captures. Only a
// primitive result is allowed back into ordinary value evaluation.
if (
@@ -25,11 +25,9 @@ import { finalizeLocatedFindings } from "./javascriptStaticAnalysisFindings.js";
import {
createJavaScriptAnalysisAccumulator,
type JavaScriptAnalysisAccumulator as AnalysisAccumulator,
type JavaScriptFindingContext,
} from "./javascriptStaticAnalysisState.js";
import type { JavaScriptStaticAnalysis } from "./javascriptStaticAnalysisTypes.js";
import {
classifyParsedJavaScriptElectronBindings,
classifyParsedJavaScriptOpenReceivers,
type JavaScriptOpenReceiverFact,
} from "./javascriptSemanticAnalysis.js";
@@ -59,24 +57,21 @@ export const analyzeParsedJavaScriptStaticSource = (
openReceiverFacts?: ReadonlyMap<number, JavaScriptOpenReceiverFact>,
): JavaScriptStaticAnalysis => {
const accumulator = createJavaScriptAnalysisAccumulator(source.length);
traverseStaticSource(file, {
source,
accumulator,
...(openReceiverFacts === undefined ? {} : { openReceiverFacts }),
electronBindings: classifyParsedJavaScriptElectronBindings(file),
});
traverseStaticSource(source, file, accumulator, openReceiverFacts);
addSourceMapDirectives(source, file.comments ?? [], accumulator);
return finalizeStaticAnalysis(source, file, accumulator);
};
const traverseStaticSource = (
source: string,
file: ParsedJavaScriptSource,
findings: JavaScriptFindingContext,
accumulator: AnalysisAccumulator,
openReceiverFacts?: ReadonlyMap<number, JavaScriptOpenReceiverFact>,
): void => {
traverseJavaScriptAst(file, {
enter: (node) => {
findings.accumulator.visitedNodes += 1;
inspectNode(node, findings);
accumulator.visitedNodes += 1;
inspectNode(source, node, accumulator, openReceiverFacts);
return undefined;
},
});
@@ -161,10 +156,16 @@ const finalizeStaticAnalysis = (
};
const inspectNode = (
source: string,
node: t.Node,
findings: JavaScriptFindingContext,
accumulator: AnalysisAccumulator,
openReceiverFacts?: ReadonlyMap<number, JavaScriptOpenReceiverFact>,
): void => {
const { source, accumulator } = findings;
const findings = {
source,
accumulator,
...(openReceiverFacts === undefined ? {} : { openReceiverFacts }),
};
inspectElectronStaticNode(node, findings);
if (t.isCallExpression(node)) {
inspectBundlerRegistration(source, node, accumulator);
@@ -306,32 +306,13 @@ export const addSourceMapDirectives = (
end === null
)
continue;
const commentSource = source.slice(start, end);
if (
!(commentSource.startsWith("//") || commentSource.startsWith("/*")) ||
!(commentSource[2] === "#" || commentSource[2] === "@")
)
continue;
// Single-class searches avoid regex backtracking over unbounded whitespace
// or inline URLs, which can exhaust the engine stack on two-byte strings.
const nameOffset = commentSource.slice(3).search(/\S/u);
if (nameOffset === -1) continue;
const nameStart = 3 + nameOffset;
if (!commentSource.startsWith("sourceMappingURL", nameStart)) continue;
const nameEnd = nameStart + "sourceMappingURL".length;
const equalsOffset = commentSource.slice(nameEnd).search(/\S/u);
if (equalsOffset === -1) continue;
const equalsStart = nameEnd + equalsOffset;
if (!commentSource.startsWith("=", equalsStart)) continue;
const urlOffset = commentSource.slice(equalsStart + 1).search(/\S/u);
if (urlOffset === -1) continue;
const declaredStart = equalsStart + 1 + urlOffset;
const terminator = commentSource.slice(declaredStart).search(/[\s*]/u);
const declaredEnd =
terminator === -1 ? commentSource.length : declaredStart + terminator;
if (declaredEnd === declaredStart) continue;
const declared = commentSource.slice(declaredStart, declaredEnd);
const location = rangeForOffsets(source, start, start + declaredEnd);
const match = /^(?:\/\/|\/\*)[#@]\s*sourceMappingURL\s*=\s*([^\s*]+)/u.exec(
source.slice(start, end),
);
if (match === null) continue;
const declared = match[1];
if (declared === undefined || declared.length === 0) continue;
const location = rangeForOffsets(source, start, start + match[0].length);
addFindingOnce(accumulator, `source-map\0${declared}`, () =>
accumulator.sourceMaps.push({ declared_url: declared, location }),
);

Some files were not shown because too many files have changed in this diff Show More