Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9d4dce1263 | ||
|
|
1e034a80fe | ||
|
|
e28762a5c7 | ||
|
|
c1f1570a0c | ||
|
|
bcc3ad6014 | ||
|
|
258ecd361b |
@@ -2,7 +2,7 @@
|
||||
name: reverse-engineer-anything
|
||||
description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.
|
||||
metadata:
|
||||
version: "37"
|
||||
version: "36"
|
||||
---
|
||||
|
||||
# REA
|
||||
@@ -25,18 +25,57 @@ a skill installed from repository main may describe capabilities absent from
|
||||
an older npm release. Keep complete inline Evidence when that server does not
|
||||
advertise retained references.
|
||||
|
||||
If tools are absent, a call is blocked by the client, or a required provider is
|
||||
unavailable, read [connection and recovery](references/connection-and-recovery.md).
|
||||
Diagnose the observed blocker, continue useful supported work, and resume the
|
||||
original task after repair. Client approval, shell PATH, registration and engine
|
||||
availability have different remedies; avoid repeating setup for all of them.
|
||||
When tools are absent or registration is stale:
|
||||
|
||||
Prefer an existing `rea` CLI. When none is installed and REA package acquisition
|
||||
is within the authorized scope, use `npx -y rea-agents@latest` in its place.
|
||||
Configuration repair starts with a read-only setup plan; show paths, backups and
|
||||
skill replacement before applying authorized changes. Preserve existing approval
|
||||
and target-execution boundaries. REA setup never installs or upgrades unrelated
|
||||
software, including Node.js, npm, Homebrew, Java, Ghidra, IDA, JADX, Binwalk or Unblob.
|
||||
1. Diagnose without changing files. For Codex, run
|
||||
`npx -y rea-agents@latest doctor --client codex --json`. Substitute the current
|
||||
supported client: `claude_code`, `claude_desktop`, `codex`, `cursor`,
|
||||
`gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`, `copilot_cli`,
|
||||
`commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`, `hermes`, or `grok_bot`. `grok_bot` has no local
|
||||
file registration: doctor reports the Grok Bot chat step, and setup does not
|
||||
write `mcp.json`. If the client is unknown, use `doctor --json` and
|
||||
inspect its registration results before choosing a setup scope.
|
||||
2. Distinguish the reason. Missing, malformed, or stale registration needs a
|
||||
scoped configuration repair. An aligned registration with no tools in the
|
||||
active session needs a restart/reconnection; doctor cannot prove that the
|
||||
current agent has connected. A missing provider affects only tasks requiring
|
||||
that provider: static JavaScript inspection needs neither Hopper nor Ghidra,
|
||||
and Android inspection has separate bring-your-own JADX/Java prerequisites.
|
||||
3. For a configuration repair, prepare the read-only plan:
|
||||
`npx -y rea-agents@latest setup --client codex --dry-run --json`.
|
||||
Use the current client's ID, show its exact proposed paths, backups, and
|
||||
changes, and obtain approval before setup writes configuration or installs
|
||||
Hopper. Setup normally installs the matching bundled skill too; include that
|
||||
replacement in the reviewed plan. After approval, apply the same scope with
|
||||
`npx -y rea-agents@latest setup --client codex --yes`. Add `--install-hopper`
|
||||
only if that separate installation was needed and explicitly approved.
|
||||
4. Restart/reconnect the affected agent. Verify that REA tools actually appear
|
||||
in the session, then resume the original investigation. If they remain
|
||||
absent, inspect the client's MCP launch error rather than repeating setup.
|
||||
|
||||
REA setup never installs or upgrades Node.js, npm, Homebrew, Java, Ghidra, IDA,
|
||||
JADX, Binwalk, or Unblob. Use existing prerequisites; do not install unrelated
|
||||
software to repair
|
||||
MCP registration. For an unsupported client, use manual stdio registration
|
||||
with a version-pinned `rea-agents` package or continue through the CLI.
|
||||
|
||||
A concrete CLI fallback for an operator-supplied JavaScript tree or ASAR is:
|
||||
|
||||
```bash
|
||||
npx -y rea-agents@latest analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
|
||||
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
|
||||
npx -y rea-agents@latest inspect-analysis-view app-view.json
|
||||
```
|
||||
|
||||
No MCP registration or native engine is required. The CLI returns the complete
|
||||
Evidence record directly, and a real application can produce hundreds of
|
||||
megabytes, so save it to a file rather than reading it into context. Start from
|
||||
the summary view, then request a module page
|
||||
(`{"kind": "page", "collection": "modules", "offset": 0, "limit": 32}`) or one
|
||||
module the same way, or pass the saved Evidence inline to
|
||||
`trace-application-feature`. Read graph, limitations, and unknowns with the
|
||||
same care as an MCP result. This fallback does not establish that MCP is
|
||||
configured. Native CLI tasks still require their selected engine.
|
||||
|
||||
## Route the target first
|
||||
|
||||
@@ -101,11 +140,9 @@ after opening; IDA does not supply `binary_overview`. Consult session availabili
|
||||
when composing broader workflows. Modern direct callers and unsupported dossier
|
||||
facets remain unknown; live IDA results are not replayed from snapshots.
|
||||
|
||||
If the user says only "my app" without a name, path, or target already selected
|
||||
for this task, ask which app before searching for artifacts. Paths in unrelated
|
||||
saved reports and nearby test fixtures do not select the user's target. Resolve
|
||||
an actual human-readable app name to one clear installed artifact when possible;
|
||||
ask when matches are ambiguous. Never choose an example app on the user's behalf.
|
||||
If the app is missing, ask which app to inspect. Resolve a human-readable app
|
||||
name to one clear installed artifact when possible; ask only when matches are
|
||||
ambiguous. Never choose an example app on the user's behalf.
|
||||
|
||||
In a target-free session, use `open_binary` to bind any archive/package or
|
||||
native target whose analysis tool operates on the active target. Do not call an
|
||||
@@ -120,11 +157,7 @@ For a JavaScript application or ELF layout whose size is unknown or large,
|
||||
call `analyze_javascript_application` or `inspect_binary_layout` with
|
||||
`"detail": "summary"`, then inspect the retained `parent_evidence_id` with
|
||||
`inspect_analysis_view`.
|
||||
Page module identities and select the returned `node_id` before requesting an
|
||||
item; a single file path can represent several graph roles. Do not repeat an
|
||||
identical analysis or rejected call without a changed input or resolved blocker.
|
||||
Recheck session state when a lifecycle change makes that status useful.
|
||||
Make a focused follow-up only when the
|
||||
Do not repeat an identical tool call. Make a focused follow-up only when the
|
||||
returned result leaves a specific question unanswered.
|
||||
|
||||
If MCP reports `resource_constraint` with `details.resource: "transport"`, call
|
||||
@@ -135,14 +168,6 @@ session to a selected path. The analysis remains complete in the ledger; avoid
|
||||
repeating it merely to request the same oversized response. Complete CLI JSON
|
||||
output also streams without a single MCP frame.
|
||||
|
||||
A client can also truncate a successful tool result without a REA transport
|
||||
error. Treat that preview as incomplete; use the retained parent ID for smaller
|
||||
views or export when the full record is needed. Do not parse the preview as
|
||||
canonical JSON or repeat the producer. For JavaScript, read the
|
||||
[module-view and trace recipes](references/javascript-applications.md) before
|
||||
drilling into a large graph; a broad bidirectional trace can expand unrelated
|
||||
containment relationships even after summary analysis.
|
||||
|
||||
Every conclusion must distinguish observations, inferences, and unknowns. Cite
|
||||
Evidence IDs, preserve limitations and incomplete coverage, and never imply
|
||||
that static analysis observed execution. Runtime requests execute the declared
|
||||
@@ -150,9 +175,7 @@ target and lifecycle; do not broaden the target or action beyond those fields.
|
||||
|
||||
Within the user's requested investigation, call available analysis tools
|
||||
directly. REA does not require permission grants or per-call approval flags.
|
||||
The agent client can independently require approval. Follow the declared request
|
||||
scope and the host's actual access requirements; use the recovery guide for an
|
||||
observed client-policy rejection.
|
||||
Follow the declared request scope and the host's actual access requirements.
|
||||
|
||||
## Plan broader investigations
|
||||
|
||||
|
||||
@@ -1,127 +0,0 @@
|
||||
# Connection and task recovery
|
||||
|
||||
Read this guide only for an observed connection, command, approval or provider
|
||||
blocker. Keep the user's target and question in view. Explain what prevented the
|
||||
operation and the next useful step; after repair, continue the investigation.
|
||||
|
||||
Commands below use an installed `rea`. If none is available, use the package
|
||||
invocation in `SKILL.md` when acquiring REA is authorized. Follow the connected
|
||||
server's schemas and the installed CLI's help rather than assuming repository
|
||||
main and an older release expose the same arguments.
|
||||
|
||||
## Choose the remedy from the observed failure
|
||||
|
||||
| Observation | Meaning and next step |
|
||||
| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| No REA tools in the active session | Inspect registration with `rea doctor --client codex --json`. A skill installation alone does not register MCP. |
|
||||
| Missing, malformed or stale registration | Prepare a scoped setup plan; apply only authorized changes with backups. |
|
||||
| Aligned registration, but no connected tools | Restart/reconnect the client. Doctor checks registration; it cannot establish the current agent's connection. Inspect the client's launch error if reconnection fails. |
|
||||
| `MCP tool call requires approval, but approval policy is never` | Codex rejected the call before REA executed it. Do not install an engine or repeat setup to solve this client-policy decision. See the approval section below. |
|
||||
| REA returns `provider_unavailable` or an unavailable tool entry | Read that operation's provider diagnostics or `binary_session.result.tool_availability`. Resolve only the prerequisite needed for the task; a complete tool catalog does not imply every operation is available. |
|
||||
| `rea` is missing in a child shell, or Node is incompatible | Inspect the actual shell PATH and launcher/runtime. A prefix installation may be absent from a login shell's PATH; this does not establish a broken MCP registration. |
|
||||
|
||||
If the client cannot launch its shell/sandbox helper or read the installed skill,
|
||||
inspect its launch diagnostic. For example, Codex can refuse helper aliases when
|
||||
CODEX_HOME is under `/tmp`; use an authorized profile outside the OS temporary
|
||||
directory. REA setup cannot repair the client's executor. Preserve the sandbox
|
||||
instead of silently disabling it to make an analysis command work.
|
||||
|
||||
Static JavaScript and ordinary ZIP inventory need neither Hopper nor Ghidra.
|
||||
HAR inspection is offline and does not require a live browser. Native
|
||||
decompilation, Android analysis and firmware extraction have their own
|
||||
bring-your-own prerequisites. Use supported inventory/static evidence when it
|
||||
answers part of the question, and name the unavailable evidence separately.
|
||||
Do not substitute another target. Label observations from ordinary shell tools
|
||||
as such; only actual REA output supplies REA Evidence.
|
||||
|
||||
## Repair registration when needed
|
||||
|
||||
For Codex:
|
||||
|
||||
```bash
|
||||
rea doctor --client codex --json
|
||||
rea setup --client codex --dry-run --json
|
||||
```
|
||||
|
||||
Use the current supported client's ID: `claude_code`, `claude_desktop`, `codex`,
|
||||
`cursor`, `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`,
|
||||
`copilot_cli`, `commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`,
|
||||
`hermes`, or `grok_bot`. If unknown, inspect `doctor --json` before choosing a
|
||||
scope. `grok_bot` uses a chat registration step rather than local `mcp.json`.
|
||||
For unsupported clients, use version-pinned manual stdio registration or CLI.
|
||||
|
||||
Show the plan's exact paths, backups, changes and bundled-skill replacement.
|
||||
Obtain approval for writes when that concrete scope is not already authorized,
|
||||
then apply the same scope with `rea setup --client codex --yes`. Add
|
||||
`--install-hopper` only when separately needed and explicitly approved. Do not
|
||||
install or upgrade unrelated prerequisites. Restart/reconnect, verify tools
|
||||
actually appear, and resume the original task. If they remain absent, inspect
|
||||
the launch failure rather than looping through setup.
|
||||
|
||||
## Codex tool approval is separate from REA readiness
|
||||
|
||||
REA tools have no permission-grant handshake. Codex can still require approval
|
||||
because an operation creates owned resources, launches a helper or has cleanup
|
||||
effects. An advertised tool or successful doctor result does not override that
|
||||
client policy. `workspace-write` alone does not resolve an MCP approval
|
||||
rejection, and `approval_policy = "never"` does not mean every MCP tool is
|
||||
preapproved.
|
||||
|
||||
When interactive approval is available, use the client's approval flow for the
|
||||
requested operation. In an unattended run, continue through an authorized CLI
|
||||
workflow if the execution environment permits it, or report the blocked
|
||||
operation and the approval needed to proceed. Do not retry the same rejected
|
||||
call, silently broaden approvals, or rewrite MCP effects as read-only.
|
||||
|
||||
For an operator who explicitly wants to preapprove named operations in a
|
||||
disposable Codex profile, current Codex supports per-tool configuration. For
|
||||
example, opening and closing an analysis session can be scoped independently:
|
||||
|
||||
```toml
|
||||
[mcp_servers.rea.tools.open_binary]
|
||||
approval_mode = "approve"
|
||||
|
||||
[mcp_servers.rea.tools.close_binary]
|
||||
approval_mode = "approve"
|
||||
```
|
||||
|
||||
Use the actual server ID and only the operations whose effects were approved;
|
||||
this example does not approve inspection, extraction or runtime execution.
|
||||
Show the proposed config edit and preserve unrelated entries and backups before
|
||||
writing. Managed policy can constrain client settings. Consult the
|
||||
[Codex configuration reference](https://learn.chatgpt.com/docs/config-file/config-reference)
|
||||
for the installed client's supported settings. This is an optional recovery
|
||||
choice, not a prerequisite for every REA task.
|
||||
|
||||
## Continue through an installed CLI
|
||||
|
||||
Check the command environment only if the launcher fails:
|
||||
|
||||
```bash
|
||||
command -v rea
|
||||
command -v node
|
||||
node --version
|
||||
rea --version
|
||||
```
|
||||
|
||||
Use the public launcher at its known installation path with a supported existing
|
||||
Node on PATH when the child shell loses a prefix installation. Avoid internal
|
||||
`dist/cli.js` entrypoints or global toolchain changes. MCP may have its own valid
|
||||
launcher even when a shell cannot resolve `rea`.
|
||||
|
||||
For a user-supplied JavaScript tree or ASAR, select output files in the authorized
|
||||
workspace and save complete Evidence before reading a view:
|
||||
|
||||
```bash
|
||||
rea analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
|
||||
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
|
||||
rea inspect-analysis-view app-view.json
|
||||
```
|
||||
|
||||
These commands do not register MCP or install an analysis engine. CLI JavaScript
|
||||
JSON is complete and can be very large; redirect it rather than loading it into
|
||||
context. Use an existing JSON processor if `jq` is absent, without installing
|
||||
one just for this recipe. Inspect a module page or item using the same inline
|
||||
source and the [JavaScript recipes](javascript-applications.md). Preserve graph
|
||||
coverage, limitations and unknowns. Native CLI analysis still requires its
|
||||
selected engine, and all CLI operations retain the host's execution permissions.
|
||||
@@ -1,106 +1,20 @@
|
||||
# JavaScript and Electron application artifacts
|
||||
|
||||
Use `analyze_javascript_application` directly on the operator-supplied ASAR or
|
||||
extracted tree. When size is unknown or large and the connected schema advertises
|
||||
`detail`, request `detail: "summary"`. This returns an `inspect_analysis_view`
|
||||
Evidence record with a distinct ID and a `normalized_result.parent_evidence_id`
|
||||
for the complete retained application analysis. Source-map contents are part of
|
||||
that static analysis. Read findings, coverage, limitations and unknowns, then
|
||||
follow a specific unanswered question. Older servers without `detail` return
|
||||
the complete application Evidence inline when it fits.
|
||||
|
||||
When the connected schema advertises `max_heap_mb` and `analysis_timeout_ms`,
|
||||
these select the isolated source analyzer's old-space heap and per-source
|
||||
deadline. Defaults are 1024 MiB and 300000 ms. They do not change the MCP
|
||||
process heap. Resource-limited analyses preserve completed static facts and
|
||||
report partial semantic coverage with specific unknown scopes; this does not
|
||||
mean the source is malformed. Inspect those limits before retrying a source.
|
||||
When expanded relationships exceed the parent's available memory, inspect the
|
||||
file's modules item view for retained `static_analysis` and
|
||||
`semantic_module_analysis`. Their facts remain available even when graph links
|
||||
are unknown. Cumulative result exhaustion can stop analysis before every file
|
||||
is examined; the inventory and remaining unknown scopes distinguish that case.
|
||||
Timeout or cancellation after source analysis begins retains partial Evidence
|
||||
in the same session. A returned `error.details.partial_observation` reference
|
||||
can be supplied directly to `inspect_analysis_view`. If the SDK cancels locally, retrieve its recorded ID from
|
||||
the Evidence bundle before inspecting the partial analysis.
|
||||
|
||||
## Read a module without repeating analysis
|
||||
|
||||
Use the actual returned parent ID, on the same connection, as `source.evidence_id`.
|
||||
The following IDs are placeholders to replace with returned values. When the
|
||||
module's node ID is not already known, page identities before selecting an item.
|
||||
Pages return only node IDs, paths and roles; choose a useful page size and follow
|
||||
`coverage.next_offset` when another page matters:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": {
|
||||
"kind": "retained-evidence",
|
||||
"evidence_id": "ev_ACTUAL_PARENT_ID"
|
||||
},
|
||||
"view": { "kind": "page", "collection": "modules", "offset": 0, "limit": 16 }
|
||||
}
|
||||
```
|
||||
|
||||
Select the relevant page entry by its exact `node_id` and recorded role:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": {
|
||||
"kind": "retained-evidence",
|
||||
"evidence_id": "ev_ACTUAL_PARENT_ID"
|
||||
},
|
||||
"view": {
|
||||
"kind": "item",
|
||||
"collection": "modules",
|
||||
"selector": { "node_id": "jag_node_ACTUAL_NODE_ID" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Module pages include asset, bundled and source module roles, so one path can
|
||||
identify several entries. Use `selector.path` only when uniqueness is established;
|
||||
the returned node ID avoids needing a complete inventory to prove uniqueness. On ambiguity,
|
||||
select the matching node ID from the page or the error's candidates. Follow
|
||||
`next_offset` when another page matters. An item exposes that module node's
|
||||
recorded observations, identity, hashes, exports and source ranges; it does not
|
||||
include every child syntax observation, related edge or original source file.
|
||||
An absent item fact is not evidence of absence from the whole application.
|
||||
|
||||
## Trace only the remaining question
|
||||
|
||||
Use `trace_application_feature` on the parent application Evidence for one
|
||||
literal node ID, route, string, API, IPC channel, module, or native export.
|
||||
Select `incoming` for relationships leading to the seed, `outgoing` for those
|
||||
leaving it, and `both` when the question needs both. Trace traversal includes
|
||||
containment relationships, so even a narrow seed can expand many nodes; a
|
||||
bidirectional trace can reach unrelated sibling modules. A module item or page
|
||||
is preferable when its observations already answer the question. Do not treat
|
||||
all reached nodes as participants in the feature.
|
||||
|
||||
Reuse retained parent Evidence when the server advertises it, or supply complete
|
||||
Evidence inline. Include Hopper or Ghidra Evidence only when its artifact digest
|
||||
matches exactly.
|
||||
|
||||
## Recover an oversized display
|
||||
extracted tree. The complete result, graph, and Evidence context are returned
|
||||
inline when they fit the MCP frame. Source-map contents are part of the static analysis. Start from its
|
||||
findings, coverage, unknowns, and graph context, then make a focused follow-up
|
||||
only when a specific question remains unanswered.
|
||||
|
||||
An oversized response reports `resource_constraint` and
|
||||
`details.resource: "transport"`. Reuse
|
||||
`details.reported_limits.evidence_reference` with `inspect_analysis_view` for a
|
||||
summary, one module, or a stable page of module identities.
|
||||
summary, one module, or a stable page of module identities. Use
|
||||
`trace_application_feature` once a module, route, or string seed is known.
|
||||
`export_evidence_bundle` writes the complete canonical session to a
|
||||
caller-selected file. These workflows keep the original Evidence and coverage;
|
||||
a broad follow-up can also exceed framing.
|
||||
|
||||
A client can truncate a successful result inside its text envelope without a
|
||||
REA `resource_constraint`. Treat a truncation notice or incomplete JSON as a
|
||||
display limitation; do not claim the full record was examined. Reuse the known
|
||||
parent ID for smaller views, or export complete Evidence to a selected file
|
||||
while the session remains open. If the parent ID was lost from the preview,
|
||||
use the prior analysis summary or explicit ledger entries to recover it; do
|
||||
not invent an ID or rerun analysis just to reproduce the same display.
|
||||
|
||||
BrowserWindow preferences, preload and contextBridge surfaces, IPC
|
||||
registrations, utility processes, and native binding requests are static syntax
|
||||
observations. Only a unique exact literal IPC channel match supports an inferred
|
||||
@@ -108,6 +22,13 @@ pairing. Dynamic or ambiguous channels remain unresolved. A requested `.node`
|
||||
member is not a verified native export. Never claim runtime reachability,
|
||||
registration, defaults, or policy enforcement from static analysis.
|
||||
|
||||
Use `trace_application_feature` on existing application Evidence for one
|
||||
literal node ID, route, string, API, IPC channel, module, or native export.
|
||||
Choose a direction and reuse the returned application Evidence ID when the
|
||||
connected server advertises retained references, or supply complete Evidence
|
||||
inline. Include
|
||||
Hopper or Ghidra Evidence only when its artifact digest matches exactly.
|
||||
|
||||
For version comparison, analyze each version once, then call
|
||||
`compare_application_versions`. Accept only its digest, source-map, structural
|
||||
fingerprint, or non-module semantic matches. Module ordinals and minified names
|
||||
|
||||
@@ -152,7 +152,7 @@ context. PSP/stack/device state remains unmodeled. See [DOS guide](https://githu
|
||||
|
||||
### Function annotations in Ghidra
|
||||
|
||||
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier.
|
||||
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations.
|
||||
|
||||
## Offline PE resource inventory
|
||||
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"_comment": "Test files are intentionally NOT excluded: the gate used to report zero clones because it skipped **/*.test.ts, hiding all test-tree duplication. Current actual is ~2.47% lines across src+ and tests/, down from 3.4% when tests were included. The 2.6% limit leaves modest headroom for ordinary changes; lower it as duplication is removed.",
|
||||
"threshold": 2.6,
|
||||
"_comment": "Test files are intentionally NOT excluded: the gate used to report zero clones because it skipped **/*.test.ts, hiding all test-tree duplication. Current actual is ~2.4% lines across src+ and tests/, down from 3.4% when tests were included. Lower this as duplication is removed.",
|
||||
"threshold": 2.5,
|
||||
"reporters": ["consoleFull"],
|
||||
"ignore": [
|
||||
"node_modules/**",
|
||||
|
||||
@@ -120,8 +120,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
);
|
||||
private static final String[] CAPABILITIES = {
|
||||
"annotate_native_function",
|
||||
"set_address_name",
|
||||
"set_addresses_names",
|
||||
"inspect_native_load_image",
|
||||
"read_bytes",
|
||||
"address_to_file_offset",
|
||||
@@ -376,12 +374,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
case "resolve_native_call_targets" -> resolveNativeCallTargets(request.params);
|
||||
case "xrefs" -> xrefs(request.params);
|
||||
case "analyze_function" -> analyzeFunction(request.params);
|
||||
case "set_address_name", "set_addresses_names" -> {
|
||||
if (!descriptor.transport.equals("unix-socket"))
|
||||
throw new RequestFailure("method_unavailable", "Windows P0 does not admit database mutation");
|
||||
yield request.method.equals("set_address_name")
|
||||
? setAddressName(request.params) : setAddressesNames(request.params);
|
||||
}
|
||||
case "annotate_native_function" -> {
|
||||
if (!descriptor.transport.equals("unix-socket"))
|
||||
throw new RequestFailure("method_unavailable", "Windows P0 does not admit database mutation");
|
||||
@@ -425,8 +417,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
result.addProperty("analysis_timed_out", timedOut);
|
||||
result.add("capabilities", GSON.toJsonTree(
|
||||
java.util.Arrays.stream(CAPABILITIES)
|
||||
.filter(value -> !readOnly || !(value.equals("annotate_native_function")
|
||||
|| value.equals("set_address_name") || value.equals("set_addresses_names")))
|
||||
.filter(value -> !readOnly || !value.equals("annotate_native_function"))
|
||||
.toArray(String[]::new)
|
||||
));
|
||||
result.add("target", target);
|
||||
@@ -1036,11 +1027,11 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
for (String field : List.of("name", "comment", "inline_comment")) {
|
||||
if (params.has(field)) validateAnnotationText(requireText(params, field), field);
|
||||
}
|
||||
String leafName = params.has("name")
|
||||
? annotationLeafName(function, requireString(params, "name")) : null;
|
||||
int transaction = currentProgram.startTransaction("REA function annotations");
|
||||
boolean commit = false;
|
||||
try {
|
||||
String leafName = params.has("name")
|
||||
? symbolLeafName(function.getSymbol(), requireString(params, "name")) : null;
|
||||
// Validate edits through Ghidra's own setters, including name rules.
|
||||
// A later invalid name must roll back earlier comment writes.
|
||||
for (String field : List.of("comment", "inline_comment")) {
|
||||
@@ -1089,18 +1080,18 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
}
|
||||
}
|
||||
|
||||
private String symbolLeafName(Symbol symbol, String requested) throws ghidra.util.exception.InvalidInputException {
|
||||
private String annotationLeafName(Function function, String requested) {
|
||||
// Fully qualified readback must be usable as an idempotent rename input.
|
||||
if (requested.equals(symbol.getName(true))) return symbol.getName();
|
||||
String namespace = symbol.getParentNamespace().isGlobal()
|
||||
? "" : symbol.getParentNamespace().getName(true);
|
||||
if (requested.equals(procedureName(function))) return function.getName();
|
||||
String namespace = function.getParentNamespace().isGlobal()
|
||||
? "" : function.getParentNamespace().getName(true);
|
||||
String prefix = namespace + "::";
|
||||
if (!namespace.isEmpty() && requested.startsWith(prefix)) {
|
||||
String leaf = requested.substring(prefix.length());
|
||||
if (!leaf.isEmpty()) return leaf;
|
||||
throw new ghidra.util.exception.InvalidInputException(
|
||||
"Symbol name has an empty leaf within namespace " + namespace + " at " +
|
||||
canonicalAddress(symbol.getAddress()) + ": " + requested);
|
||||
throw new RequestFailure("invalid_function_name",
|
||||
"Function name has an empty leaf within namespace " + namespace + " at " +
|
||||
canonicalAddress(function.getEntryPoint()) + ": " + requested);
|
||||
}
|
||||
return requested;
|
||||
}
|
||||
@@ -1425,68 +1416,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
|
||||
return symbol == null ? JsonNull.INSTANCE : GSON.toJsonTree(symbol.getName(true));
|
||||
}
|
||||
|
||||
// Analyst naming: a function entry renames the function, other addresses get a primary label.
|
||||
private boolean applyAddressName(Address address, String name) throws Exception {
|
||||
if (!currentProgram.getMemory().contains(address))
|
||||
throw new RequestFailure("invalid_request", "Address is not mapped: " + canonicalAddress(address));
|
||||
validateAnnotationText(name, "name");
|
||||
int transaction = currentProgram.startTransaction("REA address name");
|
||||
boolean commit = false;
|
||||
try {
|
||||
Function function = currentProgram.getFunctionManager().getFunctionAt(address);
|
||||
String expected = name;
|
||||
if (function != null) {
|
||||
expected = symbolLeafName(function.getSymbol(), name);
|
||||
function.setName(expected, SourceType.USER_DEFINED);
|
||||
} else {
|
||||
var table = currentProgram.getSymbolTable();
|
||||
Symbol primary = table.getPrimarySymbol(address);
|
||||
if (primary != null && !primary.isDynamic()) {
|
||||
expected = symbolLeafName(primary, name);
|
||||
primary.setName(expected, SourceType.USER_DEFINED);
|
||||
} else {
|
||||
Symbol created = table.createLabel(address, name, SourceType.USER_DEFINED);
|
||||
created.setPrimary();
|
||||
}
|
||||
}
|
||||
invalidateAnalysisCaches();
|
||||
Symbol readback = currentProgram.getSymbolTable().getPrimarySymbol(address);
|
||||
commit = readback != null && expected.equals(readback.getName());
|
||||
return commit;
|
||||
}
|
||||
catch (ghidra.util.exception.InvalidInputException | ghidra.util.exception.DuplicateNameException exception) {
|
||||
return false;
|
||||
}
|
||||
finally {
|
||||
currentProgram.endTransaction(transaction, commit);
|
||||
}
|
||||
}
|
||||
|
||||
private JsonElement setAddressName(JsonObject params) throws Exception {
|
||||
requireKeys(params, Set.of("document", "address", "name"));
|
||||
requireDocument(params);
|
||||
return GSON.toJsonTree(applyAddressName(requireAddress(params, "address"), requireString(params, "name")));
|
||||
}
|
||||
|
||||
private JsonElement setAddressesNames(JsonObject params) throws Exception {
|
||||
requireKeys(params, Set.of("document", "names"));
|
||||
requireDocument(params);
|
||||
if (!params.has("names") || !params.get("names").isJsonObject())
|
||||
throw new RequestFailure("invalid_request", "names must be an object of address -> name");
|
||||
JsonObject result = new JsonObject();
|
||||
for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet()) {
|
||||
boolean ok;
|
||||
try {
|
||||
ok = applyAddressName(parseReaAddress(entry.getKey()), entry.getValue().getAsString());
|
||||
}
|
||||
catch (RequestFailure failure) {
|
||||
ok = false;
|
||||
}
|
||||
result.addProperty(entry.getKey(), ok);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
private JsonArray listDocuments(JsonObject params) {
|
||||
requireKeys(params, Set.of());
|
||||
JsonArray result = new JsonArray();
|
||||
|
||||
@@ -136,12 +136,6 @@ existing analysis families directly, for example
|
||||
- Pull and capture outputs are additive: the destination must not already
|
||||
exist. Re-running a completed pull refuses the existing directory instead
|
||||
of overwriting evidence.
|
||||
- Cancellation during a package pull preserves caller-selected output. The
|
||||
error includes partial observations identifying completed artifacts and the
|
||||
interrupted local path; remaining APKs were not attempted.
|
||||
- Failed package acquisition rollback retains its directory identity and retries
|
||||
cleanup before later work and during shutdown. Cleanup refuses a replacement
|
||||
directory. Terminal shutdown stops admission and drains active operations.
|
||||
- The provider holds no long-lived processes; every invocation is one
|
||||
bounded adb run. The adb server belongs to the caller.
|
||||
|
||||
|
||||
+3
-10
@@ -166,16 +166,9 @@ rea import-reference-source /absolute/path/to/source
|
||||
```
|
||||
|
||||
The import records hashes and metadata for the supplied files, separately
|
||||
from observations of the current app. It honors the source tree's `.gitignore`
|
||||
and applies default exclusions for common generated and dependency paths such
|
||||
as `node_modules/`, `dist/`, and `*.log`. Pattern-based exclusions record the
|
||||
matched pattern and whether it came from project, default, caller, or sensitive
|
||||
path policy. Set `REA_REFERENCE_SECRET_PATTERNS_JSON` to a JSON array of ignore
|
||||
patterns when you want to mark selected paths as sensitive; those patterns are
|
||||
reported separately as configured-secret exclusions.
|
||||
|
||||
Re-import saved graphs whose pattern-based exclusions lack `pattern` before
|
||||
using them in source-to-bundle comparisons.
|
||||
from observations of the current app. File names do not automatically exclude
|
||||
files. Set `REA_REFERENCE_SECRET_PATTERNS_JSON` to a JSON array of ignore
|
||||
patterns when you want to exclude selected paths.
|
||||
|
||||
JavaScript and TypeScript import parsing requires valid UTF-8. Malformed source
|
||||
bytes retain their original hashes and sizes with a decoding diagnostic; REA
|
||||
|
||||
@@ -29,13 +29,6 @@ section names or the record being inside the initial 4 KiB header probe.
|
||||
Program and section declarations must agree; duplicate, conflicting, truncated
|
||||
or out-of-file records are rejected. Stripped section tables and section-only
|
||||
records are supported. Extended table numbering is explicitly unsupported.
|
||||
For carved ELFs whose section table extends beyond the captured file, REA
|
||||
examines every complete captured section header and records the unavailable
|
||||
headers as a limitation. A valid program-header ABI record can still establish
|
||||
the supported declaration. Available section declarations must agree with it;
|
||||
their duplicate, conflicting or unreadable records remain errors. A missing
|
||||
program-header record does not make ABI declarations in uncaptured sections
|
||||
absent or supply a default ABI.
|
||||
The field layout follows
|
||||
[`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h).
|
||||
|
||||
@@ -99,11 +92,6 @@ observations and does not relabel the upstream result as verified.
|
||||
|
||||
CLI/MCP discovery and decompilation, direct calls, target/profile/Evidence
|
||||
identity, snapshot round-trip and final process cleanup remain covered.
|
||||
For both byte orders, the lane also carves away the section table while retaining
|
||||
all program-header payloads. It checks the actual Ghidra language, entry-address
|
||||
decompilation, distinct artifact digest and missing-section limitations through
|
||||
CLI Evidence and an MCP-produced snapshot. Caller-selected Ghidra heap limits
|
||||
are passed to these child workflows.
|
||||
Pseudocode presence is a liveness check, not proof of semantic equivalence.
|
||||
Instruction decoding and reader agreement likewise do not establish runtime,
|
||||
delay-slot execution or complete floating-point semantics. Assertion regressions
|
||||
|
||||
+4
-45
@@ -68,15 +68,7 @@ selects the npm release, and persistent MCP registrations are pinned to the
|
||||
version that performed setup. Installing newer instructions does not update a
|
||||
running server or its registration.
|
||||
|
||||
The npm release checked on **2026-10-10** was **6.3.0**, published from
|
||||
[`74cf9e14`](https://github.com/morluto/rea/commit/74cf9e1401fcd89482fe39c27cc898f5858e9b72),
|
||||
with bundled skill version 34. It does not include automatic Pi or Hermes
|
||||
registration: `setup --client pi` and `setup --client hermes` reject those IDs
|
||||
before writing configuration. Their setup workflows below require a source build
|
||||
or a later release containing those integrations; `@latest` alone cannot select
|
||||
unpublished changes.
|
||||
|
||||
The earlier release checked on **2026-10-07** was **5.0.0** (133 MCP tools), published
|
||||
The release checked on **2026-10-07** was **5.0.0** (133 MCP tools), published
|
||||
from the fixed checkpoint
|
||||
[`b33236ec`](https://github.com/morluto/rea/releases/tag/rea-agents-5.0.0).
|
||||
The public CLI, MCP catalog and target-free session, and isolated update from
|
||||
@@ -150,29 +142,7 @@ listed after the table because its connector is not one of these files:
|
||||
| Pi | `pi` |
|
||||
| Hermes | `hermes` |
|
||||
|
||||
For GitHub Copilot CLI 1.0.95, `gpt-4.1` model metadata can block ordinary
|
||||
chat with REA's full input-schema profile before sending a model request
|
||||
(`compaction_static_context_blocked`; [#1554](https://github.com/morluto/rea/issues/1554)).
|
||||
For the native offline OpenAI-compatible BYOK adapter, full-profile chat and
|
||||
analysis also passed with a distinct `COPILOT_PROVIDER_MODEL_ID` such as
|
||||
`rea-gpt-4.1`, `COPILOT_PROVIDER_WIRE_MODEL=gpt-4.1`, and that distinct ID selected
|
||||
through `copilot --model rea-gpt-4.1`. The local fixture requested
|
||||
`COPILOT_PROVIDER_MAX_PROMPT_TOKENS=1000000`; configure the limit supported by
|
||||
your backend. This keeps every full schema and the actual wire model intact.
|
||||
It does not establish the effective client budget or fix the built-in-ID path.
|
||||
The existing compact profile passed native chat and analysis with the complete
|
||||
tool inventory on Linux using a loopback model fixture. After setup, add
|
||||
`"REA_MCP_INPUT_SCHEMA_PROFILE": "compact"` to the `env` object of the
|
||||
`mcpServers.rea` entry in Copilot's `mcp-config.json`, preserving its other
|
||||
settings, then restart the client. Compact advertisements can reduce nested
|
||||
schema constraints; REA still validates complete canonical inputs. Full-profile
|
||||
`gpt-5.4` metadata is also verified locally. Live model API acceptance and
|
||||
Windows remain unverified; see [native client verification](testing.md).
|
||||
|
||||
For [Pi](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/mcp.md),
|
||||
automatic setup requires a build containing the integration; npm 6.3.0 does not
|
||||
include it. See [released package and main](#released-package-and-main).
|
||||
In a supported build,
|
||||
setup writes a `type: "stdio"` entry under `mcpServers.rea` in the user-level
|
||||
`~/.pi/agent/mcp.json` and installs the bundled skill under `~/.agents/skills`.
|
||||
Use `rea setup --client pi --dry-run --json` to inspect the plan, then
|
||||
@@ -231,10 +201,7 @@ registration. Doctor treats an `enabled: false` entry as active when
|
||||
`enabledServers` lists `rea`, as OMP does, unless `disabledServers` also lists
|
||||
it. Run setup under each profile that should load REA.
|
||||
|
||||
For Hermes, automatic setup also requires a build containing the integration;
|
||||
npm 6.3.0 does not include it. See
|
||||
[released package and main](#released-package-and-main).
|
||||
In a supported build, setup writes a `mcp_servers.rea` entry to `config.yaml`, preserving
|
||||
For Hermes, setup writes a `mcp_servers.rea` entry to `config.yaml`, preserving
|
||||
comments and unrelated settings. It resolves that file from `HERMES_HOME`, else
|
||||
from the platform default Hermes itself uses — `%LOCALAPPDATA%\hermes` on
|
||||
Windows, `~/.hermes` elsewhere — including any `HERMES_DATA_DIR_SUFFIX`. Hermes
|
||||
@@ -367,12 +334,6 @@ Hopper installation or any file write. Declining or pressing Ctrl-C makes no
|
||||
changes. Agent configuration writes preserve unrelated entries and comments,
|
||||
create backups, use atomic replacement, and verify their result. Setup and
|
||||
uninstall retain an existing `.rea.backup` rather than replacing the first snapshot.
|
||||
Setup also retains valid environment overrides in REA's existing server entry,
|
||||
such as `REA_MCP_INPUT_SCHEMA_PROFILE` and `REA_MCP_MAX_RESPONSE_BYTES`.
|
||||
Detected provider settings replace only matching environment keys.
|
||||
Setup and doctor report a malformed REA environment map or invalid runtime
|
||||
setting before setup writes files. Repair the named setting in the client
|
||||
configuration; setup does not silently discard the rest of the map.
|
||||
|
||||
After setup, REA reports which agents, analysis tools, and workflow files passed
|
||||
its final checks. Restart any agent named in the completion message, then begin
|
||||
@@ -637,10 +598,8 @@ dossiers, instructions, recovered data types, measured load mappings, loaded
|
||||
memory bytes, and observed file offsets. Independent load-image attestation
|
||||
supports DOS MZ and explicitly selected COM; PE returns its measurements with that limitation.
|
||||
On Linux and macOS, `annotate_native_function` also edits a function name and/or
|
||||
entry comments atomically and returns refreshed analysis, and `set_address_name`
|
||||
and `set_addresses_names` name a function entry or any other mapped address
|
||||
(creating or renaming its primary label) with per-address success results. These
|
||||
session metadata edits leave executable bytes unchanged and are discarded on close. GUI controls
|
||||
entry comments atomically and returns refreshed analysis. These session metadata
|
||||
edits leave executable bytes unchanged and are discarded on close. GUI controls
|
||||
require Hopper; Windows P0 remains read-only.
|
||||
|
||||
Windows P0 admits native x86 and x86-64 PE applications and DLLs on fixed local NTFS volumes.
|
||||
|
||||
@@ -89,9 +89,9 @@ queries use per-file range indexes, while call sites use exact-range lookups.
|
||||
Existing node budgets and coverage reporting remain in effect.
|
||||
|
||||
Progress identifies the source file being parsed and projected. Cancellation is
|
||||
checked between files and throughout result transfer. The owned analyzer can
|
||||
be stopped while its synchronous parser or semantic extractor is active;
|
||||
the CLI/MCP process remains available for control requests.
|
||||
checked between files, with event-loop yields that allow notifications and
|
||||
completed file-local allocations to be released. A single file's synchronous
|
||||
parser and semantic extraction still need to finish before that boundary.
|
||||
|
||||
Graph and Evidence identifiers hash canonical JSON incrementally, without
|
||||
assembling a single string for the whole graph. The canonical bytes and existing
|
||||
@@ -113,46 +113,6 @@ smaller view, for example `--format json --filter-output
|
||||
evidence_id,normalized_result.statistics`. Streaming output does not bound the
|
||||
memory needed to construct the analysis graph itself.
|
||||
|
||||
JavaScript source parsing, static analysis and semantic projection run in one
|
||||
owned analysis process, reused sequentially across files. Target code is never
|
||||
executed. The analyzer defaults to 1024 MiB of V8 old-space and a 300000 ms
|
||||
deadline per source, including startup and fact transfer. Select
|
||||
`max_heap_mb` and `analysis_timeout_ms` through MCP, or `--max-heap-mb` and
|
||||
`--analysis-timeout-ms` on the dedicated CLI command. These choices control the
|
||||
analyzer; the caller's `NODE_OPTIONS` still controls the CLI/MCP process heap.
|
||||
V8 old-space limits do not impose a process RSS limit.
|
||||
|
||||
A source that exhausts the analyzer heap does not terminate the CLI/MCP
|
||||
process. Completed static facts are retained, other files can still be analyzed,
|
||||
and unexamined semantic relationships remain explicit unknowns. The parent
|
||||
also checks transfer expansion and accumulated results against its remaining
|
||||
heap before accepting semantic facts. A resource-limited result has partial
|
||||
coverage, the observed limits and an `analyze-javascript-source` unknown scope
|
||||
with the specific failure, including captured diagnostics when available.
|
||||
Resource failures do not increment syntax parse failures. There is no fixed
|
||||
source-length cutoff.
|
||||
|
||||
Static observations also need memory when expanded into nodes, relationships
|
||||
and Evidence. If that expansion does not fit, the file node retains
|
||||
`static_analysis` and any admitted `semantic_module_analysis` observations,
|
||||
including their exact values and source locations. Use the public modules item
|
||||
view to read them. The graph marks those unexpanded relationships as unknown.
|
||||
If cumulative result capacity is exhausted, analysis stops; inventoried but
|
||||
unexamined files remain present with explicit unknown scopes.
|
||||
|
||||
A timeout stops the remaining analysis and returns typed failure details with
|
||||
completed partial Evidence. MCP failure details return
|
||||
`partial_observation: { kind: "retained-evidence", evidence_id: "..." }`
|
||||
after recording succeeds, so `inspect_analysis_view` can open that analysis
|
||||
directly without serializing the entire partial graph into the error response.
|
||||
CLI failure output retains the complete partial Evidence.
|
||||
Cancellation after source analysis begins likewise
|
||||
retains completed facts; an SDK client may reject its cancelled request before
|
||||
receiving that reply. On the same MCP connection, inspect the Evidence bundle
|
||||
and use the retained analysis ID with `inspect_analysis_view`. Owned analyzer
|
||||
cleanup is verified before its private files are released. Cleanup uncertainty
|
||||
is reported separately and retains ownership for retry.
|
||||
|
||||
CLI workflows parse larger JSON input files incrementally from one verified
|
||||
regular-file handle, without constructing a document-sized string. Smaller
|
||||
files retain native JSON parsing. Both paths require strict JSON and valid
|
||||
|
||||
@@ -397,12 +397,6 @@ unpaired Unicode surrogate; a rejection identifies the field and UTF-16 index
|
||||
and leaves every annotation unchanged. CRLF, supplementary Unicode characters,
|
||||
and combining characters are preserved. Windows P0 remains read-only. Ghidra has no
|
||||
GUI authority, and REA never falls back automatically to Hopper.
|
||||
`set_address_name` and `set_addresses_names` give Ghidra sessions the same
|
||||
compact naming results as Hopper: a local function entry renames that function,
|
||||
and any other mapped address, such as a global, receives a new or renamed primary
|
||||
label. A batch applies each address separately and reports one success boolean
|
||||
per address. Use them to name many globals and helpers cheaply; use
|
||||
`annotate_native_function` when you also want comments and a refreshed dossier.
|
||||
Ghidra name edits preserve the existing namespace. Supply either a leaf name
|
||||
such as `renamed` or a fully qualified name in that namespace, such as
|
||||
`alpha::renamed`. The returned qualified name can be reused as an idempotent
|
||||
|
||||
@@ -114,12 +114,6 @@ from launch, must be ordered, and must fall within `timeout_ms`. Inputs marked
|
||||
Environment overrides are recorded in the scenario commitment; inherited
|
||||
values are not copied into Evidence.
|
||||
|
||||
Terminal dimensions must satisfy `columns × (rows + scrollback) ≤ 1,000,000`.
|
||||
Every scheduled resize uses the same selected scrollback and must fit this
|
||||
working-buffer budget. REA rejects oversized scenarios before launching the
|
||||
command; it never clamps the requested dimensions. This limit is independent
|
||||
of `limits.output_bytes`, which bounds retained terminal observations.
|
||||
|
||||
The capture contains raw PTY output chunks and rendered terminal states,
|
||||
interaction dispatch outcomes, exit reason, sampled process-tree observations,
|
||||
and settlement status. When filesystem paths are selected, REA records
|
||||
|
||||
+10
-89
@@ -99,10 +99,9 @@ result that its client offloads, and Pi exercises default codemode execution.
|
||||
Use `REA_VERIFY_RUNTIME_ROOT` to select a production-only installed REA package.
|
||||
Receipts retain client versions, result digests, host coverage and owned-process
|
||||
cleanup. These POSIX lanes use a deterministic loopback model, so they do not
|
||||
prove live model-provider or native Windows compatibility. Qwen, Pi and Hermes
|
||||
bind both `HOME` and `USERPROFILE` to the disposable account. Qwen and Pi discover
|
||||
the shared personal skill through their default search paths without an explicit
|
||||
skill setting; Hermes discovers it through the selected `HERMES_HOME`.
|
||||
prove live model-provider or native Windows compatibility. Qwen and Pi use
|
||||
caller-configured skill directories to isolate the fixture account from the OS
|
||||
home; this does not prove their default home-directory discovery.
|
||||
Set `REA_VERIFY_HERMES_STICKY_PROFILE=1` to exercise native Hermes selection of
|
||||
a named sticky profile with `HERMES_HOME` still pointing to its root.
|
||||
|
||||
@@ -160,39 +159,11 @@ Use `-- chat` for ordinary
|
||||
chat or `REA_VERIFY_RUNTIME_ROOT` for a production-only installed package.
|
||||
The native offline BYOK adapter uses a loopback OpenAI completions/SSE fixture
|
||||
and `gpt-5.4` model metadata; model inference and token usage are synthetic.
|
||||
Set `REA_VERIFY_COPILOT_MODEL` to examine another client model ID and
|
||||
`REA_VERIFY_COPILOT_WIRE_MODEL` for its backend model name (defaults to the client
|
||||
ID). Every actual HTTP request's model is checked and recorded separately.
|
||||
Both `HOME` and `USERPROFILE` point to the disposable account; native `skill add`
|
||||
still selects the installed skill directory explicitly.
|
||||
|
||||
The same native client passed full-profile chat and analysis with the actual
|
||||
wire model `gpt-4.1` under a distinct BYOK model ID:
|
||||
|
||||
```bash
|
||||
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- chat
|
||||
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- call
|
||||
```
|
||||
|
||||
These runs retain full canonical input/output schemas, all tools, installed-skill
|
||||
loading and complete Evidence readback. They use the fixture's requested
|
||||
one-million-token BYOK prompt setting; effective capacity and live API acceptance
|
||||
remain unknown. With the built-in `gpt-4.1` model ID, the verified client still
|
||||
blocks the full input-schema profile before HTTP with
|
||||
Set `REA_VERIFY_COPILOT_MODEL` to examine another model configuration. In the
|
||||
verified client, `gpt-4.1` blocks the complete catalog before HTTP with
|
||||
`compaction_static_context_blocked`, even when requesting a larger BYOK prompt
|
||||
capacity. The effective capacity is unknown. The default verifier preserves
|
||||
the complete catalog and full schemas. To verify the existing compact profile:
|
||||
|
||||
```bash
|
||||
REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- chat
|
||||
REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- call
|
||||
```
|
||||
|
||||
Both native workflows passed with the complete tool inventory. The verifier selects the
|
||||
profile only in its disposable Copilot registration and records it in the
|
||||
receipt. Compact advertisements can reduce nested validation structure; the
|
||||
server still validates complete canonical inputs. This is a tested alternate
|
||||
workflow, not a fix for the full-profile client/model limit tracked in #1554.
|
||||
capacity. The effective capacity is unknown; this lane does not establish a
|
||||
fix for that client/model limit. It preserves the complete catalog and schemas.
|
||||
Live model APIs, native Windows and default OS-home skill discovery remain
|
||||
unverified.
|
||||
|
||||
@@ -261,9 +232,8 @@ unverified; skill discovery uses an explicit isolated custom directory.
|
||||
verified with `@deepseek-ai/dsh@0.2.0-rc.2`) and Git. Set
|
||||
`REA_VERIFY_DEEPSEEK_COMMAND` to select its executable. This optional POSIX lane
|
||||
installs the skill through `rea setup --skill`, writes the native Cordis MCP
|
||||
patch into an isolated `DSH_HOME`, and binds `HOME` and `USERPROFILE` to the
|
||||
disposable account. The native default `~/.agents/skills` search path discovers
|
||||
the installed skill without a `DSH_AGENTS_HOME` override. A separate Git root prevents inherited project skills from
|
||||
patch into an isolated `DSH_HOME`, and uses `DSH_AGENTS_HOME` for native personal
|
||||
skill discovery. A separate Git root prevents inherited project skills from
|
||||
masking a missing installation. The default `call` mode checks skill loading,
|
||||
complete catalog discovery, the actual forwarded input schemas and regexes,
|
||||
and full JavaScript Evidence for a Unicode path. Use `-- chat` for ordinary
|
||||
@@ -1083,39 +1053,6 @@ real local Codex CLI with:
|
||||
npm run verify:agent
|
||||
```
|
||||
|
||||
The lane runs scenarios sequentially in standalone Codex (`--no-daemon`) with a
|
||||
disposable HOME and CODEX_HOME under `~/.cache/rea-agent-evaluations/`. Set
|
||||
`REA_AGENT_EVAL_ROOT` to choose another parent directory outside the OS temporary
|
||||
directory: current Codex refuses helper aliases under `/tmp`, preventing shell
|
||||
commands and skill reads even if MCP works. Each owned run directory is removed
|
||||
after verification unless fixture retention is requested. The lane
|
||||
installs the packaged skill and MCP registration through public `rea setup`,
|
||||
rather than copying a skill into a project that also inherits the caller's
|
||||
skills and configuration. Only authentication is copied, when present, and it
|
||||
is removed even with `REA_AGENT_EVAL_KEEP_FIXTURES=true`. No user configuration,
|
||||
rules, plugins or history is copied; shell snapshots and additional agents are
|
||||
disabled. Existing API-key environment authentication remains available.
|
||||
The native missing-provider scenario preapproves only `open_binary` and
|
||||
`close_binary` in the disposable client so it reaches REA's actual provider
|
||||
boundary instead of stopping at Codex approval. This does not validate a real
|
||||
native engine or authorize arbitrary runtime execution.
|
||||
|
||||
Select a smaller run with `REA_AGENT_EVAL_SCENARIOS`, a comma-separated list of
|
||||
scenario IDs. `asar-zh` exercises the packaged desktop rubric from a Chinese
|
||||
request. `javascript-module-view` exercises summary analysis, module paging and
|
||||
item inspection in a tree with unrelated modules; its routing and text checks
|
||||
remain heuristic, so review the retained transcript for selector choice and
|
||||
coverage claims. Set `REA_CODEX_CLI` to an installed Codex executable and
|
||||
`REA_AGENT_EVAL_MODEL` to an account-supported model. Use an external resource
|
||||
limit for the whole process tree when testing on a constrained host; a Node
|
||||
heap limit alone does not bound client and child-process memory.
|
||||
|
||||
`missing-target` leaves desktop fixtures visible without selecting one in the
|
||||
request. It requires zero REA calls and a structured request for an app name or
|
||||
artifact path. Its `expectedFirstTool` is `null`, and
|
||||
`targetClarificationPassed` grades that limited routing outcome independently
|
||||
of analysis prose heuristics. Analyzing a nearby example cannot pass this case.
|
||||
|
||||
The packaged desktop and parser-comparison scenarios now assess a closed set of
|
||||
known fixture claims. Each final answer must be a strict JSON manifest containing
|
||||
exactly the requested claim IDs, values, Evidence IDs, and Evidence authority and
|
||||
@@ -1140,21 +1077,9 @@ navigation-context, and address-context scenarios have no factual rubric and
|
||||
remain `not_assessed`; the managed workflow requires both artifact and member
|
||||
inspection to answer its type and entry-point question.
|
||||
|
||||
The closed factual scenarios request complete producer Evidence on the initial
|
||||
analysis so their source selectors can authenticate all configured facts.
|
||||
Summary/page/item workflows are exercised separately by `javascript-module-view`;
|
||||
its heuristic gate does not establish full-producer factual correctness.
|
||||
|
||||
All scenarios retain routing, workflow, validation, repetition, process-exit,
|
||||
and token-use gates. Configured factual scenarios additionally require a factual
|
||||
pass. Their answer-text heuristics are diagnostic and do not affect the gate.
|
||||
Validation counts include complete text-only REA `invalid_request` envelopes
|
||||
and failed SDK calls whose arguments violate the named current REA contract.
|
||||
Client approval rejections and provider unavailability remain distinct failures;
|
||||
truncated JSON previews are not parsed into invented error codes.
|
||||
Repeated `binary_session` observations separated by a successful `open_binary`
|
||||
or `close_binary` are treated as lifecycle verification. An unchanged retry,
|
||||
including one after a failed lifecycle call, still counts as repetition.
|
||||
Scenarios without a rubric retain the legacy text-heuristic gate.
|
||||
`answerTermCoverageMet` checks case-insensitive substrings,
|
||||
`epistemicCuePresent` checks keywords, and `finalCitesEvidence` checks only an ID's
|
||||
@@ -1162,11 +1087,7 @@ presence. These metrics can still accept fabricated prose and must not be read
|
||||
as factual assessments. Set `REA_AGENT_EVAL_TRANSCRIPT_DIR` to retain complete
|
||||
tool results and final answers for review.
|
||||
|
||||
Report schema version 4 adds negative routing scenarios with nullable
|
||||
`expectedFirstTool` and a `targetClarificationPassed` outcome. Consumers must
|
||||
allow an expected absence of REA calls; this does not make an analysis scenario
|
||||
pass without its required tools. Version 3 changed the top-level
|
||||
`factualCorrectness` from a constant
|
||||
Report schema version 3 changes the top-level `factualCorrectness` from a constant
|
||||
string to an assessment summary with status, scope, and assessed/passed/failed/
|
||||
not-assessed scenario counts. Scenario records include the factual assessment and
|
||||
configured claim IDs. The evaluation scope is
|
||||
|
||||
+4
-7
@@ -85,22 +85,19 @@ if [[ "$version" =~ \+ ]]; then
|
||||
fail "npm cannot install an exact version with build metadata. Pass a version without build metadata, then retry."
|
||||
fi
|
||||
|
||||
install_args=(install --global)
|
||||
prefix_args=()
|
||||
if [[ "$platform" == "Linux" ]]; then
|
||||
install_args+=(--prefix "$HOME/.local")
|
||||
prefix_args=(--prefix "$HOME/.local")
|
||||
install_bin="$HOME/.local/bin/rea"
|
||||
else
|
||||
npm_prefix="$(npm prefix --global)" || fail "the npm global prefix could not be read. Repair the npm configuration, then retry."
|
||||
install_bin="$npm_prefix/bin/rea"
|
||||
fi
|
||||
install_args+=("$PACKAGE@$version")
|
||||
|
||||
printf 'REA install plan\n'
|
||||
printf ' Version: %s\n' "$version"
|
||||
printf ' Runtime: Node.js %s\n' "$node_version"
|
||||
printf ' Command:'
|
||||
printf ' %q' npm "${install_args[@]}"
|
||||
printf '\n'
|
||||
printf ' Command: npm install --global %s@%s\n' "$PACKAGE" "$version"
|
||||
printf ' Binary: %s\n' "$install_bin"
|
||||
printf ' Setup: %s\n' "$([[ "$start_setup" == true ]] && printf 'start when a terminal is available' || printf 'skipped')"
|
||||
|
||||
@@ -110,7 +107,7 @@ if [[ "$dry_run" == true ]]; then
|
||||
fi
|
||||
|
||||
printf 'Installing %s@%s...\n' "$PACKAGE" "$version"
|
||||
npm "${install_args[@]}" || fail "npm could not install REA. Check registry access and npm permissions, then retry."
|
||||
npm install --global ${prefix_args[@]+"${prefix_args[@]}"} "$PACKAGE@$version" || fail "npm could not install REA. Check registry access and npm permissions, then retry."
|
||||
[[ -x "$install_bin" ]] || fail "npm completed without installing the rea command. Check the npm global bin directory and PATH, then retry."
|
||||
installed_version="$("$install_bin" --version 2>/dev/null | tr -d '[:space:]')" || fail "the installed REA version could not be read. Reinstall the requested version, then retry."
|
||||
[[ "$installed_version" == "$version" ]] || fail "installed version $installed_version does not match $version. Reinstall the requested version, then retry."
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
"entry": [
|
||||
"src/main.ts",
|
||||
"src/cli.ts",
|
||||
"src/composition/javascriptAnalysisWorkerProcess.ts",
|
||||
"src/contracts/errorSchemas.ts",
|
||||
"src/evaluation/AgentEvaluationFixtures.ts",
|
||||
"src/application/RealHopperAssertions.ts",
|
||||
|
||||
@@ -30,9 +30,6 @@
|
||||
"access": "public"
|
||||
},
|
||||
"type": "module",
|
||||
"imports": {
|
||||
"#javascript-analysis-worker": "./dist/composition/javascriptAnalysisWorkerProcess.js"
|
||||
},
|
||||
"bin": {
|
||||
"rea": "scripts/rea.mjs",
|
||||
"rea-agents": "scripts/rea.mjs"
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
import { analyzeJavaScriptApplication } from "../../dist/application/javascript/JavaScriptApplicationService.js";
|
||||
import { ArtifactResourceScope } from "../../dist/artifacts/ArtifactResourceScope.js";
|
||||
|
||||
/** Analyze one JavaScript application and close its caller-owned resources. */
|
||||
export const analyzeJavaScriptApplicationWithOwnedScope = async (input) => {
|
||||
const scope = new ArtifactResourceScope();
|
||||
let outcome;
|
||||
try {
|
||||
const result = await analyzeJavaScriptApplication(input, scope);
|
||||
outcome = result.ok
|
||||
? { kind: "completed", value: result.value }
|
||||
: { kind: "failed", cause: result.error };
|
||||
} catch (cause) {
|
||||
outcome = { kind: "failed", cause };
|
||||
}
|
||||
try {
|
||||
await scope.close();
|
||||
} catch (cleanupCause) {
|
||||
if (outcome.kind === "failed")
|
||||
throw new AggregateError(
|
||||
[outcome.cause, cleanupCause],
|
||||
"JavaScript analysis and artifact cleanup both failed",
|
||||
);
|
||||
throw cleanupCause;
|
||||
}
|
||||
if (outcome.kind === "failed") throw outcome.cause;
|
||||
return outcome.value;
|
||||
};
|
||||
@@ -21,6 +21,7 @@ export async function verifyGhidraEntryAliases({
|
||||
new URL("../../tests/conformance/ghidra/entry-aliases.c", import.meta.url),
|
||||
);
|
||||
const path = join(workspace, "entry-aliases");
|
||||
const prefix = process.platform === "darwin" ? "_" : "";
|
||||
try {
|
||||
await exec(compiler, ["-O0", "-g", "-fno-inline", source, "-o", path], {
|
||||
env,
|
||||
@@ -28,8 +29,7 @@ export async function verifyGhidraEntryAliases({
|
||||
});
|
||||
const bytes = await readFile(path);
|
||||
await call("close_binary");
|
||||
const imported = await call("open_binary", { path, provider_id: "ghidra" });
|
||||
const prefix = imported.format === "mach-o" ? "_" : "";
|
||||
await call("open_binary", { path, provider_id: "ghidra" });
|
||||
const names = await call("list_names");
|
||||
const alias = names.find(
|
||||
({ value }) => value === `${prefix}rea_entry_alias`,
|
||||
@@ -46,10 +46,8 @@ export async function verifyGhidraEntryAliases({
|
||||
);
|
||||
assert.ok(canonical, "Fixture alias is not at a function entry");
|
||||
const xrefsSelectors = [];
|
||||
// CLI xrefs keeps bare hexadecimal selectors as addresses. A Mach-O
|
||||
// primary label such as "dead" is tested below as a procedure name instead.
|
||||
for (const value of [
|
||||
`${prefix}rea_alias_target`,
|
||||
canonical.value,
|
||||
alias.value,
|
||||
`${prefix}rea_xrefs_data`,
|
||||
`${prefix}rea_xrefs_unreferenced`,
|
||||
@@ -91,18 +89,6 @@ export async function verifyGhidraEntryAliases({
|
||||
);
|
||||
assert.ok(interior);
|
||||
assert.notEqual(interior.address, alias.address);
|
||||
assert.equal(
|
||||
interior.symbol.type,
|
||||
"label",
|
||||
"Fixture interior control must remain an imported label",
|
||||
);
|
||||
assert.equal(
|
||||
(await call("list_procedures")).some(
|
||||
({ address }) => address === interior.address,
|
||||
),
|
||||
false,
|
||||
"Fixture interior control must not be a Ghidra procedure entry",
|
||||
);
|
||||
const failure = await reject("procedure_address", {
|
||||
procedure: interior.value,
|
||||
});
|
||||
@@ -167,11 +153,7 @@ export async function verifyGhidraEntryAliases({
|
||||
{ env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 },
|
||||
);
|
||||
const evidence = JSON.parse(stdout);
|
||||
assert.deepEqual(
|
||||
evidence.normalized_result,
|
||||
expected,
|
||||
`CLI xrefs mismatch for ${JSON.stringify(selector)}`,
|
||||
);
|
||||
assert.deepEqual(evidence.normalized_result, expected);
|
||||
assert.equal(evidence.parameters.address, selector.at(-1));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -54,14 +54,6 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
);
|
||||
return { namespace, address: procedure.address };
|
||||
});
|
||||
const names = await call("list_names");
|
||||
const globals = ["alpha", "outer::inner"].map((namespace) => {
|
||||
const symbol = names.find(
|
||||
(item) => item.value === `${namespace}::value` && item.symbol.primary,
|
||||
);
|
||||
assert.ok(symbol, `Missing demangled fixture global ${namespace}::value`);
|
||||
return { namespace, address: symbol.address };
|
||||
});
|
||||
const ambiguous = await reject("procedure_address", { procedure: "same" });
|
||||
assert.equal(ambiguous.code, "invalid_request");
|
||||
for (const { address } of entries)
|
||||
@@ -133,16 +125,6 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
});
|
||||
assert.deepEqual(unchanged.annotations, qualified.annotations);
|
||||
}
|
||||
await verifyAddressNames(call, [entries[0], entries[2], ...globals]);
|
||||
for (const { namespace, address } of entries) {
|
||||
assert.equal(
|
||||
await call("set_address_name", {
|
||||
address,
|
||||
name: `${namespace}::qualified`,
|
||||
}),
|
||||
true,
|
||||
);
|
||||
}
|
||||
const { stdout } = await exec(
|
||||
process.execPath,
|
||||
[
|
||||
@@ -177,96 +159,6 @@ export async function verifyGhidraNamespaceAnnotations({
|
||||
}
|
||||
}
|
||||
|
||||
async function verifyAddressNames(call, symbols) {
|
||||
for (const { namespace, address } of symbols) {
|
||||
assert.equal(
|
||||
await call("set_address_name", { address, name: "address_renamed" }),
|
||||
true,
|
||||
);
|
||||
const qualified = `${namespace}::address_qualified`;
|
||||
assert.equal(
|
||||
await call("set_address_name", { address, name: qualified }),
|
||||
true,
|
||||
);
|
||||
assert.equal(await call("address_name", { address }), qualified);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address, name: "other::literal" }),
|
||||
true,
|
||||
);
|
||||
const literal = await call("address_name", { address });
|
||||
assert.equal(literal, `${namespace}::other::literal`);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address, name: literal }),
|
||||
true,
|
||||
);
|
||||
assert.equal(await call("address_name", { address }), literal);
|
||||
assert.equal(
|
||||
await call("set_address_name", { address, name: qualified }),
|
||||
true,
|
||||
);
|
||||
for (let repeat = 0; repeat < 2; repeat++) {
|
||||
const readback = await call("address_name", { address });
|
||||
assert.deepEqual(
|
||||
await call("set_addresses_names", { names: { [address]: readback } }),
|
||||
{ [address]: true },
|
||||
);
|
||||
assert.equal(await call("address_name", { address }), qualified);
|
||||
}
|
||||
assert.equal(
|
||||
await call("set_address_name", { address, name: `${namespace}::` }),
|
||||
false,
|
||||
);
|
||||
const session = await call("binary_session");
|
||||
for (const operation of [
|
||||
"set_address_name",
|
||||
"set_addresses_names",
|
||||
"address_name",
|
||||
"list_names",
|
||||
])
|
||||
assert.equal(
|
||||
session.capabilities.find((item) => item.operation === operation)
|
||||
?.available,
|
||||
true,
|
||||
`An invalid name must not mark ${operation} unavailable`,
|
||||
);
|
||||
assert.equal(await call("address_name", { address }), qualified);
|
||||
}
|
||||
const [valid, invalid] = symbols;
|
||||
assert.deepEqual(
|
||||
await call("set_addresses_names", {
|
||||
names: {
|
||||
[valid.address]: `${valid.namespace}::batch_renamed`,
|
||||
[invalid.address]: `${invalid.namespace}::`,
|
||||
},
|
||||
}),
|
||||
{ [valid.address]: true, [invalid.address]: false },
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: valid.address }),
|
||||
`${valid.namespace}::batch_renamed`,
|
||||
);
|
||||
assert.equal(
|
||||
await call("address_name", { address: invalid.address }),
|
||||
`${invalid.namespace}::address_qualified`,
|
||||
);
|
||||
const names = await call("list_names");
|
||||
for (const { namespace, address } of symbols) {
|
||||
const expected =
|
||||
address === valid.address
|
||||
? `${namespace}::batch_renamed`
|
||||
: `${namespace}::address_qualified`;
|
||||
assert.ok(
|
||||
names.some(
|
||||
(item) =>
|
||||
item.address === address &&
|
||||
item.value === expected &&
|
||||
item.symbol.primary,
|
||||
),
|
||||
`Missing refreshed primary symbol ${expected}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function requireNamespaceDemangler(env) {
|
||||
const inspection = inspectGhidraInstallation({
|
||||
environment: process.env,
|
||||
|
||||
@@ -1,18 +1,9 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import {
|
||||
chmod,
|
||||
copyFile,
|
||||
mkdir,
|
||||
mkdtemp,
|
||||
readFile,
|
||||
rm,
|
||||
writeFile,
|
||||
} from "node:fs/promises";
|
||||
import { homedir } from "node:os";
|
||||
import { delimiter, dirname, join, resolve } from "node:path";
|
||||
import { cp, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { createInterface } from "node:readline";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { z } from "zod";
|
||||
|
||||
import { evaluateCodexEvents } from "../dist/evaluation/CodexAgentEval.js";
|
||||
import {
|
||||
@@ -24,95 +15,32 @@ import {
|
||||
agentEvaluationPassed,
|
||||
summarizeFactualCorrectness,
|
||||
} from "../dist/evaluation/AgentEvaluationReport.js";
|
||||
import { PRODUCT_IDENTITY } from "../dist/identity.js";
|
||||
import { MCP_STARTUP_POLICY } from "../dist/mcpStartupPolicy.js";
|
||||
import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs";
|
||||
|
||||
const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const verifierRun = createVerifierRun();
|
||||
const evaluationRoot = await mkdtemp(join(tmpdir(), "rea-agent-eval-"));
|
||||
const fixtureRoot = join(evaluationRoot, "targets");
|
||||
const skillDestination = join(
|
||||
evaluationRoot,
|
||||
".agents/skills/reverse-engineer-anything",
|
||||
);
|
||||
const timeoutMs = Number(process.env.REA_AGENT_EVAL_TIMEOUT_MS ?? 480_000);
|
||||
const codex = process.env.REA_CODEX_CLI ?? "codex";
|
||||
const optionalModel = process.env.REA_AGENT_EVAL_MODEL;
|
||||
|
||||
if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 10_000)
|
||||
throw new Error("REA_AGENT_EVAL_TIMEOUT_MS must be an integer >= 10000");
|
||||
// Codex refuses helper aliases when CODEX_HOME is under the OS temp directory.
|
||||
// Keep the disposable account in a private cache, then remove the owned run.
|
||||
const evaluationParent = resolve(
|
||||
process.env.REA_AGENT_EVAL_ROOT ??
|
||||
join(homedir(), ".cache", "rea-agent-evaluations"),
|
||||
);
|
||||
await mkdir(evaluationParent, { recursive: true, mode: 0o700 });
|
||||
const evaluationRoot = await mkdtemp(join(evaluationParent, "run-"));
|
||||
const fixtureRoot = join(evaluationRoot, "targets");
|
||||
const account = join(evaluationRoot, "account");
|
||||
const codexHome = join(evaluationRoot, "codex");
|
||||
const authPath = join(codexHome, "auth.json");
|
||||
const skillDestination = join(
|
||||
account,
|
||||
".agents/skills/reverse-engineer-anything",
|
||||
);
|
||||
const evaluationEnvironment = {
|
||||
...Object.fromEntries(
|
||||
Object.entries(process.env).filter(
|
||||
([name]) => !/^(?:REA_|CODEX_|GHIDRA_|HOPPER_|IDA_)/u.test(name),
|
||||
),
|
||||
),
|
||||
HOME: account,
|
||||
USERPROFILE: account,
|
||||
CODEX_HOME: codexHome,
|
||||
XDG_CONFIG_HOME: join(account, ".config"),
|
||||
XDG_DATA_HOME: join(account, ".local/share"),
|
||||
XDG_CACHE_HOME: join(account, ".cache"),
|
||||
REA_PROCESS_RUN_ID: verifierRun.run_id,
|
||||
PATH: `${dirname(process.execPath)}${delimiter}${process.env.PATH ?? ""}`,
|
||||
};
|
||||
const codexVersion = await readCommandVersion(codex);
|
||||
|
||||
try {
|
||||
const codexVersion = await readCommandVersion(codex);
|
||||
await Promise.all([
|
||||
mkdir(fixtureRoot, { recursive: true }),
|
||||
mkdir(account),
|
||||
mkdir(codexHome, { mode: 0o700 }),
|
||||
]);
|
||||
// Reuse authentication only. User config, plugins, rules, history and skills
|
||||
// must not make a fresh installed-skill trial pass accidentally.
|
||||
try {
|
||||
await copyFile(
|
||||
join(process.env.CODEX_HOME ?? join(homedir(), ".codex"), "auth.json"),
|
||||
authPath,
|
||||
);
|
||||
await chmod(authPath, 0o600);
|
||||
} catch (cause) {
|
||||
if (cause?.code !== "ENOENT") throw cause;
|
||||
// Environment/API-key authentication can work without a local auth file.
|
||||
}
|
||||
await writeFile(
|
||||
join(codexHome, "config.toml"),
|
||||
'approval_policy = "never"\nsandbox_mode = "read-only"\nweb_search = "disabled"\n[features]\nmulti_agent = false\nshell_snapshot = false\n[history]\npersistence = "none"\n',
|
||||
{ mode: 0o600 },
|
||||
await mkdir(fixtureRoot, { recursive: true });
|
||||
await cp(
|
||||
join(repositoryRoot, "skills/reverse-engineer-anything"),
|
||||
skillDestination,
|
||||
{ recursive: true },
|
||||
);
|
||||
for (const approval of ["--dry-run", "--yes"]) {
|
||||
await runProcess(
|
||||
process.execPath,
|
||||
[
|
||||
join(repositoryRoot, "scripts/rea.mjs"),
|
||||
"setup",
|
||||
"--client",
|
||||
"codex",
|
||||
approval,
|
||||
"--json",
|
||||
],
|
||||
evaluationRoot,
|
||||
120_000,
|
||||
evaluationEnvironment,
|
||||
);
|
||||
}
|
||||
const installedSkill = await readFile(
|
||||
join(skillDestination, "SKILL.md"),
|
||||
"utf8",
|
||||
);
|
||||
if (!installedSkill.includes(` version: "${PRODUCT_IDENTITY.skillVersion}"`))
|
||||
throw new Error("Setup did not install the current packaged REA skill");
|
||||
const requestedScenarioIds = new Set(
|
||||
(process.env.REA_AGENT_EVAL_SCENARIOS ?? "")
|
||||
.split(",")
|
||||
@@ -129,11 +57,7 @@ try {
|
||||
expectedFirstTool: "open_binary",
|
||||
requiresEvidence: false,
|
||||
requiredAnswerTermGroups: [["unavailable", "could not", "provider"]],
|
||||
// The verifier authorizes these named session operations, not arbitrary
|
||||
// writes or runtime execution. Missing-engine behavior stays observable.
|
||||
approvedTools: ["open_binary", "close_binary"],
|
||||
requiredToolSubsequence: ["open_binary", "close_binary"],
|
||||
prompt: `Explain what static artifact evidence establishes about the shipped native program at ${targets.native}. Do not execute the target or install tools. Clearly state unavailable provider evidence, and close any analysis session you opened before finishing.`,
|
||||
prompt: `Explain what the shipped native program at ${targets.native} does. Base the answer on artifact analysis, and clearly state any facts that remain unavailable.`,
|
||||
},
|
||||
{
|
||||
id: "asar",
|
||||
@@ -146,38 +70,6 @@ try {
|
||||
],
|
||||
prompt: `Explain how the desktop application at ${targets.javascript} exposes APIs to its renderer. Base the answer on the shipped application artifact, and state what remains unknown.`,
|
||||
},
|
||||
{
|
||||
id: "asar-zh",
|
||||
fixtureScenarioId: "asar",
|
||||
expectedFirstTool: "analyze_javascript_application",
|
||||
requiresEvidence: true,
|
||||
requiredAnswerTermGroups: [["profileapi"], ["profile:read"]],
|
||||
prompt: `請靜態分析已交付的 Electron 程式 ${targets.javascript},說明 renderer 如何透過 preload 讀取 profile,以及 main handler 的關係。不要執行目標或安裝分析引擎;引用完整的產出 Evidence,保留 runtime 未知事項。`,
|
||||
},
|
||||
{
|
||||
id: "javascript-module-view",
|
||||
expectedFirstTool: "analyze_javascript_application",
|
||||
requiresEvidence: true,
|
||||
requiredToolSubsequence: [
|
||||
"analyze_javascript_application",
|
||||
"inspect_analysis_view",
|
||||
"inspect_analysis_view",
|
||||
],
|
||||
requiredAnswerTermGroups: [
|
||||
["main.js"],
|
||||
["node", "identity"],
|
||||
["observations", "recorded"],
|
||||
],
|
||||
prompt: `Inspect the recorded module identity, exports and observations for main.js in the shipped JavaScript tree ${targets.largeJavascript}. Use REA without directly reading target files or executing the target. Keep results focused on that module, cite Evidence, and state the limits of what its module view establishes.`,
|
||||
},
|
||||
{
|
||||
id: "missing-target",
|
||||
expectedFirstTool: null,
|
||||
requiresEvidence: false,
|
||||
requiredAnswerTermGroups: [],
|
||||
prompt:
|
||||
'Reverse engineer my desktop app and explain how it saves user settings. Use static analysis; do not install software. End with one JSON object containing status ("needs_target" or "analyzed"), requested_input ("app_name_or_artifact_path" or null), and question (a string or null).',
|
||||
},
|
||||
{
|
||||
id: "javascript-export-shape",
|
||||
expectedFirstTool: "analyze_javascript_application",
|
||||
@@ -253,14 +145,12 @@ try {
|
||||
const results = [];
|
||||
for (const scenario of scenarios) {
|
||||
process.stderr.write(`Running Codex agent evaluation: ${scenario.id}\n`);
|
||||
const fixtureId = scenario.fixtureScenarioId ?? scenario.id;
|
||||
const fixtureClaims = agentFixtureClaims(fixtureId, targets);
|
||||
const answerInstructions = agentFixtureAnswerInstructions(fixtureId);
|
||||
const fixtureClaims = agentFixtureClaims(scenario.id, targets);
|
||||
const answerInstructions = agentFixtureAnswerInstructions(scenario.id);
|
||||
const execution = await runCodex(
|
||||
answerInstructions.length === 0
|
||||
? scenario.prompt
|
||||
: `${scenario.prompt}\n\nThis closed factual rubric needs complete producer Evidence in the transcript. Request complete detail on each initial application analysis when supported; summary or view records alone are not accepted by this rubric.\n\n${answerInstructions}`,
|
||||
scenario.approvedTools ?? [],
|
||||
: `${scenario.prompt}\n\n${answerInstructions}`,
|
||||
);
|
||||
const transcriptDirectory = process.env.REA_AGENT_EVAL_TRANSCRIPT_DIR;
|
||||
if (transcriptDirectory !== undefined) {
|
||||
@@ -285,12 +175,6 @@ try {
|
||||
...(fixtureClaims === undefined ? {} : { fixtureClaims }),
|
||||
},
|
||||
);
|
||||
// Fixtures are deliberately visible beside prior investigation artifacts.
|
||||
// Their existence does not identify the unnamed app in this request.
|
||||
const targetClarificationPassed =
|
||||
scenario.expectedFirstTool === null
|
||||
? validTargetClarification(metrics.finalMessage)
|
||||
: undefined;
|
||||
const result = {
|
||||
id: scenario.id,
|
||||
expectedFirstTool: scenario.expectedFirstTool,
|
||||
@@ -301,9 +185,6 @@ try {
|
||||
exitCode: execution.exitCode,
|
||||
stderr: execution.stderr,
|
||||
...metrics,
|
||||
...(targetClarificationPassed === undefined
|
||||
? {}
|
||||
: { targetClarificationPassed }),
|
||||
};
|
||||
results.push(result);
|
||||
process.stderr.write(
|
||||
@@ -313,20 +194,13 @@ try {
|
||||
|
||||
const factualCorrectness = summarizeFactualCorrectness(results);
|
||||
const summary = {
|
||||
schemaVersion: 4,
|
||||
schemaVersion: 3,
|
||||
evaluationScope: "routing_workflow_and_configured_fixture_claims",
|
||||
factualCorrectness,
|
||||
verifier_run: await completeVerifierRun(verifierRun),
|
||||
codex,
|
||||
codexVersion,
|
||||
model: optionalModel ?? null,
|
||||
installation: {
|
||||
method: "rea_setup_codex",
|
||||
isolatedHome: true,
|
||||
isolatedCodexHome: true,
|
||||
skillVersion: PRODUCT_IDENTITY.skillVersion,
|
||||
authentication: "caller_auth_only_removed_after_run",
|
||||
},
|
||||
scenarios: results,
|
||||
totals: {
|
||||
scenarios: results.length,
|
||||
@@ -379,32 +253,12 @@ try {
|
||||
`Agent routing/workflow/answer checks failed: ${failed.map(({ id }) => id).join(", ")}`,
|
||||
);
|
||||
} finally {
|
||||
// Even a deliberately retained fixture directory must never retain auth.
|
||||
await rm(authPath, { force: true });
|
||||
if (process.env.REA_AGENT_EVAL_KEEP_FIXTURES !== "true")
|
||||
await rm(evaluationRoot, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function createTargets(root, includeManaged) {
|
||||
const fixtures = await createAgentEvaluationFixtures(root, repositoryRoot);
|
||||
const largeJavascript = join(root, "large-app");
|
||||
await mkdir(join(largeJavascript, "lib"), { recursive: true });
|
||||
await Promise.all([
|
||||
writeFile(
|
||||
join(largeJavascript, "package.json"),
|
||||
'{"name":"large-app","main":"main.js"}\n',
|
||||
),
|
||||
writeFile(
|
||||
join(largeJavascript, "main.js"),
|
||||
'const { ipcMain } = require("electron");\nipcMain.handle("profile:read", (_event, id) => ({ id }));\n',
|
||||
),
|
||||
...Array.from({ length: 180 }, (_, index) =>
|
||||
writeFile(
|
||||
join(largeJavascript, "lib", `formatter-${String(index)}.js`),
|
||||
`export const format = (value) => String(value).trim();\n`,
|
||||
),
|
||||
),
|
||||
]);
|
||||
|
||||
const managedOutput = join(root, "managed-output");
|
||||
if (includeManaged) {
|
||||
@@ -438,29 +292,15 @@ async function createTargets(root, includeManaged) {
|
||||
return {
|
||||
native: "/bin/true",
|
||||
...fixtures,
|
||||
largeJavascript,
|
||||
managed: join(managedOutput, "AgentEval.dll"),
|
||||
};
|
||||
}
|
||||
|
||||
function validTargetClarification(message) {
|
||||
const schema = z.strictObject({
|
||||
status: z.literal("needs_target"),
|
||||
requested_input: z.literal("app_name_or_artifact_path"),
|
||||
question: z.string().refine((value) => value.trim().length > 0),
|
||||
});
|
||||
try {
|
||||
return schema.safeParse(JSON.parse(message)).success;
|
||||
} catch (cause) {
|
||||
if (cause instanceof SyntaxError) return false;
|
||||
throw cause;
|
||||
}
|
||||
}
|
||||
|
||||
async function runCodex(prompt, approvedTools) {
|
||||
async function runCodex(prompt) {
|
||||
const arguments_ = [
|
||||
"--no-daemon",
|
||||
"exec",
|
||||
"--ignore-user-config",
|
||||
"--ignore-rules",
|
||||
"--ephemeral",
|
||||
"--skip-git-repo-check",
|
||||
"--sandbox",
|
||||
@@ -472,16 +312,18 @@ async function runCodex(prompt, approvedTools) {
|
||||
evaluationRoot,
|
||||
"-c",
|
||||
'approval_policy="never"',
|
||||
...approvedTools.flatMap((name) => [
|
||||
"-c",
|
||||
`mcp_servers.rea.tools.${name}.approval_mode="approve"`,
|
||||
]),
|
||||
"-c",
|
||||
`mcp_servers.rea.command=${JSON.stringify(process.execPath)}`,
|
||||
"-c",
|
||||
`mcp_servers.rea.args=${JSON.stringify([join(repositoryRoot, "scripts/rea.mjs"), "mcp"])}`,
|
||||
"-c",
|
||||
`mcp_servers.rea.startup_timeout_sec=${String(MCP_STARTUP_POLICY.codexStartupTimeoutSeconds)}`,
|
||||
...(optionalModel === undefined ? [] : ["--model", optionalModel]),
|
||||
prompt,
|
||||
];
|
||||
const child = spawn(codex, arguments_, {
|
||||
cwd: evaluationRoot,
|
||||
env: evaluationEnvironment,
|
||||
env: process.env,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
const events = [];
|
||||
@@ -520,16 +362,9 @@ async function runCodex(prompt, approvedTools) {
|
||||
return { events, exitCode, stderr: stderr.join("").slice(0, 32_768) };
|
||||
}
|
||||
|
||||
async function runProcess(
|
||||
command,
|
||||
arguments_,
|
||||
cwd,
|
||||
timeout,
|
||||
env = process.env,
|
||||
) {
|
||||
async function runProcess(command, arguments_, cwd, timeout) {
|
||||
const child = spawn(command, arguments_, {
|
||||
cwd,
|
||||
env,
|
||||
stdio: ["ignore", "ignore", "pipe"],
|
||||
});
|
||||
let stderr = "";
|
||||
|
||||
@@ -22,13 +22,6 @@ const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
|
||||
const command = process.env.REA_VERIFY_COPILOT_COMMAND ?? "copilot";
|
||||
const mode = process.argv[2] ?? "call";
|
||||
const model = process.env.REA_VERIFY_COPILOT_MODEL ?? "gpt-5.4";
|
||||
const wireModel = process.env.REA_VERIFY_COPILOT_WIRE_MODEL ?? model;
|
||||
const inputSchemaProfile =
|
||||
process.env.REA_VERIFY_COPILOT_SCHEMA_PROFILE ?? "full";
|
||||
assert(
|
||||
["full", "compact"].includes(inputSchemaProfile),
|
||||
"REA_VERIFY_COPILOT_SCHEMA_PROFILE must be full or compact",
|
||||
);
|
||||
assert(
|
||||
["chat", "call"].includes(mode),
|
||||
"Usage: verify-copilot-client.mjs [chat|call]",
|
||||
@@ -51,7 +44,6 @@ await exec("git", ["init", "--quiet", workspace], { timeout: 30_000 });
|
||||
const environment = {
|
||||
PATH: process.env.PATH,
|
||||
NODE_OPTIONS: process.env.NODE_OPTIONS,
|
||||
HOME: account,
|
||||
USERPROFILE: account,
|
||||
COPILOT_HOME: profile,
|
||||
COPILOT_OFFLINE: "true",
|
||||
@@ -62,7 +54,7 @@ const environment = {
|
||||
COPILOT_PROVIDER_BEARER_TOKEN: "",
|
||||
COPILOT_PROVIDER_HEADERS: "",
|
||||
COPILOT_PROVIDER_MODEL_ID: model,
|
||||
COPILOT_PROVIDER_WIRE_MODEL: wireModel,
|
||||
COPILOT_PROVIDER_WIRE_MODEL: model,
|
||||
COPILOT_PROVIDER_MAX_PROMPT_TOKENS: "1000000",
|
||||
COPILOT_PROVIDER_MAX_OUTPUT_TOKENS: "4096",
|
||||
REA_PROCESS_RUN_ID: verifier.run_id,
|
||||
@@ -120,17 +112,10 @@ const extractEvidence = (value) => {
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
const wireModels = [];
|
||||
const fixture = await createOpenAiModelFixture({
|
||||
directory: lab,
|
||||
prefix: "copilot",
|
||||
onRequest: async ({ body, tools, requestIndex }) => {
|
||||
assert.equal(
|
||||
body.model,
|
||||
wireModel,
|
||||
"Copilot must send the selected wire model",
|
||||
);
|
||||
wireModels.push(body.model);
|
||||
// Auxiliary requests without tools cannot execute the analyst's workflow.
|
||||
if (tools.length === 0)
|
||||
return { role: "assistant", content: "REA fixture title" };
|
||||
@@ -326,16 +311,6 @@ try {
|
||||
assert.deepEqual(JSON.parse(repeated.stdout).appliedActions, []);
|
||||
assert.equal(await readFile(configPath, "utf8"), configured);
|
||||
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
|
||||
if (inputSchemaProfile === "compact") {
|
||||
// Select the existing presentation in this disposable native-client profile.
|
||||
const customized = JSON.parse(configured);
|
||||
assert(customized.mcpServers.rea);
|
||||
customized.mcpServers.rea.env = {
|
||||
...customized.mcpServers.rea.env,
|
||||
REA_MCP_INPUT_SCHEMA_PROFILE: inputSchemaProfile,
|
||||
};
|
||||
await writeFile(configPath, JSON.stringify(customized, null, 2));
|
||||
}
|
||||
const installedSkill = await readFile(
|
||||
join(skillDirectory, "SKILL.md"),
|
||||
"utf8",
|
||||
@@ -433,8 +408,6 @@ try {
|
||||
clientVersion,
|
||||
mode,
|
||||
model,
|
||||
wireModel,
|
||||
inputSchemaProfile,
|
||||
passed: failure === undefined,
|
||||
artifacts: lab,
|
||||
runtimeRoot,
|
||||
@@ -448,9 +421,7 @@ try {
|
||||
skillDiscovery:
|
||||
"native COPILOT_HOME MCP configuration and caller-added isolated skill directory; default OS-home discovery unverified",
|
||||
capacityBoundary:
|
||||
inputSchemaProfile === "compact"
|
||||
? "Explicit compact profile: complete inventory and canonical server validation; advertised schemas may be reduced. This does not fix the full-profile gpt-4.1 context boundary."
|
||||
: "Complete canonical input and output schemas. Client model ID and wire model are recorded separately; effective token capacity and live provider compatibility remain unknown.",
|
||||
"gpt-4.1 model configuration blocks the full catalog before HTTP even with a ten-million-token requested prompt override; gpt-5.4 is the verified local fixture configuration",
|
||||
catalogSize,
|
||||
schemasChecked,
|
||||
spillReadRequested,
|
||||
@@ -458,7 +429,6 @@ try {
|
||||
skillLoaded,
|
||||
artifactDigest,
|
||||
finalSeen,
|
||||
wireModels,
|
||||
requests: fixture.requests,
|
||||
probes: fixture.probes,
|
||||
fixtureFailure: fixture.failure,
|
||||
|
||||
@@ -40,9 +40,9 @@ await Promise.all([
|
||||
]);
|
||||
const environment = {
|
||||
PATH: process.env.PATH ?? "",
|
||||
HOME: account,
|
||||
USERPROFILE: account,
|
||||
DSH_HOME: profile,
|
||||
DSH_AGENTS_HOME: join(account, ".agents"),
|
||||
OPENAI_API_KEY: "local-fixture-key",
|
||||
REA_PROCESS_RUN_ID: verifier.run_id,
|
||||
};
|
||||
@@ -328,7 +328,7 @@ try {
|
||||
"loopback custom OpenAI-compatible adapter; no live DeepSeek provider claim",
|
||||
modelContextWindow: 1000000,
|
||||
skillDiscovery:
|
||||
"native HOME shared personal directory; isolated project root",
|
||||
"native DSH_AGENTS_HOME personal directory; isolated project root",
|
||||
catalogSize: catalog?.length,
|
||||
patternCount,
|
||||
uniquePatterns: patterns.size,
|
||||
|
||||
@@ -54,7 +54,6 @@ await writeFile(
|
||||
);
|
||||
const environment = {
|
||||
PATH: process.env.PATH ?? "",
|
||||
HOME: account,
|
||||
USERPROFILE: account,
|
||||
[client === "pi" ? "PI_CODING_AGENT_DIR" : "HERMES_HOME"]: profile,
|
||||
PI_OFFLINE: "1",
|
||||
@@ -72,6 +71,13 @@ const original =
|
||||
})
|
||||
: "# Keep caller settings.\nmcp_servers:\n other:\n command: unrelated-server\n enabled: false\n";
|
||||
await writeFile(configPath, original);
|
||||
if (client === "pi") {
|
||||
// The CLI fixture account and native os.homedir() differ; use Pi's caller-owned skill setting.
|
||||
await writeFile(
|
||||
join(profile, "settings.json"),
|
||||
JSON.stringify({ skills: [join(account, ".agents", "skills")] }),
|
||||
);
|
||||
}
|
||||
const clientVersion = (
|
||||
await exec(command, ["--version"], { env: environment, timeout: 30_000 })
|
||||
).stdout.trim();
|
||||
@@ -442,7 +448,7 @@ try {
|
||||
"loopback deterministic OpenAI-compatible endpoint; no live provider claim",
|
||||
skillDiscovery:
|
||||
client === "pi"
|
||||
? "native HOME shared personal directory; no explicit skill setting"
|
||||
? "caller-configured isolated shared directory; default OS home discovery unverified"
|
||||
: "native HERMES_HOME personal directory",
|
||||
stickyHermes,
|
||||
catalogSize: catalog?.length,
|
||||
|
||||
@@ -45,7 +45,6 @@ await Promise.all([
|
||||
]);
|
||||
const baseEnvironment = {
|
||||
PATH: process.env.PATH ?? "",
|
||||
HOME: account,
|
||||
USERPROFILE: account,
|
||||
QWEN_HOME: profile,
|
||||
QWEN_RUNTIME_DIR: runtime,
|
||||
@@ -72,6 +71,9 @@ const original = JSON.stringify(
|
||||
},
|
||||
telemetry: { enabled: false },
|
||||
model: { name: "rea-client-fixture" },
|
||||
// Qwen uses os.homedir(), while REA's isolated CLI account uses USERPROFILE.
|
||||
// This caller-owned setting lets the real client load the REA-installed bundle.
|
||||
skills: { directories: [sharedSkills] },
|
||||
mcpServers: { other: { command: "unrelated-server", disabled: true } },
|
||||
},
|
||||
null,
|
||||
@@ -110,7 +112,7 @@ const configured = JSON.parse(
|
||||
await readFile(join(profile, "settings.json"), "utf8"),
|
||||
);
|
||||
assert.deepEqual(configured.model, { name: "rea-client-fixture" });
|
||||
assert.equal(configured.skills, undefined);
|
||||
assert.deepEqual(configured.skills, { directories: [sharedSkills] });
|
||||
assert.deepEqual(configured.mcpServers.other, {
|
||||
command: "unrelated-server",
|
||||
disabled: true,
|
||||
@@ -382,7 +384,7 @@ try {
|
||||
modelFixture:
|
||||
"loopback deterministic OpenAI-compatible endpoint; no live model-provider claim",
|
||||
skillDiscovery:
|
||||
"native HOME shared personal directory; no explicit skill setting",
|
||||
"caller-configured isolated shared directory; default OS home discovery unverified",
|
||||
catalogSize: TOOL_CONTRACTS.length,
|
||||
requests: fixture.requests,
|
||||
probes: fixture.probes,
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { mkdtemp, mkdir, readFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
@@ -60,9 +60,6 @@ const env = {
|
||||
JAVA_HOME: installation.javaHome,
|
||||
HOPPER_LAUNCHER_PATH: "/rea-unconfigured-deep-provider/hopper",
|
||||
};
|
||||
for (const setting of ["GHIDRA_HEADLESS_MAXMEM", "GHIDRA_MAXMEM"]) {
|
||||
if (process.env[setting] !== undefined) env[setting] = process.env[setting];
|
||||
}
|
||||
const reports = [];
|
||||
let primaryFailure;
|
||||
try {
|
||||
@@ -230,103 +227,6 @@ try {
|
||||
fixture.sha256,
|
||||
"Analysis modified the source executable",
|
||||
);
|
||||
// Carved firmware keeps all program headers/load bytes but loses sections.
|
||||
const read16 = (offset) =>
|
||||
byteOrder === "little"
|
||||
? sourceBytes.readUInt16LE(offset)
|
||||
: sourceBytes.readUInt16BE(offset);
|
||||
const read32 = (offset) =>
|
||||
byteOrder === "little"
|
||||
? sourceBytes.readUInt32LE(offset)
|
||||
: sourceBytes.readUInt32BE(offset);
|
||||
const sectionOffset = read32(32);
|
||||
const cut = sectionOffset - 1;
|
||||
const programOffset = read32(28);
|
||||
const programCount = read16(44);
|
||||
assert.equal(read16(42), 32);
|
||||
assert.ok(cut >= programOffset + programCount * 32);
|
||||
for (let i = 0; i < programCount; i += 1) {
|
||||
const offset = programOffset + i * 32;
|
||||
assert.ok(read32(offset + 4) + read32(offset + 16) <= cut);
|
||||
}
|
||||
const carvedBytes = sourceBytes.subarray(0, cut);
|
||||
const carvedPath = join(workspace, `${byteOrder}-carved.elf`);
|
||||
const carvedSha256 = createHash("sha256")
|
||||
.update(carvedBytes)
|
||||
.digest("hex");
|
||||
await writeFile(carvedPath, carvedBytes);
|
||||
const carvedResolved = await parseBinaryTarget(carvedPath);
|
||||
if (!carvedResolved.ok) throw carvedResolved.error;
|
||||
assert.deepEqual(
|
||||
carvedResolved.value.mips.abiFlags,
|
||||
independent.mips.abiFlags,
|
||||
);
|
||||
assert.match(
|
||||
carvedResolved.value.mips.limitations.join(" "),
|
||||
/section header table/u,
|
||||
);
|
||||
const carvedTarget = await call("open_binary", {
|
||||
path: carvedPath,
|
||||
provider_id: "ghidra",
|
||||
});
|
||||
assert.equal(carvedTarget.sha256, carvedSha256);
|
||||
const carvedImage = await call("inspect_native_load_image");
|
||||
assertMipsLoadedImage(carvedImage.observations, byteOrder, carvedSha256);
|
||||
const carvedProcedures = await call("list_procedures");
|
||||
assert.ok(
|
||||
carvedProcedures.some(
|
||||
(p) =>
|
||||
BigInt(p.address) === BigInt(independent.symbols.rea_mips_entry),
|
||||
),
|
||||
);
|
||||
const carvedDossier = functionDossierSchema.parse(
|
||||
await call("analyze_function", {
|
||||
procedure: independent.symbols.rea_mips_entry,
|
||||
}),
|
||||
);
|
||||
assert.ok(carvedDossier.pseudocode.trim().length > 0);
|
||||
const carvedSnapshotPath = join(
|
||||
workspace,
|
||||
`${byteOrder}-carved.snapshot.json`,
|
||||
);
|
||||
await call("close_binary", { snapshot_path: carvedSnapshotPath });
|
||||
const carvedSnapshot = parseAnalysisSnapshot(
|
||||
JSON.parse(await readFile(carvedSnapshotPath, "utf8")),
|
||||
);
|
||||
assert.equal(carvedSnapshot.target.sha256, carvedSha256);
|
||||
assert.ok(
|
||||
carvedSnapshot.evidence_bundle.records.some((record) =>
|
||||
parseEvidence(record).limitations.some((item) =>
|
||||
item.includes("section header table"),
|
||||
),
|
||||
),
|
||||
);
|
||||
const carvedCli = await execute(
|
||||
process.execPath,
|
||||
[
|
||||
entrypoint,
|
||||
"function",
|
||||
carvedPath,
|
||||
independent.symbols.rea_mips_entry,
|
||||
"--provider",
|
||||
"ghidra",
|
||||
"--json",
|
||||
],
|
||||
{ env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 },
|
||||
);
|
||||
const carvedEvidence = parseEvidence(JSON.parse(carvedCli.stdout));
|
||||
assert.equal(carvedEvidence.subject?.digest.sha256, carvedSha256);
|
||||
assert.ok(
|
||||
carvedEvidence.limitations.some((item) =>
|
||||
item.includes("section header table"),
|
||||
),
|
||||
);
|
||||
assert.equal(
|
||||
functionDossierSchema.parse(carvedEvidence.normalized_result).procedure
|
||||
.address,
|
||||
carvedDossier.procedure.address,
|
||||
);
|
||||
assert.deepEqual(await readFile(carvedPath), carvedBytes);
|
||||
reports.push({
|
||||
...fixture,
|
||||
independent_reader: independent,
|
||||
@@ -341,14 +241,6 @@ try {
|
||||
"CLI/MCP and snapshot identity",
|
||||
],
|
||||
procedures: procedures.length,
|
||||
carved: {
|
||||
sha256: carvedSha256,
|
||||
source_bytes: carvedBytes.length,
|
||||
section_header_offset: sectionOffset,
|
||||
effective_language: carvedImage.observations.language_id,
|
||||
limitations: carvedResolved.value.mips.limitations,
|
||||
procedures: carvedProcedures.length,
|
||||
},
|
||||
status: "passed",
|
||||
});
|
||||
} catch (cause) {
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
import { realpath } from "node:fs/promises";
|
||||
import { parseArgs } from "node:util";
|
||||
|
||||
import { analyzeJavaScriptApplicationWithOwnedScope } from "../../lib/javascript-application-analysis.mjs";
|
||||
import { analyzeJavaScriptApplication } from "../../../dist/application/javascript/JavaScriptApplicationService.js";
|
||||
import {
|
||||
compareApplicationVersionsEvidence,
|
||||
compareJavaScriptExportShapesEvidence,
|
||||
@@ -199,5 +199,9 @@ function selectorKey(selector) {
|
||||
}
|
||||
|
||||
async function analyze(path) {
|
||||
return analyzeJavaScriptApplicationWithOwnedScope({ input_path: path });
|
||||
const result = await analyzeJavaScriptApplication({
|
||||
input_path: path,
|
||||
});
|
||||
if (!result.ok) throw result.error;
|
||||
return result.value;
|
||||
}
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
import { readFile, realpath } from "node:fs/promises";
|
||||
import { parseArgs } from "node:util";
|
||||
|
||||
import { analyzeJavaScriptApplication } from "../../../dist/application/javascript/JavaScriptApplicationService.js";
|
||||
import { reconcileJavaScriptRuntimeEvidence } from "../../../dist/application/javascript/JavaScriptRuntimeReconciliationService.js";
|
||||
import { analyzeJavaScriptApplicationWithOwnedScope } from "../../lib/javascript-application-analysis.mjs";
|
||||
import {
|
||||
completeVerifierRun,
|
||||
createVerifierRun,
|
||||
@@ -44,13 +44,14 @@ const canonicalLayers = await Promise.all(
|
||||
);
|
||||
const staticLayers = [];
|
||||
for (const layer of canonicalLayers) {
|
||||
const analyzed = await analyzeJavaScriptApplicationWithOwnedScope({
|
||||
const analyzed = await analyzeJavaScriptApplication({
|
||||
input_path: layer.path,
|
||||
format: "auto",
|
||||
});
|
||||
if (!analyzed.ok) throw analyzed.error;
|
||||
staticLayers.push({
|
||||
role: layer.role,
|
||||
analysis: analyzed,
|
||||
analysis: analyzed.value,
|
||||
runtime_mappings: [],
|
||||
});
|
||||
}
|
||||
|
||||
+46
-348
@@ -1,6 +1,6 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { constants as fsConstants } from "node:fs";
|
||||
import { createReadStream, constants as fsConstants } from "node:fs";
|
||||
import { access, lstat, mkdir, rm, stat, writeFile } from "node:fs/promises";
|
||||
import { basename, join, resolve } from "node:path";
|
||||
|
||||
@@ -24,11 +24,8 @@ import {
|
||||
AnalysisTimeoutError,
|
||||
AnalysisUnsupportedTargetError,
|
||||
} from "../domain/analysisErrorCore.js";
|
||||
import { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { hashStableFile } from "../filesystem/StableFileHash.js";
|
||||
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
|
||||
import {
|
||||
execFileOutput,
|
||||
execFileOutputFailure,
|
||||
@@ -100,18 +97,11 @@ const STDERR_EXCERPT_BYTES = 512;
|
||||
const UNPARSED_LINES_REPORTED = 10;
|
||||
const STOP_GRACE_MS = 5_000;
|
||||
|
||||
const hashAdbFile = async (
|
||||
path: string,
|
||||
operation: string,
|
||||
signal: AbortSignal | undefined,
|
||||
) => {
|
||||
try {
|
||||
return await hashStableFile(path, signal);
|
||||
} catch (cause: unknown) {
|
||||
if (signal?.aborted === true)
|
||||
throw new AnalysisCancelledError(operation, { cause });
|
||||
throw cause;
|
||||
}
|
||||
/** Stream a local file digest without retaining large files in memory. */
|
||||
const sha256File = async (path: string): Promise<string> => {
|
||||
const hash = createHash("sha256");
|
||||
for await (const chunk of createReadStream(path)) hash.update(chunk);
|
||||
return hash.digest("hex");
|
||||
};
|
||||
|
||||
/** adb joins shell operands into a remote POSIX command; quote at that boundary. */
|
||||
@@ -345,48 +335,6 @@ export interface AdbProviderOptions {
|
||||
readonly environment?: Readonly<Record<string, string | undefined>>;
|
||||
}
|
||||
|
||||
interface PendingPackageDirectoryCleanup {
|
||||
readonly path: string;
|
||||
readonly identity: PackageDirectoryIdentity | undefined;
|
||||
readonly primary: AnalysisError;
|
||||
readonly remove: () => Promise<void>;
|
||||
}
|
||||
|
||||
interface PackageDirectoryIdentity {
|
||||
readonly dev: number;
|
||||
readonly ino: number;
|
||||
readonly mode: number;
|
||||
}
|
||||
|
||||
const removeOwnedPackageDirectory = async (
|
||||
path: string,
|
||||
identity: PackageDirectoryIdentity | undefined,
|
||||
): Promise<void> => {
|
||||
if (identity === undefined)
|
||||
throw new Error(
|
||||
`Cannot verify the created package directory identity; refusing to remove ${path}`,
|
||||
);
|
||||
let current: Awaited<ReturnType<typeof lstat>>;
|
||||
try {
|
||||
current = await lstat(path);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof Error && Reflect.get(cause, "code") === "ENOENT")
|
||||
return;
|
||||
throw cause;
|
||||
}
|
||||
if (
|
||||
!current.isDirectory() ||
|
||||
current.isSymbolicLink() ||
|
||||
current.dev !== identity.dev ||
|
||||
current.ino !== identity.ino ||
|
||||
current.mode !== identity.mode
|
||||
)
|
||||
throw new Error(
|
||||
`The created package directory was replaced or changed; refusing to remove ${path}`,
|
||||
);
|
||||
await rm(path, { recursive: true, force: true });
|
||||
};
|
||||
|
||||
/**
|
||||
* ADB device inspection and acquisition over a caller-selected adb binary.
|
||||
*
|
||||
@@ -398,12 +346,6 @@ const removeOwnedPackageDirectory = async (
|
||||
*/
|
||||
export class AdbProvider {
|
||||
private readonly environment: Readonly<Record<string, string | undefined>>;
|
||||
readonly #pendingPackageDirectoryCleanup =
|
||||
new Set<PendingPackageDirectoryCleanup>();
|
||||
readonly #activeOperations = new Set<Promise<void>>();
|
||||
#closing = false;
|
||||
#closePromise: Promise<void> | undefined;
|
||||
#packageCleanupPromise: Promise<void> | undefined;
|
||||
|
||||
constructor(options: AdbProviderOptions = {}) {
|
||||
this.environment = options.environment ?? process.env;
|
||||
@@ -436,121 +378,12 @@ export class AdbProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/** Retry removal of package directories retained after failed rollback. */
|
||||
close(): Promise<void> {
|
||||
this.#closing = true;
|
||||
if (this.#closePromise !== undefined) return this.#closePromise;
|
||||
const closing = (async () => {
|
||||
await Promise.all(this.#activeOperations);
|
||||
await this.#retryPackageDirectoryCleanup();
|
||||
})().finally(() => {
|
||||
if (this.#closePromise === closing) this.#closePromise = undefined;
|
||||
});
|
||||
this.#closePromise = closing;
|
||||
return this.#closePromise;
|
||||
}
|
||||
|
||||
#retryPackageDirectoryCleanup(): Promise<void> {
|
||||
if (this.#packageCleanupPromise !== undefined)
|
||||
return this.#packageCleanupPromise;
|
||||
const cleanup = this.#performPackageDirectoryCleanup().finally(() => {
|
||||
if (this.#packageCleanupPromise === cleanup)
|
||||
this.#packageCleanupPromise = undefined;
|
||||
});
|
||||
this.#packageCleanupPromise = cleanup;
|
||||
return cleanup;
|
||||
}
|
||||
|
||||
async #performPackageDirectoryCleanup(): Promise<void> {
|
||||
const failures: AnalysisError[] = [];
|
||||
for (const owner of this.#pendingPackageDirectoryCleanup) {
|
||||
try {
|
||||
await owner.remove();
|
||||
this.#pendingPackageDirectoryCleanup.delete(owner);
|
||||
} catch (cause: unknown) {
|
||||
const cleanup = new ProviderCleanupError(
|
||||
ADB_PROVIDER_IDENTITY.id,
|
||||
[owner.path],
|
||||
{
|
||||
reason: cause instanceof Error ? cause.message : String(cause),
|
||||
},
|
||||
{ operation: "pull_adb_package", cause },
|
||||
);
|
||||
failures.push(
|
||||
analysisErrorWithCleanupFailure(
|
||||
owner.primary,
|
||||
cleanup,
|
||||
"pull_adb_package",
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
if (failures.length === 1) throw failures[0];
|
||||
if (failures.length > 1)
|
||||
throw new ProviderCleanupError(
|
||||
ADB_PROVIDER_IDENTITY.id,
|
||||
[...this.#pendingPackageDirectoryCleanup].map(({ path }) => path),
|
||||
{
|
||||
reason:
|
||||
"Failed package output directories remain after cleanup retry",
|
||||
failures: failures.map(({ message, cleanupResources }) => ({
|
||||
message,
|
||||
resources: [...cleanupResources],
|
||||
})),
|
||||
},
|
||||
{
|
||||
operation: "pull_adb_package",
|
||||
cause: new AggregateError(
|
||||
failures,
|
||||
"Multiple ADB package rollback cleanups failed",
|
||||
),
|
||||
},
|
||||
);
|
||||
}
|
||||
/** The provider holds no long-lived resources; the adb server is caller-owned. */
|
||||
async close(): Promise<void> {}
|
||||
|
||||
async execute(
|
||||
request: AdbRequest,
|
||||
options?: ExecutionOptions,
|
||||
): Promise<Result<AnalysisExecution, AnalysisError>> {
|
||||
if (this.#closing)
|
||||
return err(new AnalysisCancelledError(request.operation));
|
||||
let release!: () => void;
|
||||
const active = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
this.#activeOperations.add(active);
|
||||
try {
|
||||
if (this.#pendingPackageDirectoryCleanup.size > 0) {
|
||||
try {
|
||||
await this.#retryPackageDirectoryCleanup();
|
||||
} catch (cause: unknown) {
|
||||
return err(
|
||||
cause instanceof AnalysisError
|
||||
? cause
|
||||
: new ProviderCleanupError(
|
||||
ADB_PROVIDER_IDENTITY.id,
|
||||
[...this.#pendingPackageDirectoryCleanup].map(
|
||||
({ path }) => path,
|
||||
),
|
||||
{
|
||||
reason:
|
||||
cause instanceof Error ? cause.message : String(cause),
|
||||
},
|
||||
{ operation: request.operation, cause },
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
return await this.#executeOperation(request, options);
|
||||
} finally {
|
||||
this.#activeOperations.delete(active);
|
||||
release();
|
||||
}
|
||||
}
|
||||
|
||||
async #executeOperation(
|
||||
request: AdbRequest,
|
||||
options?: ExecutionOptions,
|
||||
): Promise<Result<AnalysisExecution, AnalysisError>> {
|
||||
const signal = options?.signal;
|
||||
if (signal?.aborted === true)
|
||||
@@ -1177,11 +1010,9 @@ export class AdbProvider {
|
||||
);
|
||||
if (!directory.ok) return directory;
|
||||
const packageDirectory = join(directory.value, packageName);
|
||||
let packageDirectoryCreated = false;
|
||||
let packageDirectoryIdentity: PackageDirectoryIdentity | undefined;
|
||||
try {
|
||||
if (
|
||||
await lstat(packageDirectory).then(
|
||||
await stat(packageDirectory).then(
|
||||
() => true,
|
||||
() => false,
|
||||
)
|
||||
@@ -1190,44 +1021,14 @@ export class AdbProvider {
|
||||
`The package output directory already exists: ${packageDirectory}`,
|
||||
);
|
||||
await mkdir(packageDirectory);
|
||||
packageDirectoryCreated = true;
|
||||
const created = await lstat(packageDirectory);
|
||||
if (!created.isDirectory() || created.isSymbolicLink())
|
||||
throw new Error(
|
||||
`The created package output is not a regular directory: ${packageDirectory}`,
|
||||
);
|
||||
packageDirectoryIdentity = {
|
||||
dev: created.dev,
|
||||
ino: created.ino,
|
||||
mode: created.mode,
|
||||
};
|
||||
} catch (cause) {
|
||||
const primary = new AnalysisOutputError(
|
||||
operation,
|
||||
cause instanceof Error ? cause.message : String(cause),
|
||||
{ cause },
|
||||
return err(
|
||||
new AnalysisOutputError(
|
||||
operation,
|
||||
cause instanceof Error ? cause.message : String(cause),
|
||||
{ cause },
|
||||
),
|
||||
);
|
||||
if (!packageDirectoryCreated) return err(primary);
|
||||
const owner: PendingPackageDirectoryCleanup = {
|
||||
path: packageDirectory,
|
||||
identity: packageDirectoryIdentity,
|
||||
primary,
|
||||
remove: () =>
|
||||
removeOwnedPackageDirectory(
|
||||
packageDirectory,
|
||||
packageDirectoryIdentity,
|
||||
),
|
||||
};
|
||||
this.#pendingPackageDirectoryCleanup.add(owner);
|
||||
const cleanup = new ProviderCleanupError(
|
||||
ADB_PROVIDER_IDENTITY.id,
|
||||
[packageDirectory],
|
||||
{
|
||||
reason: "Could not establish an owned package directory identity",
|
||||
},
|
||||
{ operation, cause },
|
||||
);
|
||||
return err(analysisErrorWithCleanupFailure(primary, cleanup, operation));
|
||||
}
|
||||
const artifacts: {
|
||||
device_path: string;
|
||||
@@ -1243,37 +1044,6 @@ export class AdbProvider {
|
||||
stage: "resolve_paths" | "pull" | "digest";
|
||||
message: string;
|
||||
}[] = [];
|
||||
const cancelledPackagePull = (
|
||||
error: AnalysisCancelledError,
|
||||
localPath: string,
|
||||
devicePath: string,
|
||||
stage: "pull" | "digest",
|
||||
) => {
|
||||
error.retainPartialObservation({
|
||||
provider_id: "adb",
|
||||
operation,
|
||||
result: {
|
||||
client,
|
||||
serial,
|
||||
package_name: packageName,
|
||||
output_directory: directory.value,
|
||||
artifacts,
|
||||
failures: [
|
||||
...failures,
|
||||
{
|
||||
device_path: devicePath,
|
||||
stage,
|
||||
message:
|
||||
stage === "pull"
|
||||
? `Pull cancelled; any incomplete file is retained at ${localPath}. Remaining APKs were not attempted.`
|
||||
: `Digesting the pulled file was cancelled; the file is retained at ${localPath}. Remaining APKs were not attempted.`,
|
||||
},
|
||||
],
|
||||
coverage: "partial",
|
||||
},
|
||||
});
|
||||
return err(error);
|
||||
};
|
||||
const seenFileNames = new Set<string>();
|
||||
for (const entry of parsed.paths) {
|
||||
if (seenFileNames.has(entry.file_name)) {
|
||||
@@ -1295,13 +1065,7 @@ export class AdbProvider {
|
||||
signal,
|
||||
);
|
||||
if (!pulled.ok) {
|
||||
if (pulled.error instanceof AnalysisCancelledError)
|
||||
return cancelledPackagePull(
|
||||
pulled.error,
|
||||
localPath,
|
||||
entry.device_path,
|
||||
"pull",
|
||||
);
|
||||
if (pulled.error instanceof AnalysisCancelledError) return pulled;
|
||||
failures.push({
|
||||
device_path: entry.device_path,
|
||||
stage: "pull",
|
||||
@@ -1310,24 +1074,18 @@ export class AdbProvider {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const digest = await hashAdbFile(localPath, operation, signal);
|
||||
const info = await stat(localPath);
|
||||
const sha256 = await sha256File(localPath);
|
||||
artifacts.push({
|
||||
device_path: entry.device_path,
|
||||
file_name: entry.file_name,
|
||||
local_path: localPath,
|
||||
bytes: digest.bytes,
|
||||
sha256: digest.sha256,
|
||||
bytes: info.size,
|
||||
sha256,
|
||||
role: entry.role,
|
||||
role_basis: "file_name",
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisCancelledError)
|
||||
return cancelledPackagePull(
|
||||
cause,
|
||||
localPath,
|
||||
entry.device_path,
|
||||
"digest",
|
||||
);
|
||||
} catch {
|
||||
failures.push({
|
||||
device_path: entry.device_path,
|
||||
stage: "digest",
|
||||
@@ -1336,36 +1094,14 @@ export class AdbProvider {
|
||||
}
|
||||
}
|
||||
if (artifacts.length === 0 && failures.length > 0) {
|
||||
const primary = new AnalysisProtocolError(
|
||||
`Every APK pull failed for ${packageName} on ${serial}: ${failures[0]!.message}`,
|
||||
await rm(packageDirectory, { recursive: true, force: true }).catch(
|
||||
() => undefined,
|
||||
);
|
||||
return err(
|
||||
new AnalysisProtocolError(
|
||||
`Every APK pull failed for ${packageName} on ${serial}: ${failures[0]!.message}`,
|
||||
),
|
||||
);
|
||||
const owner: PendingPackageDirectoryCleanup = {
|
||||
path: packageDirectory,
|
||||
identity: packageDirectoryIdentity,
|
||||
primary,
|
||||
remove: () =>
|
||||
removeOwnedPackageDirectory(
|
||||
packageDirectory,
|
||||
packageDirectoryIdentity,
|
||||
),
|
||||
};
|
||||
try {
|
||||
await owner.remove();
|
||||
} catch (cause: unknown) {
|
||||
this.#pendingPackageDirectoryCleanup.add(owner);
|
||||
const cleanup = new ProviderCleanupError(
|
||||
ADB_PROVIDER_IDENTITY.id,
|
||||
[packageDirectory],
|
||||
{
|
||||
reason: cause instanceof Error ? cause.message : String(cause),
|
||||
},
|
||||
{ operation: operation, cause },
|
||||
);
|
||||
return err(
|
||||
analysisErrorWithCleanupFailure(primary, cleanup, operation),
|
||||
);
|
||||
}
|
||||
return err(primary);
|
||||
}
|
||||
return ok(
|
||||
createAnalysisExecution(
|
||||
@@ -1420,7 +1156,8 @@ export class AdbProvider {
|
||||
);
|
||||
if (!pulled.ok) return pulled;
|
||||
try {
|
||||
const digest = await hashAdbFile(localPath, operation, signal);
|
||||
const info = await stat(localPath);
|
||||
const sha256 = await sha256File(localPath);
|
||||
return ok(
|
||||
createAnalysisExecution(
|
||||
{
|
||||
@@ -1428,8 +1165,8 @@ export class AdbProvider {
|
||||
serial: input.serial,
|
||||
device_path: input.device_path,
|
||||
local_path: localPath,
|
||||
bytes: digest.bytes,
|
||||
sha256: digest.sha256,
|
||||
bytes: info.size,
|
||||
sha256,
|
||||
},
|
||||
provider,
|
||||
{
|
||||
@@ -1440,20 +1177,6 @@ export class AdbProvider {
|
||||
),
|
||||
);
|
||||
} catch (cause) {
|
||||
if (cause instanceof AnalysisCancelledError) {
|
||||
cause.retainPartialObservation({
|
||||
provider_id: "adb",
|
||||
operation,
|
||||
result: {
|
||||
client,
|
||||
serial: input.serial,
|
||||
local_path: localPath,
|
||||
device_path: input.device_path,
|
||||
digest_status: "unknown",
|
||||
},
|
||||
});
|
||||
return err(cause);
|
||||
}
|
||||
return err(
|
||||
new AnalysisOutputError(
|
||||
operation,
|
||||
@@ -1488,13 +1211,7 @@ export class AdbProvider {
|
||||
cause: new Error(`The local file does not exist: ${localPath}`),
|
||||
}),
|
||||
);
|
||||
let digest: Awaited<ReturnType<typeof hashAdbFile>>;
|
||||
try {
|
||||
digest = await hashAdbFile(localPath, operation, signal);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisCancelledError) return err(cause);
|
||||
throw cause;
|
||||
}
|
||||
const sha256 = await sha256File(localPath);
|
||||
const existsProbe = await runAdb(
|
||||
operation,
|
||||
input.device_path,
|
||||
@@ -1540,7 +1257,7 @@ export class AdbProvider {
|
||||
);
|
||||
if (deviceDigest.ok) {
|
||||
const digestMatch = /^([a-f0-9]{64})\s/u.exec(deviceDigest.value.stdout);
|
||||
if (digestMatch !== null && digestMatch[1] !== digest.sha256)
|
||||
if (digestMatch !== null && digestMatch[1] !== sha256)
|
||||
return err(
|
||||
new AnalysisArtifactChangedError(
|
||||
operation,
|
||||
@@ -1555,8 +1272,8 @@ export class AdbProvider {
|
||||
serial: input.serial,
|
||||
local_path: localPath,
|
||||
device_path: input.device_path,
|
||||
bytes: digest.bytes,
|
||||
sha256: digest.sha256,
|
||||
bytes: info.size,
|
||||
sha256,
|
||||
device_sha256: digestMatch?.[1] ?? null,
|
||||
overwritten: deviceExists,
|
||||
device_digest_source: "device_sha256sum",
|
||||
@@ -1577,8 +1294,8 @@ export class AdbProvider {
|
||||
serial: input.serial,
|
||||
local_path: localPath,
|
||||
device_path: input.device_path,
|
||||
bytes: digest.bytes,
|
||||
sha256: digest.sha256,
|
||||
bytes: info.size,
|
||||
sha256,
|
||||
device_sha256: null,
|
||||
overwritten: deviceExists,
|
||||
device_digest_source: "unavailable",
|
||||
@@ -1617,13 +1334,7 @@ export class AdbProvider {
|
||||
cause: new Error(`The local APK does not exist: ${apkPath}`),
|
||||
}),
|
||||
);
|
||||
let digest: Awaited<ReturnType<typeof hashAdbFile>>;
|
||||
try {
|
||||
digest = await hashAdbFile(apkPath, operation, signal);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof AnalysisCancelledError) return err(cause);
|
||||
throw cause;
|
||||
}
|
||||
const sha256 = await sha256File(apkPath);
|
||||
const installed = await runAdb(
|
||||
operation,
|
||||
input.serial,
|
||||
@@ -1652,8 +1363,8 @@ export class AdbProvider {
|
||||
client,
|
||||
serial: input.serial,
|
||||
apk_path: apkPath,
|
||||
bytes: digest.bytes,
|
||||
sha256: digest.sha256,
|
||||
bytes: info.size,
|
||||
sha256,
|
||||
replaced: input.replace,
|
||||
},
|
||||
provider,
|
||||
@@ -2023,15 +1734,16 @@ export class AdbProvider {
|
||||
),
|
||||
);
|
||||
try {
|
||||
const digest = await hashAdbFile(reportedPath, operation, signal);
|
||||
const info = await stat(reportedPath);
|
||||
const sha256 = await sha256File(reportedPath);
|
||||
return ok(
|
||||
createAnalysisExecution(
|
||||
{
|
||||
client,
|
||||
serial: input.serial,
|
||||
local_path: reportedPath,
|
||||
bytes: digest.bytes,
|
||||
sha256: digest.sha256,
|
||||
bytes: info.size,
|
||||
sha256,
|
||||
adb_reported_path: reportedPath,
|
||||
},
|
||||
provider,
|
||||
@@ -2043,20 +1755,6 @@ export class AdbProvider {
|
||||
),
|
||||
);
|
||||
} catch (cause) {
|
||||
if (cause instanceof AnalysisCancelledError) {
|
||||
cause.retainPartialObservation({
|
||||
provider_id: "adb",
|
||||
operation,
|
||||
result: {
|
||||
client,
|
||||
serial: input.serial,
|
||||
local_path: reportedPath,
|
||||
adb_reported_path: reportedPath,
|
||||
digest_status: "unknown",
|
||||
},
|
||||
});
|
||||
return err(cause);
|
||||
}
|
||||
return err(
|
||||
new AnalysisOutputError(
|
||||
operation,
|
||||
|
||||
@@ -1,43 +1,40 @@
|
||||
import { constants, createReadStream } from "node:fs";
|
||||
import { chmod, copyFile } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import type { ReadStream, Stats, WriteStream } from "node:fs";
|
||||
import { chmod, open, rm, stat, type FileHandle } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { Transform, type TransformCallback } from "node:stream";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import {
|
||||
AnalysisInputError,
|
||||
AnalysisCapabilityUnavailableError,
|
||||
} from "../domain/analysisErrorCore.js";
|
||||
import {
|
||||
openRegularFile,
|
||||
sameRegularFileState,
|
||||
} from "../filesystem/RegularFile.js";
|
||||
import type { AndroidOperation } from "../domain/android/androidAnalysis.js";
|
||||
|
||||
/** Stream a local file digest without retaining the APK or JAR in memory. */
|
||||
export const hashAndroidFile = async (path: string): Promise<string> => {
|
||||
const hash = createHash("sha256");
|
||||
for await (const chunk of createReadStream(path)) hash.update(chunk);
|
||||
return hash.digest("hex");
|
||||
};
|
||||
|
||||
/** Bind decompiler input to the admitted artifact, even if the original changes. */
|
||||
export const snapshotAndroidTarget = async (
|
||||
path: string,
|
||||
sha256: string,
|
||||
root: string,
|
||||
operation: AndroidOperation,
|
||||
signal?: AbortSignal,
|
||||
): Promise<string> => {
|
||||
const snapshot = join(root, "target.apk");
|
||||
try {
|
||||
await copyAndroidSnapshot(path, snapshot, sha256, signal);
|
||||
signal?.throwIfAborted();
|
||||
await chmod(snapshot, 0o400);
|
||||
return snapshot;
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof AndroidSnapshotChangedError)) throw cause;
|
||||
throw new AnalysisInputError(operation, { cause }, [
|
||||
await copyFile(path, snapshot, constants.COPYFILE_EXCL);
|
||||
await chmod(snapshot, 0o400);
|
||||
if ((await hashAndroidFile(snapshot)) !== sha256)
|
||||
throw new AnalysisInputError(operation, undefined, [
|
||||
{
|
||||
path: ["path"],
|
||||
reason: "invalid_value",
|
||||
message: `APK bytes changed after admission at ${path}; retry against a stable file.`,
|
||||
message:
|
||||
"APK bytes changed after admission; retry against a stable file.",
|
||||
},
|
||||
]);
|
||||
}
|
||||
return snapshot;
|
||||
};
|
||||
|
||||
/** Fingerprint the actual immutable engine bytes executed for this observation. */
|
||||
@@ -46,120 +43,15 @@ export const snapshotAndroidEngine = async (
|
||||
sha256: string,
|
||||
root: string,
|
||||
operation: AndroidOperation,
|
||||
signal?: AbortSignal,
|
||||
): Promise<{ path: string; sha256: string }> => {
|
||||
const path = join(root, "engine.jar");
|
||||
try {
|
||||
const observed = await copyAndroidSnapshot(source, path, sha256, signal);
|
||||
signal?.throwIfAborted();
|
||||
await chmod(path, 0o400);
|
||||
return { path, sha256: observed };
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof AndroidSnapshotChangedError)) throw cause;
|
||||
await copyFile(source, path, constants.COPYFILE_EXCL);
|
||||
await chmod(path, 0o400);
|
||||
if ((await hashAndroidFile(path)) !== sha256)
|
||||
throw new AnalysisCapabilityUnavailableError(
|
||||
"jadx",
|
||||
operation,
|
||||
`REA_JADX_MCP_JAR bytes changed during admission at ${source}; retry with a stable engine file.`,
|
||||
{ cause },
|
||||
"REA_JADX_MCP_JAR bytes changed during admission; retry with a stable engine file.",
|
||||
);
|
||||
}
|
||||
return { path, sha256 };
|
||||
};
|
||||
|
||||
const copyAndroidSnapshot = async (
|
||||
sourcePath: string,
|
||||
snapshotPath: string,
|
||||
expectedSha256: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<string> => {
|
||||
signal?.throwIfAborted();
|
||||
const source = await openRegularFile(sourcePath, {
|
||||
symlinks: "follow",
|
||||
signal,
|
||||
});
|
||||
let destination: FileHandle | undefined;
|
||||
let inputStream: ReadStream | undefined;
|
||||
let outputStream: WriteStream | undefined;
|
||||
let removeSnapshot = false;
|
||||
let copiedBytes = 0;
|
||||
const hash = createHash("sha256");
|
||||
try {
|
||||
const initial = await source.stat();
|
||||
signal?.throwIfAborted();
|
||||
const digest = new Transform({
|
||||
transform(
|
||||
chunk: Buffer,
|
||||
_encoding: BufferEncoding,
|
||||
callback: TransformCallback,
|
||||
) {
|
||||
copiedBytes += chunk.length;
|
||||
if (copiedBytes > initial.size) {
|
||||
callback(
|
||||
new AndroidSnapshotChangedError(
|
||||
`Android input grew while being copied: ${sourcePath}`,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
hash.update(chunk);
|
||||
callback(null, chunk);
|
||||
},
|
||||
});
|
||||
try {
|
||||
destination = await open(snapshotPath, "wx", 0o600);
|
||||
signal?.throwIfAborted();
|
||||
inputStream = source.createReadStream({
|
||||
start: 0,
|
||||
end: initial.size,
|
||||
autoClose: false,
|
||||
});
|
||||
outputStream = destination.createWriteStream({ autoClose: false });
|
||||
await pipeline(inputStream, digest, outputStream, { signal });
|
||||
signal?.throwIfAborted();
|
||||
await verifySourceState(source, sourcePath, initial, copiedBytes);
|
||||
const observedSha256 = hash.digest("hex");
|
||||
if (observedSha256 !== expectedSha256)
|
||||
throw new AndroidSnapshotChangedError(
|
||||
`Android snapshot digest mismatch for ${sourcePath}`,
|
||||
);
|
||||
return observedSha256;
|
||||
} catch (cause: unknown) {
|
||||
removeSnapshot = destination !== undefined;
|
||||
throw cause;
|
||||
}
|
||||
} finally {
|
||||
// Non-auto-closing streams retain a reference to their FileHandle even
|
||||
// after pipeline completion. Release those references before awaiting
|
||||
// descriptor closure, after the admitted source state has been verified.
|
||||
inputStream?.destroy();
|
||||
outputStream?.destroy();
|
||||
try {
|
||||
await Promise.all([source.close(), destination?.close()]);
|
||||
} finally {
|
||||
if (removeSnapshot)
|
||||
await rm(snapshotPath, { force: true }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const verifySourceState = async (
|
||||
handle: Awaited<ReturnType<typeof openRegularFile>>,
|
||||
path: string,
|
||||
initial: Stats,
|
||||
observedBytes: number,
|
||||
): Promise<void> => {
|
||||
const [opened, currentPath] = await Promise.all([handle.stat(), stat(path)]);
|
||||
if (
|
||||
observedBytes !== initial.size ||
|
||||
!sameRegularFileState(initial, opened) ||
|
||||
!sameRegularFileState(initial, currentPath)
|
||||
)
|
||||
throw new AndroidSnapshotChangedError(
|
||||
`Android input changed while being read: ${path}`,
|
||||
);
|
||||
};
|
||||
|
||||
class AndroidSnapshotChangedError extends Error {
|
||||
constructor(message: string, options?: ErrorOptions) {
|
||||
super(message, options);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -52,11 +52,7 @@ export const inspectJadxAvailability = async (
|
||||
signal?: AbortSignal,
|
||||
): Promise<ProviderAvailability> => {
|
||||
try {
|
||||
const configuration = await readJadxConfigurationInputs(
|
||||
environment,
|
||||
signal,
|
||||
);
|
||||
await validateJadxConfigurationContents(configuration, environment, signal);
|
||||
const configuration = await readJadxConfiguration(environment, signal);
|
||||
return {
|
||||
status: "available",
|
||||
code: null,
|
||||
@@ -81,83 +77,52 @@ export const inspectJadxAvailability = async (
|
||||
}
|
||||
};
|
||||
|
||||
/** Resolve caller-selected paths and JVM options without probing their contents. */
|
||||
export const resolveJadxConfigurationInputs = async (
|
||||
/** Resolve caller-selected tools and preserve the same prerequisite diagnostics. */
|
||||
export const resolveJadxConfiguration = async (
|
||||
environment: Readonly<Record<string, string | undefined>>,
|
||||
operation: AndroidOperation,
|
||||
signal?: AbortSignal,
|
||||
): Promise<JadxConfiguration> => {
|
||||
try {
|
||||
return await readJadxConfigurationInputs(environment, signal);
|
||||
return await readJadxConfiguration(environment, signal);
|
||||
} catch (cause) {
|
||||
if (signal?.aborted === true) throw cause;
|
||||
if (!(cause instanceof JadxConfigurationFailure)) throw cause;
|
||||
throw projectConfigurationFailure(operation, cause);
|
||||
const output = cause.diagnostics;
|
||||
throw new AnalysisCapabilityUnavailableError(
|
||||
"jadx",
|
||||
operation,
|
||||
cause.message,
|
||||
{
|
||||
cause,
|
||||
userMessage: cause.message,
|
||||
...(typeof output.stdout === "string" &&
|
||||
typeof output.stderr === "string"
|
||||
? {
|
||||
capturedOutput: {
|
||||
stdout: output.stdout,
|
||||
stderr: output.stderr,
|
||||
truncated: output.output_truncated === true,
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
/** Probe Java and archive prerequisites before starting a new engine session. */
|
||||
export const validateJadxConfiguration = async (
|
||||
configuration: JadxConfiguration,
|
||||
environment: Readonly<Record<string, string | undefined>>,
|
||||
operation: AndroidOperation,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> => {
|
||||
try {
|
||||
await validateJadxConfigurationContents(configuration, environment, signal);
|
||||
} catch (cause) {
|
||||
if (signal?.aborted === true) throw cause;
|
||||
if (!(cause instanceof JadxConfigurationFailure)) throw cause;
|
||||
throw projectConfigurationFailure(operation, cause);
|
||||
}
|
||||
};
|
||||
|
||||
const readJadxConfigurationInputs = async (
|
||||
const readJadxConfiguration = async (
|
||||
environment: Readonly<Record<string, string | undefined>>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<JadxConfiguration> => {
|
||||
signal?.throwIfAborted();
|
||||
requireSupportedHost();
|
||||
const jar = await resolveJar(environment);
|
||||
const java = await resolveJava(environment, signal);
|
||||
await requireJavaHomeExecutable(environment.JAVA_HOME, java);
|
||||
signal?.throwIfAborted();
|
||||
return { jar, java, jvmArguments: resolveJvmArguments(environment) };
|
||||
};
|
||||
|
||||
const validateJadxConfigurationContents = async (
|
||||
configuration: JadxConfiguration,
|
||||
environment: Readonly<Record<string, string | undefined>>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> => {
|
||||
signal?.throwIfAborted();
|
||||
await inspectJavaRuntime(configuration.java, environment, signal);
|
||||
await requireEngineArchive(configuration.jar, signal);
|
||||
};
|
||||
|
||||
const projectConfigurationFailure = (
|
||||
operation: AndroidOperation,
|
||||
cause: JadxConfigurationFailure,
|
||||
): AnalysisCapabilityUnavailableError => {
|
||||
const output = cause.diagnostics;
|
||||
return new AnalysisCapabilityUnavailableError(
|
||||
"jadx",
|
||||
operation,
|
||||
cause.message,
|
||||
{
|
||||
cause,
|
||||
userMessage: cause.message,
|
||||
...(typeof output.stdout === "string" && typeof output.stderr === "string"
|
||||
? {
|
||||
capturedOutput: {
|
||||
stdout: output.stdout,
|
||||
stderr: output.stderr,
|
||||
truncated: output.output_truncated === true,
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
);
|
||||
const jvmArguments = resolveJvmArguments(environment);
|
||||
await inspectJavaRuntime(java, environment, signal);
|
||||
await requireEngineArchive(jar, signal);
|
||||
return { jar, java, jvmArguments };
|
||||
};
|
||||
|
||||
const requireEngineArchive = async (
|
||||
|
||||
+28
-98
@@ -21,14 +21,12 @@ import { err, ok, type Result } from "../domain/result.js";
|
||||
import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js";
|
||||
import {
|
||||
snapshotAndroidEngine,
|
||||
hashAndroidFile,
|
||||
snapshotAndroidTarget,
|
||||
} from "./AndroidTargetSnapshot.js";
|
||||
import { hashStableFile } from "../filesystem/StableFileHash.js";
|
||||
import { RegularFileChangedError } from "../filesystem/RegularFile.js";
|
||||
import {
|
||||
inspectJadxAvailability,
|
||||
resolveJadxConfigurationInputs,
|
||||
validateJadxConfiguration,
|
||||
resolveJadxConfiguration,
|
||||
} from "./JadxConfiguration.js";
|
||||
import type { JadxLauncher } from "./JadxMcpTransport.js";
|
||||
|
||||
@@ -106,50 +104,6 @@ const executionError = (context: {
|
||||
});
|
||||
};
|
||||
|
||||
const androidTargetChanged = (
|
||||
operation: string,
|
||||
path: string,
|
||||
cause?: unknown,
|
||||
): AnalysisInputError =>
|
||||
new AnalysisInputError(
|
||||
operation,
|
||||
cause === undefined ? undefined : { cause },
|
||||
[
|
||||
{
|
||||
path: ["path"],
|
||||
reason: "invalid_value",
|
||||
message: `APK bytes changed after admission at ${path}; retry against a stable file.`,
|
||||
},
|
||||
],
|
||||
);
|
||||
|
||||
const androidSourceChanged = (
|
||||
operation: string,
|
||||
source: string,
|
||||
cause?: unknown,
|
||||
): AnalysisCapabilityUnavailableError =>
|
||||
new AnalysisCapabilityUnavailableError(
|
||||
"jadx",
|
||||
operation,
|
||||
`Selected JADX input bytes changed during admission at ${source}; retry with a stable installation.`,
|
||||
cause === undefined ? undefined : { cause },
|
||||
);
|
||||
|
||||
const hashAndroidSource = async (
|
||||
path: string,
|
||||
operation: string,
|
||||
signal: AbortSignal,
|
||||
): Promise<string> => {
|
||||
try {
|
||||
return (await hashStableFile(path, signal)).sha256;
|
||||
} catch (cause: unknown) {
|
||||
if (signal.aborted) throw cause;
|
||||
if (cause instanceof RegularFileChangedError)
|
||||
throw androidSourceChanged(operation, path, cause);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
|
||||
interface PendingCleanup {
|
||||
session: JadxSession | undefined;
|
||||
root: PrivateRuntimeRoot | undefined;
|
||||
@@ -193,18 +147,17 @@ const cleanup = async (
|
||||
await root?.close();
|
||||
resources.root = undefined;
|
||||
} catch (cause) {
|
||||
const error = new ProviderCleanupError(
|
||||
"jadx",
|
||||
[root?.path ?? "unknown"],
|
||||
{
|
||||
...diagnostics,
|
||||
reason: cause instanceof Error ? cause.message : String(cause),
|
||||
},
|
||||
{ cause: previousError ?? cause },
|
||||
return err(
|
||||
new ProviderCleanupError(
|
||||
"jadx",
|
||||
[root?.path ?? "unknown"],
|
||||
{
|
||||
...diagnostics,
|
||||
reason: cause instanceof Error ? cause.message : String(cause),
|
||||
},
|
||||
{ cause: previousError ?? cause },
|
||||
),
|
||||
);
|
||||
if (previousError?.partialObservation !== undefined)
|
||||
error.retainPartialObservation(previousError.partialObservation);
|
||||
return err(error);
|
||||
}
|
||||
return ok(undefined);
|
||||
};
|
||||
@@ -334,33 +287,23 @@ export class JadxProvider implements AndroidAnalysisPort {
|
||||
request.operation,
|
||||
`Target ${target.path} is ${target.format}; provide one standalone APK.`,
|
||||
);
|
||||
const configuration = await resolveJadxConfigurationInputs(
|
||||
const configuration = await resolveJadxConfiguration(
|
||||
this.environment,
|
||||
request.operation,
|
||||
signal,
|
||||
);
|
||||
signal.throwIfAborted();
|
||||
const jarHash = await hashAndroidSource(
|
||||
configuration.jar,
|
||||
request.operation,
|
||||
signal,
|
||||
);
|
||||
const bridgeHash = await hashAndroidSource(
|
||||
BRIDGE_SOURCE,
|
||||
request.operation,
|
||||
signal,
|
||||
);
|
||||
let targetHash: string;
|
||||
try {
|
||||
targetHash = (await hashStableFile(target.path, signal)).sha256;
|
||||
} catch (cause: unknown) {
|
||||
if (signal.aborted) throw cause;
|
||||
if (cause instanceof RegularFileChangedError)
|
||||
throw androidTargetChanged(request.operation, target.path, cause);
|
||||
throw cause;
|
||||
}
|
||||
if (targetHash !== target.sha256)
|
||||
throw androidTargetChanged(request.operation, target.path);
|
||||
const jarHash = await hashAndroidFile(configuration.jar);
|
||||
const bridgeHash = await hashAndroidFile(BRIDGE_SOURCE);
|
||||
if ((await hashAndroidFile(target.path)) !== target.sha256)
|
||||
throw new AnalysisInputError(request.operation, undefined, [
|
||||
{
|
||||
path: ["path"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
"APK bytes changed after admission; retry against a stable file.",
|
||||
},
|
||||
]);
|
||||
const key = JSON.stringify({
|
||||
path: target.path,
|
||||
sha256: target.sha256,
|
||||
@@ -383,40 +326,27 @@ export class JadxProvider implements AndroidAnalysisPort {
|
||||
}
|
||||
let retained = this.#retained;
|
||||
if (retained === undefined) {
|
||||
await validateJadxConfiguration(
|
||||
configuration,
|
||||
this.environment,
|
||||
request.operation,
|
||||
signal,
|
||||
);
|
||||
root = await PrivateRuntimeRoot.create({ prefix: "rea-android-" });
|
||||
const engine = await snapshotAndroidEngine(
|
||||
configuration.jar,
|
||||
jarHash,
|
||||
root.path,
|
||||
request.operation,
|
||||
signal,
|
||||
);
|
||||
const snapshot = await snapshotAndroidTarget(
|
||||
target.path,
|
||||
target.sha256,
|
||||
root.path,
|
||||
request.operation,
|
||||
signal,
|
||||
);
|
||||
const bridge = join(root.path, "ReaJadxBridge.java");
|
||||
await copyFile(BRIDGE_SOURCE, bridge);
|
||||
await chmod(bridge, 0o400);
|
||||
const copiedBridgeHash = await hashAndroidSource(
|
||||
bridge,
|
||||
request.operation,
|
||||
signal,
|
||||
);
|
||||
if (copiedBridgeHash !== bridgeHash)
|
||||
throw androidSourceChanged(
|
||||
if ((await hashAndroidFile(bridge)) !== bridgeHash)
|
||||
throw new AnalysisCapabilityUnavailableError(
|
||||
"jadx",
|
||||
request.operation,
|
||||
BRIDGE_SOURCE,
|
||||
undefined,
|
||||
"REA Android bridge bytes changed during admission; retry with a stable installation.",
|
||||
);
|
||||
signal.throwIfAborted();
|
||||
session = new JadxSession(
|
||||
|
||||
@@ -373,13 +373,7 @@ const readExecutableMetadata = async (
|
||||
if (!abiFlags.ok) return abiFlags;
|
||||
return ok({
|
||||
...parsed.value,
|
||||
mips: {
|
||||
...parsed.value.mips,
|
||||
abiFlags: abiFlags.value.abiFlags,
|
||||
...(abiFlags.value.limitations.length === 0
|
||||
? {}
|
||||
: { limitations: abiFlags.value.limitations }),
|
||||
},
|
||||
mips: { ...parsed.value.mips, abiFlags: abiFlags.value },
|
||||
});
|
||||
};
|
||||
|
||||
|
||||
@@ -1,22 +1,17 @@
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import {
|
||||
RegularFileCleanupFailure,
|
||||
readRegularFileText,
|
||||
} from "./RegularFileRead.js";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import {
|
||||
effectiveClientServer,
|
||||
parseClientConfiguration,
|
||||
type ClientConfigurationDocument,
|
||||
} from "./ClientConfigurationDocument.js";
|
||||
import type { SetupClient } from "./SupportedClients.js";
|
||||
import { clientRegistrationEnvironment } from "./ClientRegistrationEnvironment.js";
|
||||
|
||||
/** A named source failure that setup can explain without losing the bad path. */
|
||||
export interface ClientConfigurationFileError {
|
||||
readonly kind: "unreadable" | "malformed";
|
||||
readonly path: string;
|
||||
readonly detail: string;
|
||||
readonly cleanupFailure?: RegularFileCleanupFailure;
|
||||
}
|
||||
|
||||
/** Preserve named read failures and distinguish a missing source from an empty file. */
|
||||
@@ -24,7 +19,7 @@ export const readConfigurationText = async (
|
||||
path: string,
|
||||
): Promise<Result<string | undefined, ClientConfigurationFileError>> => {
|
||||
try {
|
||||
return ok(await readRegularFileText(path));
|
||||
return ok(await readFile(path, "utf8"));
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
|
||||
return ok(undefined);
|
||||
@@ -32,9 +27,6 @@ export const readConfigurationText = async (
|
||||
kind: "unreadable",
|
||||
path,
|
||||
detail: `Configuration ${path} could not be read: ${cause instanceof Error ? cause.message : String(cause)}`,
|
||||
...(cause instanceof RegularFileCleanupFailure
|
||||
? { cleanupFailure: cause }
|
||||
: {}),
|
||||
});
|
||||
}
|
||||
};
|
||||
@@ -71,16 +63,7 @@ export const readClientConfigurationFiles = async (
|
||||
for (const path of client.configPaths ?? [client.configPath]) {
|
||||
const document = await readConfigurationFile(path, client.format);
|
||||
if (!document.ok) return document;
|
||||
if (document.value !== undefined) {
|
||||
const environment = clientRegistrationEnvironment(document.value);
|
||||
if (!environment.ok)
|
||||
return err({
|
||||
kind: "malformed",
|
||||
path,
|
||||
detail: `Configuration ${path} has an invalid REA server environment: ${environment.error.settings.map(({ setting, constraint }) => `${setting}: ${constraint}`).join("; ")}. Repair it before rerunning setup.`,
|
||||
});
|
||||
documents.push(document.value);
|
||||
}
|
||||
if (document.value !== undefined) documents.push(document.value);
|
||||
}
|
||||
return ok(documents);
|
||||
};
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
import { z } from "zod";
|
||||
|
||||
import { parseEnvironment } from "../config/environment.js";
|
||||
import { ConfigurationError } from "../domain/configurationErrors.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import {
|
||||
effectiveClientServer,
|
||||
type ClientConfigurationDocument,
|
||||
} from "./ClientConfigurationDocument.js";
|
||||
import { PRODUCT_IDENTITY } from "../identity.js";
|
||||
|
||||
/** Read explicit REA overrides without dropping peers when one value is invalid. */
|
||||
export const clientRegistrationEnvironment = (
|
||||
parsed: ClientConfigurationDocument | undefined,
|
||||
overrides: Readonly<Record<string, string>> = {},
|
||||
): Result<Readonly<Record<string, string>>, ConfigurationError> => {
|
||||
const key =
|
||||
parsed?.dialect === "opencode" || parsed?.dialect === "opencode_v2"
|
||||
? "environment"
|
||||
: "env";
|
||||
const registration =
|
||||
parsed === undefined
|
||||
? undefined
|
||||
: effectiveClientServer(parsed, PRODUCT_IDENTITY.mcpServerKey);
|
||||
const raw =
|
||||
typeof registration === "object" && registration !== null
|
||||
? Reflect.get(registration, key)
|
||||
: undefined;
|
||||
const existing = z
|
||||
.record(z.string(), z.string())
|
||||
.safeParse(raw === undefined ? {} : raw);
|
||||
if (!existing.success)
|
||||
return err(
|
||||
new ConfigurationError("Invalid REA server environment", {
|
||||
settings: existing.error.issues.map((issue) => ({
|
||||
setting: [key, ...issue.path].join("."),
|
||||
constraint:
|
||||
"Server environment must be an object containing only string values",
|
||||
})),
|
||||
}),
|
||||
);
|
||||
// Validate known settings with the runtime's parser, but retain the original
|
||||
// string overrides, including unknown client-specific keys and no defaults.
|
||||
const environment = { ...existing.data, ...overrides };
|
||||
const validated = parseEnvironment(environment);
|
||||
if (!validated.ok) return validated;
|
||||
return ok(
|
||||
Object.fromEntries(
|
||||
Object.entries(environment).sort(([left], [right]) =>
|
||||
left.localeCompare(right),
|
||||
),
|
||||
),
|
||||
);
|
||||
};
|
||||
@@ -5,22 +5,27 @@ import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import { jsonValueSchema } from "../domain/jsonValue.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { compareBundles } from "../domain/bundleComparison.js";
|
||||
import { EvidenceLedger } from "./investigation/EvidenceLedger.js";
|
||||
import {
|
||||
readEvidenceBundle,
|
||||
writeEvidenceBundle,
|
||||
} from "./EvidenceBundleFiles.js";
|
||||
|
||||
/** Validate a standalone bundle and report its complete record/revision counts. */
|
||||
/** Validate and merge one bundle using the session Evidence ledger. */
|
||||
export const importEvidenceBundleCommand = async (
|
||||
path: string,
|
||||
): Promise<Result<JsonValue, AnalysisError>> => {
|
||||
const loaded = await readEvidenceBundle(path);
|
||||
if (!loaded.ok) return loaded;
|
||||
return ok({
|
||||
imported: loaded.value.records.length,
|
||||
unknowns_added: loaded.value.unknowns.length,
|
||||
total: loaded.value.records.length,
|
||||
});
|
||||
const ledger = createLedger();
|
||||
const imported = ledger.import(loaded.value);
|
||||
return imported.ok
|
||||
? ok({
|
||||
imported: imported.value.recordsAdded,
|
||||
unknowns_added: imported.value.unknownsAdded,
|
||||
total: ledger.export().records.length,
|
||||
})
|
||||
: imported;
|
||||
};
|
||||
|
||||
/** Validate a source bundle and atomically export canonical bytes. */
|
||||
@@ -59,6 +64,8 @@ export const compareEvidenceBundlesCommand = async (input: {
|
||||
}
|
||||
};
|
||||
|
||||
const createLedger = (): EvidenceLedger => new EvidenceLedger();
|
||||
|
||||
const projectWrite = (
|
||||
bundle: EvidenceBundle,
|
||||
written: Awaited<ReturnType<typeof writeEvidenceBundle>>,
|
||||
|
||||
@@ -16,7 +16,8 @@ import {
|
||||
AnalysisResourceConstraintError,
|
||||
} from "../domain/analysisErrorCore.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { readJsonInputFile } from "./JsonInputFile.js";
|
||||
import { parseUtf8Json } from "./Utf8JsonInput.js";
|
||||
import { readRegularFile } from "./RegularFileRead.js";
|
||||
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
|
||||
|
||||
/** Request control and its owning operation for an interruptible atomic write. */
|
||||
@@ -36,20 +37,12 @@ export const readJsonFile = async (
|
||||
> => {
|
||||
const requestedPath = resolve(path);
|
||||
try {
|
||||
const decoded = await readJsonInputFile(
|
||||
requestedPath,
|
||||
"read_evidence_file",
|
||||
);
|
||||
const encoded = await readRegularFile(requestedPath);
|
||||
const decoded = parseUtf8Json(encoded, "read_evidence_file", requestedPath);
|
||||
if (!decoded.ok) {
|
||||
if (decoded.error instanceof AnalysisResourceConstraintError)
|
||||
return err(decoded.error);
|
||||
return err(
|
||||
new EvidenceFileError("read", "invalid-json", {
|
||||
cause:
|
||||
decoded.error.cause ??
|
||||
new SyntaxError(
|
||||
decoded.error.issues[0]?.message ?? decoded.error.message,
|
||||
),
|
||||
cause: decoded.cause ?? new SyntaxError(decoded.error),
|
||||
path: requestedPath,
|
||||
}),
|
||||
);
|
||||
|
||||
@@ -5,12 +5,6 @@ import type {
|
||||
} from "../domain/binaryTargetTypes.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
|
||||
/** Observed ABI declarations and coverage of their source tables. */
|
||||
export interface MipsAbiFlagsInspection {
|
||||
readonly abiFlags: MipsAbiFlags | null;
|
||||
readonly limitations: readonly string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Read ELF32 MIPS ABI declarations from the caller's already-open file.
|
||||
* Inspect both typed tables (not section names), including records beyond the
|
||||
@@ -21,7 +15,7 @@ export const readMipsElfAbiFlags = async (
|
||||
handle: FileHandle,
|
||||
metadata: MipsElfMetadata,
|
||||
checkCancelled: () => void,
|
||||
): Promise<Result<MipsAbiFlagsInspection, string>> => {
|
||||
): Promise<Result<MipsAbiFlags | null, string>> => {
|
||||
checkCancelled();
|
||||
if (metadata.elfClass !== 32)
|
||||
return err("MIPS ABI record inspection currently supports ELF32 only");
|
||||
@@ -87,7 +81,6 @@ export const readMipsElfAbiFlags = async (
|
||||
dataOffset: 4,
|
||||
sizeOffset: 16,
|
||||
recordType: 0x70000003, // PT_MIPS_ABIFLAGS
|
||||
optional: false,
|
||||
},
|
||||
{
|
||||
label: "section header table",
|
||||
@@ -99,7 +92,6 @@ export const readMipsElfAbiFlags = async (
|
||||
dataOffset: 16,
|
||||
sizeOffset: 20,
|
||||
recordType: 0x7000002a, // SHT_MIPS_ABIFLAGS
|
||||
optional: true,
|
||||
},
|
||||
];
|
||||
// Extended numbering needs a separate section-zero reader. Refuse it rather
|
||||
@@ -107,7 +99,6 @@ export const readMipsElfAbiFlags = async (
|
||||
if (u16(bytes, 44) === 0xffff || u16(bytes, 48) >= 0xff00)
|
||||
return err("extended or reserved MIPS ELF table numbering is unsupported");
|
||||
let record: Buffer | undefined;
|
||||
const limitations: string[] = [];
|
||||
for (const table of tables) {
|
||||
if (table.count === 0) {
|
||||
if (table.offset !== 0)
|
||||
@@ -116,27 +107,16 @@ export const readMipsElfAbiFlags = async (
|
||||
}
|
||||
if (table.offset < 52 || table.stride !== table.entrySize)
|
||||
return err(`invalid MIPS ${table.label} offset or entry size`);
|
||||
const count = table.optional
|
||||
? Math.min(
|
||||
table.count,
|
||||
Math.max(0, Math.floor((fileSize - table.offset) / table.entrySize)),
|
||||
)
|
||||
: table.count;
|
||||
if (count < table.count)
|
||||
limitations.push(
|
||||
`MIPS ${table.label} at file offset ${table.offset} extends beyond the captured file: examined ${count} of ${table.count} complete headers. ABI declarations in unavailable headers remain unknown.`,
|
||||
);
|
||||
if (count === 0) continue;
|
||||
// Counts are bounded by the ELF16 fields and strides by their exact ELF32
|
||||
// structure sizes. Never allocate according to an untrusted section size.
|
||||
const contents = await read(
|
||||
table.offset,
|
||||
count * table.entrySize,
|
||||
table.count * table.entrySize,
|
||||
table.label,
|
||||
);
|
||||
if (!contents.ok) return contents;
|
||||
let found = false;
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
for (let i = 0; i < table.count; i += 1) {
|
||||
checkCancelled();
|
||||
const position = i * table.entrySize;
|
||||
if (u32(contents.value, position + table.typeOffset) !== table.recordType)
|
||||
@@ -154,10 +134,10 @@ export const readMipsElfAbiFlags = async (
|
||||
record = candidate.value;
|
||||
}
|
||||
}
|
||||
if (record === undefined) return ok({ abiFlags: null, limitations });
|
||||
if (record === undefined) return ok(null);
|
||||
// Preserve declarations verbatim, including unknown versions/values. Provider
|
||||
// admission, not parsing, decides which interpretations have been verified.
|
||||
const abiFlags: MipsAbiFlags = {
|
||||
return ok({
|
||||
version: u16(record, 0),
|
||||
isaLevel: record.readUInt8(2),
|
||||
isaRevision: record.readUInt8(3),
|
||||
@@ -169,6 +149,5 @@ export const readMipsElfAbiFlags = async (
|
||||
ases: u32(record, 12),
|
||||
flags1: u32(record, 16),
|
||||
flags2: u32(record, 20),
|
||||
};
|
||||
return ok({ abiFlags, limitations });
|
||||
});
|
||||
};
|
||||
|
||||
@@ -1,11 +1,3 @@
|
||||
import { javascriptApplicationAnalysisResultSchema } from "../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../contracts/javascript/javascriptRuntimeReconciliationExample.js";
|
||||
import {
|
||||
createJavaScriptSemanticGraph,
|
||||
createJavaScriptSemanticGraphNode,
|
||||
JavaScriptSemanticEvidenceContextRegistry,
|
||||
} from "../domain/javascript/javascriptSemanticGraph.js";
|
||||
import { proofEvidence } from "./ReconstructionObligationLedger.fixture.js";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { createEvidence } from "../domain/evidence.js";
|
||||
@@ -179,115 +171,3 @@ describe("reconstruction obligation candidates", () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps each semantic candidate bound to its own file and authority context", () => {
|
||||
const contexts = new JavaScriptSemanticEvidenceContextRegistry();
|
||||
const base = javascriptApplicationAnalysisResultSchema.parse(
|
||||
JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE.normalized_result,
|
||||
);
|
||||
const sourceRecords = [
|
||||
proofEvidence("first-file"),
|
||||
proofEvidence("second-file"),
|
||||
];
|
||||
const module = createJavaScriptSemanticGraphNode(
|
||||
{
|
||||
kind: "module",
|
||||
identity: {
|
||||
artifact_sha256: base.root_artifact_sha256,
|
||||
module_path: "app.js",
|
||||
source_range: null,
|
||||
role_key: "module",
|
||||
},
|
||||
function_node_id: null,
|
||||
application_node_ids: [],
|
||||
label: "app.js",
|
||||
properties: {},
|
||||
evidence: artifactEvidence(base.root_artifact_sha256, "app.js"),
|
||||
},
|
||||
contexts,
|
||||
);
|
||||
const requests = sourceRecords.flatMap((source, fileIndex) =>
|
||||
[1, 2].map((line) => {
|
||||
const sha256 = String(fileIndex + 4).repeat(64);
|
||||
const path = `file-${String(fileIndex)}.js`;
|
||||
return createJavaScriptSemanticGraphNode(
|
||||
{
|
||||
kind: "request",
|
||||
identity: {
|
||||
artifact_sha256: sha256,
|
||||
module_path: path,
|
||||
source_range: {
|
||||
start: { line, column: 0 },
|
||||
end: { line, column: 8 },
|
||||
},
|
||||
role_key: `request-${String(line)}`,
|
||||
},
|
||||
function_node_id: null,
|
||||
application_node_ids: [],
|
||||
label: `${path}:${String(line)}`,
|
||||
properties: {},
|
||||
evidence: {
|
||||
...artifactEvidence(sha256, path, "ast-static-analysis"),
|
||||
evidence_ids: [source.evidence_id],
|
||||
},
|
||||
},
|
||||
contexts,
|
||||
);
|
||||
}),
|
||||
);
|
||||
const { graph_id: _graphId, ...semantic } = base.semantic_graph;
|
||||
const semanticGraph = createJavaScriptSemanticGraph({
|
||||
...semantic,
|
||||
root_node_ids: [module.node_id],
|
||||
nodes: [module, ...requests],
|
||||
evidence_contexts: contexts.contexts,
|
||||
});
|
||||
const example = JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE;
|
||||
const application = createEvidence(
|
||||
{
|
||||
path: base.input_path,
|
||||
sha256: base.root_artifact_sha256,
|
||||
format: base.format,
|
||||
},
|
||||
example.provider,
|
||||
{
|
||||
predicateType: example.predicate_type,
|
||||
operation: example.operation,
|
||||
parameters: example.parameters,
|
||||
result: jsonValueSchema.parse({ ...base, semantic_graph: semanticGraph }),
|
||||
confidence: "derived",
|
||||
authority: "shipped-artifact",
|
||||
evidenceLinks: sourceRecords.map(({ evidence_id }) => evidence_id),
|
||||
},
|
||||
);
|
||||
const generated = deriveReconstructionObligationCandidates(
|
||||
createEvidenceBundle([...sourceRecords, application]),
|
||||
[],
|
||||
);
|
||||
expect(generated.limitations).toEqual([]);
|
||||
expect(generated.candidates).toHaveLength(requests.length);
|
||||
for (const [index, node] of requests.entries()) {
|
||||
const source = sourceRecords[Math.floor(index / 2)];
|
||||
expect(
|
||||
generated.candidates.find(
|
||||
({ target }) => target.semantic_node_id === node.node_id,
|
||||
),
|
||||
).toMatchObject({
|
||||
title: node.label,
|
||||
family: "request",
|
||||
source_state: "candidate",
|
||||
target: {
|
||||
artifact_sha256: node.identity.artifact_sha256,
|
||||
semantic_node_id: node.node_id,
|
||||
},
|
||||
authority_references: [
|
||||
{
|
||||
evidence_id: source?.evidence_id,
|
||||
authority: "controlled-replay",
|
||||
state: "candidate",
|
||||
location: `${semanticGraph.graph_id}/node/${node.node_id}`,
|
||||
},
|
||||
],
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
@@ -12,6 +12,7 @@ import type {
|
||||
ReconstructionObligation,
|
||||
ReviewedReconstructionObligation,
|
||||
} from "../domain/reconstructionObligationLedgerSchemas.js";
|
||||
import { resolveJavaScriptSemanticEvidence } from "../domain/javascript/javascriptSemanticGraph.js";
|
||||
import { parseApplicationGraphEvidence } from "./javascript/JavaScriptApplicationEvidenceGraph.js";
|
||||
import {
|
||||
applicationObligationPolicy,
|
||||
@@ -114,21 +115,14 @@ const deriveApplicationCandidates = (
|
||||
}),
|
||||
);
|
||||
}
|
||||
if (source.semanticGraph !== null) {
|
||||
const evidenceContexts = new Map(
|
||||
source.semanticGraph.evidence_contexts.map((context) => [
|
||||
context.context_id,
|
||||
context,
|
||||
]),
|
||||
);
|
||||
if (source.semanticGraph !== null)
|
||||
for (const node of source.semanticGraph.nodes) {
|
||||
const candidatePolicy = semanticObligationPolicy(node.kind);
|
||||
if (candidatePolicy === undefined) continue;
|
||||
const evidenceContext = evidenceContexts.get(node.evidence.context_id);
|
||||
if (evidenceContext === undefined)
|
||||
throw new TypeError(
|
||||
"Semantic evidence reference names an absent context",
|
||||
);
|
||||
const evidenceContext = resolveJavaScriptSemanticEvidence(
|
||||
source.semanticGraph,
|
||||
node.evidence,
|
||||
);
|
||||
addCandidate(
|
||||
candidates,
|
||||
generatedCandidate({
|
||||
@@ -150,7 +144,6 @@ const deriveApplicationCandidates = (
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
if (source.graph.coverage.status !== "complete")
|
||||
limitations.add(
|
||||
`Application graph ${source.graph.graph_id} coverage is ${source.graph.coverage.status}; omitted candidates remain unknown.`,
|
||||
|
||||
@@ -8,24 +8,14 @@ import {
|
||||
type HistoricalSourceGraphInput,
|
||||
} from "../domain/referenceSourceGraph.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import type { AnalysisCleanupObservation } from "../domain/analysisErrorBase.js";
|
||||
import { readReferenceSource } from "../reference/ReferenceSourceReader.js";
|
||||
import type {
|
||||
ReferenceSourceEntryKind,
|
||||
ReferenceSourceRead,
|
||||
ReferenceSourceReaderError,
|
||||
} from "../reference/ReferenceSourceReaderTypes.js";
|
||||
import { parseReferenceSourceEntries } from "./ReferenceSourceImportEntries.js";
|
||||
import { readReferenceSourceVcs } from "./ReferenceSourceVcsAdapter.js";
|
||||
import {
|
||||
type ReferenceSourceImportError,
|
||||
type ReferenceSourceImportOptions,
|
||||
} from "./ReferenceSourceImportTypes.js";
|
||||
import {
|
||||
prepareReferenceSourceImport,
|
||||
type PreparedReferenceSourceImport,
|
||||
} from "./ReferenceSourceImportPolicy.js";
|
||||
import { prepareReferenceSourceImport } from "./ReferenceSourceImportPolicy.js";
|
||||
|
||||
const failure = (
|
||||
code: ReferenceSourceImportError["code"],
|
||||
@@ -36,12 +26,8 @@ const failure = (
|
||||
message,
|
||||
});
|
||||
|
||||
const cancelled = (
|
||||
partial?: ReferenceSourceRead,
|
||||
): ReferenceSourceImportError => ({
|
||||
...failure("cancelled", "Reference source import cancelled"),
|
||||
...(partial === undefined ? {} : { partial }),
|
||||
});
|
||||
const cancelled = (): ReferenceSourceImportError =>
|
||||
failure("cancelled", "Reference source import cancelled");
|
||||
|
||||
const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true;
|
||||
|
||||
@@ -129,48 +115,9 @@ const sortExclusions = (
|
||||
[...exclusions].sort((left, right) => {
|
||||
const byPath = compareUnicodeCodePoints(left.path, right.path);
|
||||
if (byPath !== 0) return byPath;
|
||||
const byReason = compareUnicodeCodePoints(left.reason, right.reason);
|
||||
if (byReason !== 0) return byReason;
|
||||
return compareUnicodeCodePoints(
|
||||
"pattern" in left ? left.pattern : "",
|
||||
"pattern" in right ? right.pattern : "",
|
||||
);
|
||||
return compareUnicodeCodePoints(left.reason, right.reason);
|
||||
});
|
||||
|
||||
const createShouldExclude =
|
||||
(
|
||||
exclusions: HistoricalSourceGraphInput["exclusions"],
|
||||
secrets: PreparedReferenceSourceImport["secrets"],
|
||||
ignored: PreparedReferenceSourceImport["ignored"],
|
||||
): ((path: string, kind: ReferenceSourceEntryKind) => boolean) =>
|
||||
(path, kind) => {
|
||||
const patternPath = kind === "directory" ? `${path}/` : path;
|
||||
const secretMatch = secrets.test(patternPath);
|
||||
if (secretMatch.ignored) {
|
||||
if (!secretMatch.rule)
|
||||
throw new Error(`Ignored secret path has no matching rule: ${path}`);
|
||||
exclusions.push({
|
||||
path,
|
||||
reason: "configured-secret",
|
||||
pattern: secretMatch.rule.pattern,
|
||||
});
|
||||
return true;
|
||||
}
|
||||
const match = ignored.test(patternPath);
|
||||
if (!match.ignored) return false;
|
||||
if (!match.rule)
|
||||
throw new Error(`Ignored path has no matching rule: ${path}`);
|
||||
const reason = match.rule.mark;
|
||||
if (
|
||||
reason !== "project-ignored" &&
|
||||
reason !== "default-ignored" &&
|
||||
reason !== "caller-excluded"
|
||||
)
|
||||
throw new Error(`Ignored path has unknown rule origin: ${path}`);
|
||||
exclusions.push({ path, reason, pattern: match.rule.pattern });
|
||||
return true;
|
||||
};
|
||||
|
||||
/**
|
||||
* Import a reference source directory into a committed historical source graph.
|
||||
*
|
||||
@@ -180,41 +127,43 @@ const createShouldExclude =
|
||||
*/
|
||||
export const importReferenceSource = async (
|
||||
options: ReferenceSourceImportOptions,
|
||||
resources: ArtifactResourceScope,
|
||||
): Promise<Result<HistoricalSourceGraph, ReferenceSourceImportError>> => {
|
||||
if (isAborted(options.signal)) return err(cancelled());
|
||||
const prepared = await prepareReferenceSourceImport(options, resources);
|
||||
const prepared = await prepareReferenceSourceImport(options);
|
||||
if (!prepared.ok) return prepared;
|
||||
const { ignored, root, secrets } = prepared.value;
|
||||
if (isAborted(options.signal)) return err(cancelled());
|
||||
|
||||
const exclusions: HistoricalSourceGraphInput["exclusions"] = [];
|
||||
const shouldExclude = createShouldExclude(exclusions, secrets, ignored);
|
||||
const shouldExclude = (path: string): boolean => {
|
||||
if (secrets.ignores(path)) {
|
||||
exclusions.push({ path, reason: "configured-secret" });
|
||||
return true;
|
||||
}
|
||||
if (ignored.ignores(path)) {
|
||||
exclusions.push({ path, reason: "caller-excluded" });
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
const [readResult, vcsResult] = await Promise.all([
|
||||
readReferenceSource(root, resources, {
|
||||
const [readResult, vcs] = await Promise.all([
|
||||
readReferenceSource(root, {
|
||||
...(options.signal === undefined ? {} : { signal: options.signal }),
|
||||
shouldExclude,
|
||||
}),
|
||||
readReferenceSourceVcs(root, resources, options.signal),
|
||||
readReferenceSourceVcs(root, options.signal),
|
||||
]);
|
||||
|
||||
if (!readResult.ok)
|
||||
return err(
|
||||
combineReadFailures(
|
||||
readResult.error,
|
||||
vcsResult.ok ? undefined : vcsResult.error,
|
||||
readResult.error.partial,
|
||||
),
|
||||
);
|
||||
if (!vcsResult.ok)
|
||||
return err(
|
||||
combineReadFailures(vcsResult.error, undefined, readResult.value),
|
||||
);
|
||||
if (!readResult.ok) {
|
||||
const error = readResult.error;
|
||||
if (error.code === "cancelled") return err(cancelled());
|
||||
if (error.code === "unsupported")
|
||||
return err(failure("unsupported", error.message));
|
||||
return err(failure("io", error.message));
|
||||
}
|
||||
|
||||
const vcs = vcsResult.value;
|
||||
|
||||
if (isAborted(options.signal)) return err(cancelled(readResult.value));
|
||||
if (isAborted(options.signal)) return err(cancelled());
|
||||
|
||||
const read = readResult.value;
|
||||
const filePaths = new Set(
|
||||
@@ -226,7 +175,7 @@ export const importReferenceSource = async (
|
||||
const { entries, relationships, parseFailures, limitations } =
|
||||
parseReferenceSourceEntries(read, filePaths, options.signal);
|
||||
|
||||
if (isAborted(options.signal)) return err(cancelled(read));
|
||||
if (isAborted(options.signal)) return err(cancelled());
|
||||
|
||||
const uniqueRelationships = deduplicateRelationships(relationships);
|
||||
const uniqueFailures = normalizeHistoricalSourceParseFailures(parseFailures);
|
||||
@@ -272,45 +221,3 @@ export const importReferenceSource = async (
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
const combineReadFailures = (
|
||||
primary: ReferenceSourceReaderError,
|
||||
other: ReferenceSourceReaderError | undefined,
|
||||
partial: ReferenceSourceRead | undefined,
|
||||
): ReferenceSourceImportError => {
|
||||
const code =
|
||||
primary.code === "cancelled" || other?.code === "cancelled"
|
||||
? "cancelled"
|
||||
: primary.code;
|
||||
const cleanup = mergeCleanup(primary.cleanup, other?.cleanup);
|
||||
const message =
|
||||
other === undefined
|
||||
? primary.message
|
||||
: `${primary.message}; additional reference metadata failure: ${other.message}`;
|
||||
const cause =
|
||||
other === undefined
|
||||
? primary.cause
|
||||
: new AggregateError(
|
||||
[primary, other],
|
||||
"Reference source inventory and metadata both failed",
|
||||
{ cause: primary },
|
||||
);
|
||||
return {
|
||||
...failure(code, message),
|
||||
...(cleanup === undefined ? {} : { cleanup }),
|
||||
...(partial === undefined ? {} : { partial }),
|
||||
...(cause === undefined ? {} : { cause }),
|
||||
};
|
||||
};
|
||||
|
||||
const mergeCleanup = (
|
||||
first: AnalysisCleanupObservation | undefined,
|
||||
second: AnalysisCleanupObservation | undefined,
|
||||
): AnalysisCleanupObservation | undefined => {
|
||||
if (first === undefined) return second;
|
||||
if (second === undefined) return first;
|
||||
return {
|
||||
reason: `${first.reason}; ${second.reason}`,
|
||||
resources: [...new Set([...first.resources, ...second.resources])],
|
||||
};
|
||||
};
|
||||
|
||||
@@ -1,22 +1,14 @@
|
||||
import { lstat, realpath, stat } from "node:fs/promises";
|
||||
import { readFile, realpath, stat } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
|
||||
import ignore from "ignore";
|
||||
import {
|
||||
readRegularFileText,
|
||||
RegularFileCleanupFailure,
|
||||
retryRegularFileCleanup,
|
||||
} from "./RegularFileRead.js";
|
||||
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
|
||||
import {
|
||||
DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS,
|
||||
type ReferenceSourceImportError,
|
||||
type ReferenceSourceImportOptions,
|
||||
} from "./ReferenceSourceImportTypes.js";
|
||||
import { rootFilesystemFailure } from "../reference/ReferenceSourceReaderErrors.js";
|
||||
|
||||
/** Validated inputs ready for filesystem traversal. */
|
||||
export interface PreparedReferenceSourceImport {
|
||||
@@ -34,158 +26,54 @@ const failure = (
|
||||
message,
|
||||
});
|
||||
|
||||
const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true;
|
||||
|
||||
const addMarkedPatterns = (
|
||||
matcher: ReturnType<typeof ignore>,
|
||||
patterns: readonly string[],
|
||||
mark: "project-ignored" | "default-ignored" | "caller-excluded",
|
||||
): void => {
|
||||
for (const pattern of patterns)
|
||||
for (const line of pattern.split(/\r?\n/u))
|
||||
matcher.add({ pattern: line, mark });
|
||||
};
|
||||
|
||||
const resolveRoot = async (
|
||||
requestedRoot: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Result<string, ReferenceSourceImportError>> => {
|
||||
if (isAborted(signal))
|
||||
return err(failure("cancelled", "Reference source import cancelled"));
|
||||
try {
|
||||
const metadata = await stat(requestedRoot);
|
||||
if (isAborted(signal))
|
||||
return err(failure("cancelled", "Reference source import cancelled"));
|
||||
if (!metadata.isDirectory())
|
||||
if (!(await stat(requestedRoot)).isDirectory())
|
||||
return err(
|
||||
failure(
|
||||
"invalid-root",
|
||||
`Reference source root is not a directory: ${requestedRoot}`,
|
||||
),
|
||||
failure("invalid-root", "Reference source root is not a directory"),
|
||||
);
|
||||
const canonicalRoot = await realpath(resolve(requestedRoot));
|
||||
if (isAborted(signal))
|
||||
return err(failure("cancelled", "Reference source import cancelled"));
|
||||
return ok(canonicalRoot);
|
||||
} catch (cause: unknown) {
|
||||
if (isAborted(signal))
|
||||
return err(failure("cancelled", "Reference source import cancelled"));
|
||||
const rootFailure = rootFilesystemFailure(
|
||||
cause,
|
||||
"Reference source root could not be resolved",
|
||||
void cause;
|
||||
return err(
|
||||
failure("invalid-root", "Reference source root could not be resolved"),
|
||||
);
|
||||
if (rootFailure === undefined) throw cause;
|
||||
return err({ ...failure(rootFailure.code, rootFailure.message), cause });
|
||||
}
|
||||
};
|
||||
|
||||
const buildIgnored = async (
|
||||
root: string,
|
||||
excludePaths: readonly string[],
|
||||
resources: ArtifactResourceScope,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Result<ReturnType<typeof ignore>, ReferenceSourceImportError>> => {
|
||||
): Promise<ReturnType<typeof ignore>> => {
|
||||
const ignored = ignore();
|
||||
const policyPath = join(root, ".gitignore");
|
||||
let present = true;
|
||||
try {
|
||||
await lstat(policyPath);
|
||||
ignored.add(await readFile(join(root, ".gitignore"), "utf8"));
|
||||
} catch (cause: unknown) {
|
||||
if (
|
||||
!(
|
||||
typeof cause === "object" &&
|
||||
cause !== null &&
|
||||
"code" in cause &&
|
||||
cause.code === "ENOENT"
|
||||
)
|
||||
)
|
||||
throw cause;
|
||||
present = false;
|
||||
// Missing or unreadable ignore file does not authorize broader access.
|
||||
void cause;
|
||||
}
|
||||
signal?.throwIfAborted();
|
||||
if (present) {
|
||||
let text: string;
|
||||
try {
|
||||
text = await readRegularFileText(policyPath, { signal });
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof RegularFileCleanupFailure)) throw cause;
|
||||
const retry = await retryRegularFileCleanup(cause, resources);
|
||||
if (retry.cleanup !== undefined) {
|
||||
const primary =
|
||||
cause.outcome.kind === "failed"
|
||||
? cause.outcome.cause
|
||||
: cause.cleanupCause;
|
||||
return err({
|
||||
...failure(
|
||||
signal?.aborted === true ? "cancelled" : "io",
|
||||
`Reference source ignore policy could not be read at ${policyPath}: ${errorMessage(primary)}`,
|
||||
),
|
||||
cleanup: retry.cleanup,
|
||||
cause: primary,
|
||||
});
|
||||
}
|
||||
if (retry.outcome.kind === "failed") throw retry.outcome.cause;
|
||||
text = retry.outcome.value;
|
||||
}
|
||||
addMarkedPatterns(ignored, [text], "project-ignored");
|
||||
}
|
||||
addMarkedPatterns(
|
||||
ignored,
|
||||
DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS,
|
||||
"default-ignored",
|
||||
);
|
||||
ignored.add([...DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS]);
|
||||
for (const path of excludePaths) {
|
||||
addMarkedPatterns(ignored, [path, `${path}/`], "caller-excluded");
|
||||
ignored.add(path);
|
||||
ignored.add(`${path}/`);
|
||||
}
|
||||
return ok(ignored);
|
||||
return ignored;
|
||||
};
|
||||
|
||||
/** Resolve the caller-selected directory and build path filters. */
|
||||
export const prepareReferenceSourceImport = async (
|
||||
options: ReferenceSourceImportOptions,
|
||||
resources: ArtifactResourceScope,
|
||||
): Promise<
|
||||
Result<PreparedReferenceSourceImport, ReferenceSourceImportError>
|
||||
> => {
|
||||
try {
|
||||
return await resources.run(async () => {
|
||||
const root = await resolveRoot(options.root, options.signal);
|
||||
if (!root.ok) return root;
|
||||
try {
|
||||
const ignored = await buildIgnored(
|
||||
root.value,
|
||||
options.excludePaths ?? [],
|
||||
resources,
|
||||
options.signal,
|
||||
);
|
||||
if (!ignored.ok) return ignored;
|
||||
return ok({
|
||||
root: root.value,
|
||||
ignored: ignored.value,
|
||||
secrets: ignore().add([...options.policy.secretPatterns]),
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
return err({
|
||||
...failure(
|
||||
options.signal?.aborted === true ? "cancelled" : "io",
|
||||
`Reference source ignore policy could not be read at ${join(root.value, ".gitignore")}: ${errorMessage(cause)}`,
|
||||
),
|
||||
cause,
|
||||
});
|
||||
}
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof ArtifactReaderFailure)) throw cause;
|
||||
return err({
|
||||
...failure(
|
||||
cause.reason === "cancelled" ? "cancelled" : "io",
|
||||
cause.message,
|
||||
),
|
||||
...(cause.cleanup === undefined ? {} : { cleanup: cause.cleanup }),
|
||||
cause,
|
||||
});
|
||||
}
|
||||
const root = await resolveRoot(options.root);
|
||||
if (!root.ok) return root;
|
||||
return ok({
|
||||
root: root.value,
|
||||
ignored: await buildIgnored(root.value, options.excludePaths ?? []),
|
||||
secrets: ignore().add([...options.policy.secretPatterns]),
|
||||
});
|
||||
};
|
||||
|
||||
const errorMessage = (cause: unknown): string =>
|
||||
cause instanceof Error ? cause.message : String(cause);
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
projectReferenceSourceImportError,
|
||||
type ReferenceSourceImportError,
|
||||
} from "./ReferenceSourceImportTypes.js";
|
||||
import type { ReferenceSourceRead } from "../reference/ReferenceSourceReaderTypes.js";
|
||||
|
||||
describe("reference-source import error projection", () => {
|
||||
it("preserves primary evidence with category-specific recovery guidance", () => {
|
||||
const expectedCategories = {
|
||||
cancelled: "cancelled",
|
||||
"invalid-root": "invalid_input",
|
||||
unsupported: "unsupported_host",
|
||||
io: "execution_failure",
|
||||
parse: "execution_failure",
|
||||
} as const;
|
||||
for (const [code, category] of Object.entries(expectedCategories)) {
|
||||
const projected = projectReferenceSourceImportError({
|
||||
tag: "reference-source-import",
|
||||
code: code as ReferenceSourceImportError["code"],
|
||||
message: "EIO while reading /selected/source/src/main.ts",
|
||||
});
|
||||
expect(projected.category).toBe(category);
|
||||
expect(projected.message).toContain(
|
||||
"EIO while reading /selected/source/src/main.ts",
|
||||
);
|
||||
expect(projected.message).toMatch(
|
||||
/try again|when ready|Check that|REA on Linux/u,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("preserves cancellation, partial facts, and cleanup diagnostics", () => {
|
||||
const bytes = Buffer.from("export const observed = true;\n");
|
||||
const partial: ReferenceSourceRead = {
|
||||
root: "/selected/source",
|
||||
entries: [
|
||||
{
|
||||
status: "read",
|
||||
kind: "file",
|
||||
path: "src/main.ts",
|
||||
bytes,
|
||||
size: bytes.length,
|
||||
},
|
||||
],
|
||||
bytesRead: bytes.length,
|
||||
limitations: [],
|
||||
};
|
||||
const projected = projectReferenceSourceImportError({
|
||||
tag: "reference-source-import",
|
||||
code: "cancelled",
|
||||
message: "Traversal cancelled after reading src/main.ts",
|
||||
cleanup: {
|
||||
reason: "directory close rejected",
|
||||
resources: ["/selected/source"],
|
||||
},
|
||||
partial,
|
||||
});
|
||||
|
||||
expect(projected).toMatchObject({
|
||||
category: "cancelled",
|
||||
message: expect.stringContaining(
|
||||
"Traversal cancelled after reading src/main.ts",
|
||||
),
|
||||
cleanup: {
|
||||
reason: "directory close rejected",
|
||||
resources: ["/selected/source"],
|
||||
},
|
||||
partial: {
|
||||
root: "/selected/source",
|
||||
entries: [{ path: "src/main.ts", bytes }],
|
||||
},
|
||||
});
|
||||
expect(projected.message).toContain("Cleanup failed during import");
|
||||
expect(projected.message).toContain("directory close rejected");
|
||||
expect(projected.message).toContain("/selected/source");
|
||||
expect(projected.message).not.toContain("remains unconfirmed");
|
||||
});
|
||||
});
|
||||
@@ -1,67 +1,44 @@
|
||||
import type { ReferenceSourcePolicy } from "../domain/referenceSourcePolicy.js";
|
||||
import type { AnalysisCleanupObservation } from "../domain/analysisErrorBase.js";
|
||||
import type { ReferenceSourceRead } from "../reference/ReferenceSourceReaderTypes.js";
|
||||
|
||||
/** Typed expected failure returned by historical-source imports. */
|
||||
export interface ReferenceSourceImportError {
|
||||
readonly tag: "reference-source-import";
|
||||
readonly code: "cancelled" | "invalid-root" | "unsupported" | "io" | "parse";
|
||||
readonly message: string;
|
||||
readonly cleanup?: AnalysisCleanupObservation;
|
||||
readonly partial?: ReferenceSourceRead;
|
||||
readonly cause?: unknown;
|
||||
}
|
||||
|
||||
const IMPORT_FAILURE_GUIDANCE = {
|
||||
cancelled: {
|
||||
category: "cancelled",
|
||||
message:
|
||||
"Reference source import was cancelled. Start it again when ready.",
|
||||
},
|
||||
"invalid-root": {
|
||||
category: "invalid_input",
|
||||
message:
|
||||
"Reference source directory could not be opened. Check that the path exists, is readable, and points to a directory.",
|
||||
},
|
||||
unsupported: {
|
||||
category: "unsupported_host",
|
||||
message:
|
||||
"Safe no-follow file opens are unavailable on this host. Import the source tree with REA on Linux (including WSL) or macOS.",
|
||||
},
|
||||
io: {
|
||||
category: "execution_failure",
|
||||
message:
|
||||
"Reference source files could not be read. Check directory permissions and try again.",
|
||||
},
|
||||
parse: {
|
||||
category: "execution_failure",
|
||||
message:
|
||||
"Reference source could not be indexed. Check that the source tree is readable, then try again.",
|
||||
},
|
||||
} satisfies Record<
|
||||
ReferenceSourceImportError["code"],
|
||||
{ readonly category: string; readonly message: string }
|
||||
>;
|
||||
|
||||
/** Safe CLI projection for a historical-source import failure. */
|
||||
export const projectReferenceSourceImportError = (
|
||||
error: ReferenceSourceImportError,
|
||||
): Readonly<{
|
||||
category: string;
|
||||
message: string;
|
||||
cleanup?: AnalysisCleanupObservation;
|
||||
partial?: ReferenceSourceRead;
|
||||
}> => {
|
||||
const primary = IMPORT_FAILURE_GUIDANCE[error.code];
|
||||
const cleanupMessage =
|
||||
error.cleanup === undefined
|
||||
? ""
|
||||
: ` Cleanup failed during import for ${error.cleanup.resources.join(", ")}: ${error.cleanup.reason}.`;
|
||||
): Readonly<{ category: string; message: string }> => {
|
||||
if (error.code === "cancelled")
|
||||
return {
|
||||
category: "cancelled",
|
||||
message:
|
||||
"Reference source import was cancelled. Start it again when ready.",
|
||||
};
|
||||
if (error.code === "invalid-root")
|
||||
return {
|
||||
category: "invalid_input",
|
||||
message:
|
||||
"Reference source directory could not be opened. Check that the path exists, is readable, and points to a directory.",
|
||||
};
|
||||
if (error.code === "unsupported")
|
||||
return {
|
||||
category: "unsupported_host",
|
||||
message:
|
||||
"Safe no-follow file opens are unavailable on this host. Import the source tree with REA on Linux (including WSL) or macOS.",
|
||||
};
|
||||
if (error.code === "io")
|
||||
return {
|
||||
category: "execution_failure",
|
||||
message:
|
||||
"Reference source files could not be read. Check directory permissions and try again.",
|
||||
};
|
||||
return {
|
||||
category: primary.category,
|
||||
message: `${error.message} ${primary.message}${cleanupMessage}`,
|
||||
...(error.cleanup === undefined ? {} : { cleanup: error.cleanup }),
|
||||
...(error.partial === undefined ? {} : { partial: error.partial }),
|
||||
category: "execution_failure",
|
||||
message:
|
||||
"Reference source could not be indexed. Check that the source tree is readable, then try again.",
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
@@ -1,15 +1,6 @@
|
||||
import fs from "node:fs";
|
||||
import { lstat } from "node:fs/promises";
|
||||
import { lstat, readFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
|
||||
import {
|
||||
readRegularFile,
|
||||
RegularFileCleanupFailure,
|
||||
retryRegularFileCleanup,
|
||||
} from "./RegularFileRead.js";
|
||||
import type { ReferenceSourceReaderError } from "../reference/ReferenceSourceReaderTypes.js";
|
||||
|
||||
export type ReferenceSourceVcsInfo =
|
||||
| {
|
||||
@@ -27,174 +18,34 @@ export type ReferenceSourceVcsInfo =
|
||||
*/
|
||||
export const readReferenceSourceVcs = async (
|
||||
root: string,
|
||||
resources: ArtifactResourceScope,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Result<ReferenceSourceVcsInfo, ReferenceSourceReaderError>> => {
|
||||
): Promise<ReferenceSourceVcsInfo> => {
|
||||
if (isAborted(signal)) return { kind: "unknown", head: null, dirty: null };
|
||||
try {
|
||||
return await resources.run(async () => {
|
||||
if (isAborted(signal)) return ok(unknownVcs());
|
||||
try {
|
||||
await lstat(join(root, ".git"));
|
||||
} catch (cause: unknown) {
|
||||
return ok(
|
||||
errorCode(cause) === "ENOENT"
|
||||
? { kind: "none", head: null, dirty: null }
|
||||
: unknownVcs(),
|
||||
);
|
||||
}
|
||||
try {
|
||||
const head = await resolveSourceHead(root, resources, signal);
|
||||
if (isAborted(signal)) return ok(unknownVcs());
|
||||
return ok({ kind: "git", head, dirty: null });
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof ArtifactReaderFailure) throw cause;
|
||||
return ok(unknownVcs());
|
||||
}
|
||||
});
|
||||
await lstat(join(root, ".git"));
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof ArtifactReaderFailure)) throw cause;
|
||||
return err({
|
||||
tag: "reference-source-reader",
|
||||
code: cause.reason === "cancelled" ? "cancelled" : "io",
|
||||
message: cause.message,
|
||||
...(cause.cleanup === undefined ? {} : { cleanup: cause.cleanup }),
|
||||
cause,
|
||||
});
|
||||
return errorCode(cause) === "ENOENT"
|
||||
? { kind: "none", head: null, dirty: null }
|
||||
: { kind: "unknown", head: null, dirty: null };
|
||||
}
|
||||
try {
|
||||
const head = await resolveSourceHead(root);
|
||||
if (isAborted(signal)) return { kind: "unknown", head: null, dirty: null };
|
||||
return { kind: "git", head, dirty: null };
|
||||
} catch (cause: unknown) {
|
||||
// Unresolvable refs mean VCS state is unknown, not absent.
|
||||
void cause;
|
||||
return { kind: "unknown", head: null, dirty: null };
|
||||
}
|
||||
};
|
||||
|
||||
const unknownVcs = (): ReferenceSourceVcsInfo => ({
|
||||
kind: "unknown",
|
||||
head: null,
|
||||
dirty: null,
|
||||
});
|
||||
|
||||
const resolveSourceHead = async (
|
||||
root: string,
|
||||
resources: ArtifactResourceScope,
|
||||
signal?: AbortSignal,
|
||||
): Promise<string> => {
|
||||
const resolveSourceHead = async (root: string): Promise<string> => {
|
||||
// isomorphic-git is loaded on first use so CLI and MCP startup skip it.
|
||||
const { resolveRef } = await import("isomorphic-git");
|
||||
let failedRead: { readonly cause: unknown } | undefined;
|
||||
const cleanupFailures: ArtifactReaderFailure[] = [];
|
||||
const readMetadataBytes = async (path: string): Promise<Buffer> => {
|
||||
try {
|
||||
return await readRegularFile(path, { signal });
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof RegularFileCleanupFailure)) throw cause;
|
||||
const retry = await retryRegularFileCleanup(cause, resources);
|
||||
if (
|
||||
retry.outcome.kind === "failed" &&
|
||||
errorCode(retry.outcome.cause) !== "ENOENT"
|
||||
)
|
||||
failedRead ??= { cause: retry.outcome.cause };
|
||||
if (
|
||||
retry.cleanup === undefined &&
|
||||
retry.outcome.kind === "failed" &&
|
||||
!isAborted(signal)
|
||||
)
|
||||
throw retry.outcome.cause;
|
||||
const primary =
|
||||
retry.outcome.kind === "failed"
|
||||
? retry.outcome.cause
|
||||
: cause.cleanupCause;
|
||||
throw new ArtifactReaderFailure(
|
||||
isAborted(signal) ? "cancelled" : "io",
|
||||
`Git metadata could not be read safely: ${errorMessage(primary)}`,
|
||||
{
|
||||
cause: primary,
|
||||
...(retry.cleanup === undefined ? {} : { cleanup: retry.cleanup }),
|
||||
},
|
||||
);
|
||||
}
|
||||
};
|
||||
const readMetadataText = async (path: string): Promise<string> =>
|
||||
(await readMetadataBytes(path)).toString("utf8");
|
||||
const throwCleanupFailures = (primary: unknown): never => {
|
||||
const observations = cleanupFailures.flatMap((failure) =>
|
||||
failure.cleanup === undefined ? [] : [failure.cleanup],
|
||||
);
|
||||
throw new ArtifactReaderFailure(
|
||||
isAborted(signal) ? "cancelled" : "io",
|
||||
`Git metadata could not be read safely: ${errorMessage(primary)}`,
|
||||
{
|
||||
cause: new AggregateError(
|
||||
[primary, ...cleanupFailures],
|
||||
"Git metadata reads and cleanup failed",
|
||||
{ cause: primary },
|
||||
),
|
||||
...(observations.length === 0
|
||||
? {}
|
||||
: {
|
||||
cleanup: {
|
||||
reason: observations.map(({ reason }) => reason).join("; "),
|
||||
resources: [
|
||||
...new Set(
|
||||
observations.flatMap(({ resources }) => resources),
|
||||
),
|
||||
],
|
||||
},
|
||||
}),
|
||||
},
|
||||
);
|
||||
};
|
||||
const referenceFs = {
|
||||
...fs,
|
||||
promises: {
|
||||
...fs.promises,
|
||||
// Let isomorphic-git continue from a failed loose ref to packed-refs,
|
||||
// then report the original read failure after resolution. This preserves
|
||||
// both observations when packed metadata also fails cleanup.
|
||||
readFile: async (
|
||||
path: string | undefined,
|
||||
options:
|
||||
| BufferEncoding
|
||||
| { readonly encoding?: BufferEncoding | null }
|
||||
| null = null,
|
||||
) => {
|
||||
// The library probes readFile() without a path to detect promise APIs.
|
||||
// Reject that invocation without attributing it to a source read.
|
||||
if (path === undefined)
|
||||
throw new TypeError("Git filesystem read requires a path");
|
||||
try {
|
||||
const bytes = await readMetadataBytes(path);
|
||||
const encoding =
|
||||
typeof options === "string" ? options : options?.encoding;
|
||||
return encoding === undefined || encoding === null
|
||||
? bytes
|
||||
: bytes.toString(encoding);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof ArtifactReaderFailure) {
|
||||
cleanupFailures.push(cause);
|
||||
} else if (errorCode(cause) !== "ENOENT") failedRead ??= { cause };
|
||||
throw cause;
|
||||
}
|
||||
},
|
||||
},
|
||||
};
|
||||
const readRef = async (options: {
|
||||
readonly gitdir?: string;
|
||||
readonly ref: string;
|
||||
}) => {
|
||||
let head: string;
|
||||
try {
|
||||
head = await resolveRef({ fs: referenceFs, dir: root, ...options });
|
||||
} catch (cause: unknown) {
|
||||
if (cleanupFailures.length > 0)
|
||||
throwCleanupFailures(failedRead?.cause ?? cause);
|
||||
if (failedRead !== undefined) throw failedRead.cause;
|
||||
throw cause;
|
||||
}
|
||||
if (cleanupFailures.length > 0)
|
||||
throwCleanupFailures(failedRead?.cause ?? cleanupFailures[0]?.cause);
|
||||
signal?.throwIfAborted();
|
||||
if (failedRead !== undefined) throw failedRead.cause;
|
||||
return head;
|
||||
};
|
||||
const dotgit = join(root, ".git");
|
||||
if (!(await lstat(dotgit)).isFile()) return readRef({ ref: "HEAD" });
|
||||
const pointer = (await readMetadataText(dotgit)).replace(/\r?\n$/u, "");
|
||||
if (!(await lstat(dotgit)).isFile())
|
||||
return resolveRef({ fs, dir: root, ref: "HEAD" });
|
||||
const pointer = (await readFile(dotgit, "utf8")).replace(/\r?\n$/u, "");
|
||||
if (!pointer.startsWith("gitdir: ") || pointer.length === 8)
|
||||
throw new Error("Invalid Git directory pointer");
|
||||
const gitdir = resolve(root, pointer.slice(8));
|
||||
@@ -202,22 +53,25 @@ const resolveSourceHead = async (
|
||||
try {
|
||||
commonDirectory = resolve(
|
||||
gitdir,
|
||||
(await readMetadataText(join(gitdir, "commondir"))).replace(
|
||||
(await readFile(join(gitdir, "commondir"), "utf8")).replace(
|
||||
/\r?\n$/u,
|
||||
"",
|
||||
),
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
if (errorCode(cause) !== "ENOENT") throw cause;
|
||||
return readRef({ gitdir, ref: "HEAD" });
|
||||
return resolveRef({ fs, dir: root, gitdir, ref: "HEAD" });
|
||||
}
|
||||
const head = (await readMetadataText(join(gitdir, "HEAD"))).trim();
|
||||
if (!head.startsWith("ref: ")) return readRef({ gitdir, ref: "HEAD" });
|
||||
const head = (await readFile(join(gitdir, "HEAD"), "utf8")).trim();
|
||||
if (!head.startsWith("ref: "))
|
||||
return resolveRef({ fs, dir: root, gitdir, ref: "HEAD" });
|
||||
const ref = head.slice(5);
|
||||
const privateRef = ["refs/bisect/", "refs/rewritten/", "refs/worktree/"].some(
|
||||
(prefix) => ref.startsWith(prefix),
|
||||
);
|
||||
return readRef({
|
||||
return resolveRef({
|
||||
fs,
|
||||
dir: root,
|
||||
gitdir: privateRef ? gitdir : commonDirectory,
|
||||
ref,
|
||||
});
|
||||
@@ -228,7 +82,4 @@ const errorCode = (cause: unknown): string | undefined =>
|
||||
? String(cause.code)
|
||||
: undefined;
|
||||
|
||||
const errorMessage = (cause: unknown): string =>
|
||||
cause instanceof Error ? cause.message : String(cause);
|
||||
|
||||
const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true;
|
||||
|
||||
@@ -1,158 +1,29 @@
|
||||
import type { Stats } from "node:fs";
|
||||
import { lstat, stat, type FileHandle } from "node:fs/promises";
|
||||
import type { FileHandle } from "node:fs/promises";
|
||||
|
||||
import {
|
||||
RegularFileAdmissionFailure,
|
||||
openRegularFile,
|
||||
sameRegularFileState,
|
||||
RegularFileChangedError,
|
||||
} from "../filesystem/RegularFile.js";
|
||||
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
|
||||
import { readBoundedFileBytes } from "../process/BoundedFileBytes.js";
|
||||
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import type { AnalysisCleanupObservation } from "../domain/analysisErrorBase.js";
|
||||
|
||||
/** Filesystem admission policy for one selected regular input. */
|
||||
export interface RegularFileReadOptions {
|
||||
readonly signal?: AbortSignal | undefined;
|
||||
readonly symlinks?: "follow" | "reject";
|
||||
}
|
||||
|
||||
export type RegularFileReadOutcome<Value> =
|
||||
| { readonly kind: "completed"; readonly value: Value }
|
||||
| { readonly kind: "failed"; readonly cause: unknown };
|
||||
|
||||
/** A close failed after reading; this error transfers the exact handle owner. */
|
||||
export class RegularFileCleanupFailure<Value = unknown> extends Error {
|
||||
constructor(
|
||||
readonly path: string,
|
||||
readonly owner: OwnedFileHandle,
|
||||
readonly outcome: RegularFileReadOutcome<Value>,
|
||||
readonly cleanupCause: unknown,
|
||||
) {
|
||||
super(`Regular file cleanup failed: ${path}`, {
|
||||
cause: outcome.kind === "failed" ? outcome.cause : cleanupCause,
|
||||
});
|
||||
this.name = "RegularFileCleanupFailure";
|
||||
}
|
||||
}
|
||||
|
||||
/** Retry a failed close through its retained owner and report any remaining cleanup. */
|
||||
export const retryRegularFileCleanup = async <Value>(
|
||||
failure: RegularFileCleanupFailure<Value>,
|
||||
resources: ArtifactResourceScope,
|
||||
): Promise<{
|
||||
readonly outcome: RegularFileReadOutcome<Value>;
|
||||
readonly cleanup?: AnalysisCleanupObservation;
|
||||
}> => {
|
||||
const released = await resources.release({
|
||||
kind: "file-handle",
|
||||
handle: failure.owner,
|
||||
resource: failure.path,
|
||||
});
|
||||
if (released.kind === "released") return { outcome: failure.outcome };
|
||||
return {
|
||||
outcome: failure.outcome,
|
||||
cleanup: {
|
||||
reason: `${failureMessage(failure.cleanupCause)}; retry: ${failureMessage(released.cause)}`,
|
||||
resources: [failure.path],
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
const failureMessage = (cause: unknown): string =>
|
||||
cause instanceof Error ? cause.message : String(cause);
|
||||
import { openRegularFile } from "../filesystem/RegularFile.js";
|
||||
|
||||
/** Admit one regular-file handle, retain it through the consumer, and always close it. */
|
||||
export const withRegularFile = async <Value>(
|
||||
path: string,
|
||||
read: (handle: FileHandle, stats: Stats) => Promise<Value>,
|
||||
options: RegularFileReadOptions = {},
|
||||
signal?: AbortSignal,
|
||||
): Promise<Value> => {
|
||||
const { signal } = options;
|
||||
let handle: FileHandle;
|
||||
try {
|
||||
handle = await openRegularFile(path, {
|
||||
symlinks: options.symlinks ?? "follow",
|
||||
signal,
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof RegularFileAdmissionFailure)) throw cause;
|
||||
throw new RegularFileCleanupFailure(
|
||||
path,
|
||||
cause.owner,
|
||||
{ kind: "failed", cause: cause.cause },
|
||||
cause.cleanupCause,
|
||||
);
|
||||
}
|
||||
const owner = new OwnedFileHandle(handle);
|
||||
let outcome: RegularFileReadOutcome<Value>;
|
||||
const handle = await openRegularFile(path, { symlinks: "follow", signal });
|
||||
try {
|
||||
const stats = await handle.stat();
|
||||
signal?.throwIfAborted();
|
||||
const value = await read(handle, stats);
|
||||
signal?.throwIfAborted();
|
||||
const [opened, currentPath] = await Promise.all([
|
||||
handle.stat(),
|
||||
options.symlinks === "reject" ? lstat(path) : stat(path),
|
||||
]);
|
||||
if (
|
||||
!sameRegularFileState(stats, opened) ||
|
||||
!sameRegularFileState(stats, currentPath)
|
||||
)
|
||||
throw new RegularFileChangedError(path);
|
||||
outcome = { kind: "completed", value };
|
||||
} catch (cause: unknown) {
|
||||
outcome = { kind: "failed", cause };
|
||||
return value;
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
try {
|
||||
await owner.close();
|
||||
} catch (cleanupCause: unknown) {
|
||||
throw new RegularFileCleanupFailure(path, owner, outcome, cleanupCause);
|
||||
}
|
||||
if (outcome.kind === "failed") throw outcome.cause;
|
||||
return outcome.value;
|
||||
};
|
||||
|
||||
/** Admit a regular file without waiting for a pipe, then read its verified handle. */
|
||||
export const readRegularFile = (
|
||||
path: string,
|
||||
options: RegularFileReadOptions = {},
|
||||
signal?: AbortSignal,
|
||||
): Promise<Buffer> =>
|
||||
withRegularFile(
|
||||
path,
|
||||
async (handle, stats) => {
|
||||
const bytes = await readBoundedFileBytes(
|
||||
handle,
|
||||
stats.size,
|
||||
options.signal,
|
||||
);
|
||||
if (bytes === undefined || bytes.length !== stats.size)
|
||||
throw new RegularFileChangedError(path);
|
||||
return bytes;
|
||||
},
|
||||
options,
|
||||
);
|
||||
|
||||
/** Read one admitted regular file as UTF-8 without waiting on a pipe. */
|
||||
export const readRegularFileText = async (
|
||||
path: string,
|
||||
options: RegularFileReadOptions = {},
|
||||
): Promise<string> => {
|
||||
try {
|
||||
return (await readRegularFile(path, options)).toString("utf8");
|
||||
} catch (cause: unknown) {
|
||||
if (
|
||||
cause instanceof RegularFileCleanupFailure &&
|
||||
cause.outcome.kind === "completed" &&
|
||||
Buffer.isBuffer(cause.outcome.value)
|
||||
)
|
||||
throw new RegularFileCleanupFailure(
|
||||
cause.path,
|
||||
cause.owner,
|
||||
{ kind: "completed", value: cause.outcome.value.toString("utf8") },
|
||||
cause.cleanupCause,
|
||||
);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
withRegularFile(path, (handle) => handle.readFile({ signal }), signal);
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
|
||||
import { ConfigurationError } from "../domain/configurationErrors.js";
|
||||
import { SUPPORTED_NODE_VERSION_PROSE } from "../domain/runtimeVersion.js";
|
||||
import { FakeSetupHost, options } from "./Setup.fixture.js";
|
||||
import { runSetup } from "./Setup.js";
|
||||
@@ -110,76 +107,6 @@ describe("setup workflow", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("setup failure cleanup projection", () => {
|
||||
it("projects a configuration failure together with failed host cleanup", async () => {
|
||||
const configPath = "/fixture/home/vscode/settings.json";
|
||||
const primary = new ConfigurationError("Client config could not be read", {
|
||||
settings: [
|
||||
{
|
||||
setting: configPath,
|
||||
constraint: "The selected config could not be read.",
|
||||
},
|
||||
],
|
||||
cleanup: {
|
||||
reason: "The admitted config handle remains open.",
|
||||
resources: [configPath],
|
||||
},
|
||||
});
|
||||
class SetupHostWithCleanupFailure extends FakeSetupHost {
|
||||
close = async (): Promise<string> =>
|
||||
"Setup could not release its client configuration resources.";
|
||||
}
|
||||
const host = new SetupHostWithCleanupFailure();
|
||||
host.clients = [{ name: "codex", configPath }];
|
||||
host.inspectClientConfiguration = async () => {
|
||||
throw primary;
|
||||
};
|
||||
|
||||
const failure = await runSetup(
|
||||
{ ...options(true), clientIds: ["codex"] },
|
||||
host,
|
||||
).then(
|
||||
() => undefined,
|
||||
(cause: unknown) => cause,
|
||||
);
|
||||
expect(failure).toBeInstanceOf(AnalysisError);
|
||||
if (!(failure instanceof AnalysisError))
|
||||
throw new Error("Expected setup to preserve a typed analysis failure");
|
||||
|
||||
const projected = projectAnalysisError(failure);
|
||||
expect(projected).toMatchObject({
|
||||
code: "cleanup_incomplete",
|
||||
details: {
|
||||
cleanup: "incomplete",
|
||||
resources: [configPath],
|
||||
diagnostics: {
|
||||
primary_error: {
|
||||
code: "cleanup_incomplete",
|
||||
details: { resources: [configPath] },
|
||||
},
|
||||
cleanup_error: {
|
||||
code: "cleanup_incomplete",
|
||||
details: { resources: [configPath] },
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(projected.details).toMatchObject({
|
||||
diagnostics: {
|
||||
primary_error: { message: expect.stringContaining("selected config") },
|
||||
cleanup_error: {
|
||||
details: {
|
||||
diagnostics: {
|
||||
reason:
|
||||
"Setup could not release its client configuration resources.",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("setup scoped readiness", () => {
|
||||
it("limits Hopper host checks to workflows that request Hopper", async () => {
|
||||
const platformHost = new FakeSetupHost("win32");
|
||||
|
||||
@@ -23,9 +23,6 @@ import {
|
||||
hostRemediation,
|
||||
systemSetupHost,
|
||||
} from "./SetupHost.js";
|
||||
import { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
|
||||
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
|
||||
|
||||
/** Discover, approve, and apply setup actions idempotently. */
|
||||
export const runSetup = async (
|
||||
@@ -51,17 +48,6 @@ export const runSetup = async (
|
||||
}
|
||||
|
||||
if (threw) {
|
||||
if (cleanupFailure !== undefined && primaryFailure instanceof AnalysisError)
|
||||
throw analysisErrorWithCleanupFailure(
|
||||
primaryFailure,
|
||||
new ProviderCleanupError(
|
||||
"setup",
|
||||
primaryFailure.cleanupResources,
|
||||
{ reason: cleanupFailure },
|
||||
{ operation: "setup" },
|
||||
),
|
||||
"setup",
|
||||
);
|
||||
if (cleanupFailure !== undefined)
|
||||
throw new AggregateError(
|
||||
[primaryFailure, new Error(cleanupFailure)],
|
||||
|
||||
@@ -11,22 +11,17 @@ import {
|
||||
serializeClientConfiguration,
|
||||
withClientServers,
|
||||
type ClientConfigurationDocument,
|
||||
type ClientRegistrationDialect,
|
||||
} from "./ClientConfigurationDocument.js";
|
||||
import { constants as fsConstants } from "node:fs";
|
||||
import { copyFile, mkdir, rm, writeFile } from "node:fs/promises";
|
||||
import { copyFile, mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { dirname } from "node:path";
|
||||
import writeFileAtomic from "write-file-atomic";
|
||||
import { clientRegistrationEnvironment } from "./ClientRegistrationEnvironment.js";
|
||||
|
||||
import { PRODUCT_IDENTITY } from "../identity.js";
|
||||
import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js";
|
||||
import { MCP_STARTUP_POLICY } from "../mcpStartupPolicy.js";
|
||||
import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js";
|
||||
import {
|
||||
readRegularFileText,
|
||||
RegularFileCleanupFailure,
|
||||
} from "./RegularFileRead.js";
|
||||
import type { ClientConfigurationFileError } from "./ClientConfigurationFiles.js";
|
||||
import type {
|
||||
ClientConfigurationInspection,
|
||||
ClientConfigurationResult,
|
||||
@@ -42,37 +37,25 @@ export const configureClientConfiguration = (
|
||||
): Promise<ClientConfigurationResult> => {
|
||||
if (client.configPathError !== undefined)
|
||||
return Promise.resolve({ status: "failed", reason: "path" });
|
||||
return configureClientDocument(client, environment, command);
|
||||
if (client.format === undefined || client.format === "unsupported")
|
||||
return Promise.resolve({ status: "failed", reason: "readback" });
|
||||
return configureClientDocument(client, environment, command, client.format);
|
||||
};
|
||||
|
||||
const configureClientDocument = async (
|
||||
client: SetupClient,
|
||||
environment: SetupProviderEnvironment,
|
||||
command: readonly string[],
|
||||
format: NonNullable<SetupClient["format"]>,
|
||||
): Promise<ClientConfigurationResult> => {
|
||||
const format = client.format;
|
||||
if (format === undefined || format === "unsupported")
|
||||
return { status: "failed", reason: "readback" };
|
||||
const files = await readClientConfigurationFiles(client);
|
||||
if (!files.ok) {
|
||||
throwCleanupFailure(files.error);
|
||||
return { status: "failed", reason: "readback" };
|
||||
}
|
||||
if (!files.ok) return { status: "failed", reason: "readback" };
|
||||
const policy = await readClientPolicyBlock(
|
||||
client,
|
||||
undefined,
|
||||
clientConfigurationDesired(
|
||||
client,
|
||||
environment,
|
||||
command,
|
||||
files.value.at(-1),
|
||||
),
|
||||
clientConfigurationDesired(client, environment, command, client.format),
|
||||
);
|
||||
if (!policy.ok) {
|
||||
throwCleanupFailure(policy.error);
|
||||
return { status: "failed", reason: "readback" };
|
||||
}
|
||||
if (policy.value !== undefined)
|
||||
if (!policy.ok || policy.value !== undefined)
|
||||
return { status: "failed", reason: "readback" };
|
||||
const transactionPath = await resolveClientConfigTransactionPath(
|
||||
client.configPath,
|
||||
@@ -81,9 +64,8 @@ const configureClientDocument = async (
|
||||
return { status: "failed", reason: "path" };
|
||||
let original: string | undefined;
|
||||
try {
|
||||
original = await readRegularFileText(transactionPath);
|
||||
original = await readFile(transactionPath, "utf8");
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof RegularFileCleanupFailure) throw cause;
|
||||
if (!isMissing(cause)) return { status: "failed", reason: "readback" };
|
||||
}
|
||||
let parsed: ClientConfigurationDocument;
|
||||
@@ -95,16 +77,11 @@ const configureClientDocument = async (
|
||||
const policy = await readClientPolicyBlock(
|
||||
client,
|
||||
parsed,
|
||||
clientConfigurationDesired(client, environment, command, parsed),
|
||||
clientConfigurationDesired(client, environment, command, parsed.dialect),
|
||||
);
|
||||
if (!policy.ok) {
|
||||
throwCleanupFailure(policy.error);
|
||||
return { status: "failed", reason: "readback" };
|
||||
}
|
||||
if (policy.value !== undefined)
|
||||
if (!policy.ok || policy.value !== undefined)
|
||||
return { status: "failed", reason: "readback" };
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof RegularFileCleanupFailure) throw cause;
|
||||
// Malformed existing configuration fails the readback gate.
|
||||
void cause;
|
||||
return { status: "failed", reason: "readback" };
|
||||
@@ -113,36 +90,9 @@ const configureClientDocument = async (
|
||||
client,
|
||||
environment,
|
||||
command,
|
||||
parsed,
|
||||
parsed.dialect,
|
||||
);
|
||||
if (registrationCurrent(parsed, desired)) return { status: "unchanged" };
|
||||
return persistClientConfiguration({
|
||||
client,
|
||||
transactionPath,
|
||||
format,
|
||||
original,
|
||||
parsed,
|
||||
desired,
|
||||
});
|
||||
};
|
||||
|
||||
interface ClientConfigurationUpdate {
|
||||
readonly client: SetupClient;
|
||||
readonly transactionPath: string;
|
||||
readonly format: NonNullable<SetupClient["format"]>;
|
||||
readonly original: string | undefined;
|
||||
readonly parsed: ClientConfigurationDocument;
|
||||
readonly desired: unknown;
|
||||
}
|
||||
|
||||
const persistClientConfiguration = async ({
|
||||
client,
|
||||
transactionPath,
|
||||
format,
|
||||
original,
|
||||
parsed,
|
||||
desired,
|
||||
}: ClientConfigurationUpdate): Promise<ClientConfigurationResult> => {
|
||||
const backupPath =
|
||||
original === undefined ? undefined : `${client.configPath}.rea.backup`;
|
||||
if (
|
||||
@@ -199,7 +149,7 @@ const persistClientConfiguration = async ({
|
||||
}
|
||||
try {
|
||||
const readback = parseClientConfiguration(
|
||||
await readRegularFileText(transactionPath),
|
||||
await readFile(transactionPath, "utf8"),
|
||||
format,
|
||||
);
|
||||
if (!registrationCurrent(readback, desired)) {
|
||||
@@ -207,7 +157,6 @@ const persistClientConfiguration = async ({
|
||||
return { status: "failed", reason: "readback" };
|
||||
}
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof RegularFileCleanupFailure) throw cause;
|
||||
// Readback failure restores the transaction before reporting.
|
||||
void cause;
|
||||
await restoreConfig(transactionPath, original);
|
||||
@@ -228,22 +177,21 @@ export const clientConfigurationAligned = async (
|
||||
if (client.configPathError !== undefined) return false;
|
||||
try {
|
||||
const files = await readClientConfigurationFiles(client);
|
||||
if (!files.ok) {
|
||||
throwCleanupFailure(files.error);
|
||||
return false;
|
||||
}
|
||||
const original = await readRegularFileText(client.configPath);
|
||||
if (!files.ok) return false;
|
||||
const original = await readFile(client.configPath, "utf8");
|
||||
const parsed = parseClientConfiguration(original, client.format);
|
||||
const policy = await readClientPolicyBlock(
|
||||
client,
|
||||
parsed,
|
||||
clientConfigurationDesired(client, providerEnvironment, command, parsed),
|
||||
clientConfigurationDesired(
|
||||
client,
|
||||
providerEnvironment,
|
||||
command,
|
||||
parsed.dialect,
|
||||
),
|
||||
);
|
||||
if (!policy.ok) {
|
||||
throwCleanupFailure(policy.error);
|
||||
return false;
|
||||
}
|
||||
return (
|
||||
policy.ok &&
|
||||
policy.value === undefined &&
|
||||
registrationCurrent(
|
||||
parsed,
|
||||
@@ -251,12 +199,11 @@ export const clientConfigurationAligned = async (
|
||||
client,
|
||||
providerEnvironment,
|
||||
command,
|
||||
parsed,
|
||||
parsed.dialect,
|
||||
),
|
||||
)
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof RegularFileCleanupFailure) throw cause;
|
||||
// Unreadable configuration is treated as not aligned so setup repairs it.
|
||||
void cause;
|
||||
return false;
|
||||
@@ -273,13 +220,7 @@ export const inspectClientConfiguration = async (
|
||||
return { status: "invalid", remediation: client.configPathError };
|
||||
if (client.format === "unsupported") return { status: "already_current" };
|
||||
const files = await readClientConfigurationFiles(client);
|
||||
if (!files.ok) {
|
||||
throwCleanupFailure(files.error);
|
||||
return {
|
||||
status: "invalid",
|
||||
remediation: files.error.detail,
|
||||
};
|
||||
}
|
||||
if (!files.ok) return { status: "invalid", remediation: files.error.detail };
|
||||
const policy = await readClientPolicyBlock(
|
||||
client,
|
||||
undefined,
|
||||
@@ -287,16 +228,11 @@ export const inspectClientConfiguration = async (
|
||||
client,
|
||||
providerEnvironment,
|
||||
command,
|
||||
files.value.at(-1),
|
||||
client.format,
|
||||
),
|
||||
);
|
||||
if (!policy.ok) {
|
||||
throwCleanupFailure(policy.error);
|
||||
return {
|
||||
status: "invalid",
|
||||
remediation: policy.error.detail,
|
||||
};
|
||||
}
|
||||
if (!policy.ok)
|
||||
return { status: "invalid", remediation: policy.error.detail };
|
||||
if (policy.value !== undefined)
|
||||
return { status: "invalid", remediation: policy.value };
|
||||
const transactionPath = await resolveClientConfigTransactionPath(
|
||||
@@ -310,13 +246,13 @@ export const inspectClientConfiguration = async (
|
||||
};
|
||||
let original: string;
|
||||
try {
|
||||
original = await readRegularFileText(transactionPath);
|
||||
original = await readFile(transactionPath, "utf8");
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof RegularFileCleanupFailure) throw cause;
|
||||
if (isMissing(cause)) return { status: "create" };
|
||||
return {
|
||||
status: "invalid",
|
||||
remediation: `The configuration file could not be read. Check its permissions before rerunning setup. ${cause instanceof Error ? cause.message : String(cause)}`,
|
||||
remediation:
|
||||
"The configuration file could not be read. Check its permissions before rerunning setup.",
|
||||
};
|
||||
}
|
||||
try {
|
||||
@@ -324,27 +260,26 @@ export const inspectClientConfiguration = async (
|
||||
const policy = await readClientPolicyBlock(
|
||||
client,
|
||||
parsed,
|
||||
clientConfigurationDesired(client, providerEnvironment, command, parsed),
|
||||
clientConfigurationDesired(
|
||||
client,
|
||||
providerEnvironment,
|
||||
command,
|
||||
parsed.dialect,
|
||||
),
|
||||
);
|
||||
if (!policy.ok) {
|
||||
throwCleanupFailure(policy.error);
|
||||
return {
|
||||
status: "invalid",
|
||||
remediation: policy.error.detail,
|
||||
};
|
||||
}
|
||||
if (!policy.ok)
|
||||
return { status: "invalid", remediation: policy.error.detail };
|
||||
if (policy.value !== undefined)
|
||||
return { status: "invalid", remediation: policy.value };
|
||||
const desired = clientConfigurationDesired(
|
||||
client,
|
||||
providerEnvironment,
|
||||
command,
|
||||
parsed,
|
||||
parsed.dialect,
|
||||
);
|
||||
if (registrationCurrent(parsed, desired))
|
||||
return { status: "already_current" };
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof RegularFileCleanupFailure) throw cause;
|
||||
// Malformed configuration is reported with the invalid remediation.
|
||||
void cause;
|
||||
return {
|
||||
@@ -361,11 +296,6 @@ export const inspectClientConfiguration = async (
|
||||
|
||||
const isMissing = (cause: unknown): boolean =>
|
||||
cause instanceof Error && "code" in cause && cause.code === "ENOENT";
|
||||
|
||||
const throwCleanupFailure = (error: ClientConfigurationFileError): void => {
|
||||
if (error.cleanupFailure !== undefined) throw error.cleanupFailure;
|
||||
};
|
||||
|
||||
const preserveConfigBackup = async (
|
||||
source: string,
|
||||
destination: string,
|
||||
@@ -407,20 +337,17 @@ const clientConfigurationDesired = (
|
||||
client: SetupClient,
|
||||
providerEnvironment: SetupProviderEnvironment,
|
||||
command: readonly string[],
|
||||
parsed?: ClientConfigurationDocument,
|
||||
format: ClientRegistrationDialect | undefined,
|
||||
) => {
|
||||
const format = parsed?.dialect ?? client.format ?? "json";
|
||||
// Preserve explicitly configured server settings, never the ambient process
|
||||
// environment. Newly discovered provider paths win only for their own keys.
|
||||
const environment = clientRegistrationEnvironment(
|
||||
parsed,
|
||||
providerEnvironment,
|
||||
const environment = Object.fromEntries(
|
||||
Object.entries(providerEnvironment).sort(([left], [right]) =>
|
||||
left.localeCompare(right),
|
||||
),
|
||||
);
|
||||
if (!environment.ok) throw environment.error;
|
||||
const registration = clientRegistrationEntry(
|
||||
format,
|
||||
format ?? "json",
|
||||
command.length === 0 ? [PRODUCT_IDENTITY.cliBinary, "mcp"] : command,
|
||||
environment.value,
|
||||
environment,
|
||||
);
|
||||
return {
|
||||
...registration,
|
||||
|
||||
+12
-110
@@ -1,9 +1,6 @@
|
||||
import { access } from "node:fs/promises";
|
||||
import { resolve } from "node:path";
|
||||
|
||||
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
|
||||
import { ConfigurationError } from "../domain/configurationErrors.js";
|
||||
import { PRODUCT_IDENTITY } from "../identity.js";
|
||||
import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js";
|
||||
import {
|
||||
@@ -33,10 +30,6 @@ import {
|
||||
inspectClientConfiguration,
|
||||
} from "./SetupClientConfiguration.js";
|
||||
import { setupInstallFailure } from "./SetupInstallFailure.js";
|
||||
import {
|
||||
RegularFileCleanupFailure,
|
||||
retryRegularFileCleanup,
|
||||
} from "./RegularFileRead.js";
|
||||
import { providerRegistrationEnvironment } from "./SetupRegistrationEnvironment.js";
|
||||
import type {
|
||||
SetupHost,
|
||||
@@ -105,7 +98,6 @@ export const systemSetupHost = (
|
||||
platform === "darwin"
|
||||
? systemMacHopperInstallHost(homeDirectory, environment)
|
||||
: undefined;
|
||||
const resources = new ArtifactResourceScope();
|
||||
return {
|
||||
platform,
|
||||
homeDirectory,
|
||||
@@ -118,7 +110,7 @@ export const systemSetupHost = (
|
||||
nodeVersion: process.versions.node,
|
||||
macosVersion: () => doctorHost.macosVersion(),
|
||||
linuxDistribution: readLinuxDistribution,
|
||||
close: () => closeSetupResources(resources, macHopperHost),
|
||||
close: () => macHopperHost?.close() ?? Promise.resolve(undefined),
|
||||
initialSetupState: async (
|
||||
scope?: DoctorScope,
|
||||
): Promise<SetupInitialState> => {
|
||||
@@ -170,7 +162,17 @@ export const systemSetupHost = (
|
||||
detectedClients: () => detectClients(homeDirectory, platform, environment),
|
||||
supportedClients: () =>
|
||||
Promise.resolve(supportedClients(homeDirectory, platform, environment)),
|
||||
...clientConfigurationOperations(resources),
|
||||
configureClient: (client, providerEnvironment, command) =>
|
||||
client.format === "unsupported"
|
||||
? Promise.resolve({ status: "skipped" })
|
||||
: configureClientConfiguration(client, providerEnvironment, command),
|
||||
clientNeedsConfigure: (client, providerEnvironment, command) =>
|
||||
client.format === "unsupported"
|
||||
? Promise.resolve(false)
|
||||
: clientConfigurationAligned(client, providerEnvironment, command).then(
|
||||
(aligned) => !aligned,
|
||||
),
|
||||
inspectClientConfiguration: inspectClientConfiguration,
|
||||
skillNeedsInstall: (clientIds) =>
|
||||
canonicalSkillNeedsInstall(
|
||||
homeDirectory,
|
||||
@@ -184,106 +186,6 @@ export const systemSetupHost = (
|
||||
};
|
||||
};
|
||||
|
||||
const clientConfigurationOperations = (resources: ArtifactResourceScope) => ({
|
||||
configureClient: (
|
||||
client: SetupClient,
|
||||
providerEnvironment: SetupProviderEnvironment,
|
||||
command: readonly string[],
|
||||
) =>
|
||||
runClientConfigurationOperation(resources, async () =>
|
||||
client.format === "unsupported"
|
||||
? { status: "skipped" as const }
|
||||
: configureClientConfiguration(client, providerEnvironment, command),
|
||||
),
|
||||
clientNeedsConfigure: (
|
||||
client: SetupClient,
|
||||
providerEnvironment: SetupProviderEnvironment,
|
||||
command: readonly string[],
|
||||
) =>
|
||||
runClientConfigurationOperation(resources, async () =>
|
||||
client.format === "unsupported"
|
||||
? false
|
||||
: !(await clientConfigurationAligned(
|
||||
client,
|
||||
providerEnvironment,
|
||||
command,
|
||||
)),
|
||||
),
|
||||
inspectClientConfiguration: (
|
||||
client: SetupClient,
|
||||
providerEnvironment: SetupProviderEnvironment,
|
||||
command: readonly string[],
|
||||
) =>
|
||||
runClientConfigurationOperation(resources, () =>
|
||||
inspectClientConfiguration(client, providerEnvironment, command),
|
||||
),
|
||||
});
|
||||
|
||||
const closeSetupResources = async (
|
||||
resources: ArtifactResourceScope,
|
||||
macHopperHost: ReturnType<typeof systemMacHopperInstallHost> | undefined,
|
||||
): Promise<string | undefined> => {
|
||||
const [resourceCleanup, hopperCleanup] = await Promise.allSettled([
|
||||
resources.close(),
|
||||
Promise.resolve().then(() => macHopperHost?.close()),
|
||||
]);
|
||||
const failures: string[] = [];
|
||||
if (hopperCleanup.status === "rejected")
|
||||
failures.push(
|
||||
`Hopper host cleanup failed: ${errorMessage(hopperCleanup.reason)}`,
|
||||
);
|
||||
else if (hopperCleanup.value !== undefined)
|
||||
failures.push(hopperCleanup.value);
|
||||
if (resourceCleanup.status === "rejected")
|
||||
failures.push(errorMessage(resourceCleanup.reason));
|
||||
return failures.length === 0 ? undefined : failures.join("; ");
|
||||
};
|
||||
|
||||
const runClientConfigurationOperation = async <Value>(
|
||||
resources: ArtifactResourceScope,
|
||||
operation: () => Promise<Value>,
|
||||
): Promise<Value> => {
|
||||
try {
|
||||
return await resources.run(async () => {
|
||||
try {
|
||||
return await operation();
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof RegularFileCleanupFailure)) throw cause;
|
||||
const cleanup = await retryRegularFileCleanup(cause, resources);
|
||||
const primary =
|
||||
cleanup.outcome.kind === "failed"
|
||||
? cleanup.outcome.cause
|
||||
: cause.cleanupCause;
|
||||
throw configurationCleanupError(
|
||||
`Could not safely access client configuration at ${cause.path}: ${errorMessage(primary)}`,
|
||||
cause,
|
||||
cleanup.cleanup,
|
||||
);
|
||||
}
|
||||
});
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof ArtifactReaderFailure)
|
||||
throw configurationCleanupError(cause.message, cause, cause.cleanup);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
|
||||
const configurationCleanupError = (
|
||||
message: string,
|
||||
cause: unknown,
|
||||
cleanup:
|
||||
| { readonly reason: string; readonly resources: readonly string[] }
|
||||
| undefined,
|
||||
): ConfigurationError =>
|
||||
new ConfigurationError(message, {
|
||||
cause,
|
||||
settings: [{ setting: "client configuration", constraint: message }],
|
||||
...(cleanup === undefined ? {} : { cleanup }),
|
||||
});
|
||||
|
||||
const errorMessage = (cause: unknown): string =>
|
||||
cause instanceof Error ? cause.message : String(cause);
|
||||
|
||||
/** Detect supported agents from their config files or stable installation markers. */
|
||||
export const detectClients = async (
|
||||
home: string,
|
||||
|
||||
@@ -312,8 +312,8 @@ const clientConfigurationDetail = (
|
||||
.map(([name, value]) => `${name}=${value}`);
|
||||
const environmentDetail =
|
||||
entries.length === 0
|
||||
? " No additional provider environment variables. Existing server environment overrides are retained."
|
||||
: ` Detected provider environment: ${entries.join(", ")}. Existing server environment overrides are retained; detected values replace matching keys.`;
|
||||
? " No additional environment variables."
|
||||
: ` Environment: ${entries.join(", ")}.`;
|
||||
const hopperDetail = installHopper
|
||||
? environment.HOPPER_LAUNCHER_PATH === undefined
|
||||
? " If this plan installs Hopper, REA will add HOPPER_LAUNCHER_PATH using the launcher path reported by that installation."
|
||||
|
||||
@@ -4,13 +4,14 @@ import { dirname, join } from "node:path";
|
||||
import writeFileAtomic from "write-file-atomic";
|
||||
|
||||
import { PRODUCT_IDENTITY } from "../identity.js";
|
||||
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
|
||||
import { readRegularFileText } from "./RegularFileRead.js";
|
||||
import {
|
||||
NonRegularFileReadError,
|
||||
openRegularFile,
|
||||
} from "../filesystem/RegularFile.js";
|
||||
import { clientSkillDirectories } from "./SupportedClients.js";
|
||||
|
||||
export const MANAGED_SKILL_FILES = [
|
||||
"SKILL.md",
|
||||
"references/connection-and-recovery.md",
|
||||
"references/native-and-artifacts.md",
|
||||
"references/javascript-applications.md",
|
||||
"references/android-applications.md",
|
||||
@@ -22,8 +23,14 @@ export const isManagedSkillManifest = (content: string): boolean =>
|
||||
content.startsWith(`---\nname: ${PRODUCT_IDENTITY.skillName}\n`) ||
|
||||
content.startsWith(`---\r\nname: ${PRODUCT_IDENTITY.skillName}\r\n`);
|
||||
|
||||
export const readSkillFile = (path: string): Promise<string> =>
|
||||
readRegularFileText(path, { symlinks: "reject" });
|
||||
export const readSkillFile = async (path: string): Promise<string> => {
|
||||
const handle = await openRegularFile(path, { symlinks: "reject" });
|
||||
try {
|
||||
return await handle.readFile("utf8");
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
};
|
||||
|
||||
interface CanonicalSkillFile {
|
||||
readonly content: string;
|
||||
@@ -192,7 +199,7 @@ export const installCanonicalSkill = async (
|
||||
for (const { destination, content } of changed)
|
||||
await writeText(destination, content);
|
||||
for (const { destination, content } of changed)
|
||||
if ((await readSkillFile(destination)) !== content)
|
||||
if ((await readFile(destination, "utf8")) !== content)
|
||||
throw new Error(`skill readback mismatch: ${destination}`);
|
||||
return "installed";
|
||||
} catch (cause: unknown) {
|
||||
@@ -224,7 +231,7 @@ export const readInstalledSkillIdentity = async (
|
||||
let content: string | undefined;
|
||||
for (const destination of destinations) {
|
||||
try {
|
||||
content = await readSkillFile(join(destination.path, "SKILL.md"));
|
||||
content = await readFile(join(destination.path, "SKILL.md"), "utf8");
|
||||
break;
|
||||
} catch {
|
||||
// Another selected copy may provide metadata; canonical validation below
|
||||
|
||||
@@ -6,7 +6,7 @@ import {
|
||||
withClientServers,
|
||||
type ClientConfigurationDocument,
|
||||
} from "./ClientConfigurationDocument.js";
|
||||
import { copyFile, lstat, realpath, rm } from "node:fs/promises";
|
||||
import { copyFile, lstat, readFile, realpath, rm } from "node:fs/promises";
|
||||
import { constants as fsConstants } from "node:fs";
|
||||
import { homeDirectoryFromEnvironment } from "../config/homeDirectory.js";
|
||||
import { join } from "node:path";
|
||||
@@ -23,7 +23,6 @@ import {
|
||||
} from "./SetupSkill.js";
|
||||
import { isOwnedClientRegistrationCommand } from "./ClientRegistrationIdentity.js";
|
||||
import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js";
|
||||
import { readRegularFileText } from "./RegularFileRead.js";
|
||||
import {
|
||||
manualRegistrationRemediation,
|
||||
supportedClients,
|
||||
@@ -50,7 +49,7 @@ export interface UninstallFileSystem {
|
||||
}
|
||||
|
||||
const systemFileSystem: UninstallFileSystem = {
|
||||
readText: (path) => readRegularFileText(path),
|
||||
readText: (path) => readFile(path, "utf8"),
|
||||
readRegularText: readSkillFile,
|
||||
copy: (source, destination) =>
|
||||
copyFile(source, destination, fsConstants.COPYFILE_EXCL),
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { z } from "zod";
|
||||
|
||||
import { PRODUCT_IDENTITY } from "../identity.js";
|
||||
@@ -13,7 +14,6 @@ import { supportedClients } from "./SupportedClients.js";
|
||||
import { existingSkillDestinations } from "./SetupSkill.js";
|
||||
import type { SetupAction } from "./SetupTypes.js";
|
||||
import type { Result } from "../domain/result.js";
|
||||
import { readRegularFileText } from "./RegularFileRead.js";
|
||||
|
||||
/** Existing REA integrations selected for maintenance, without discovering new targets. */
|
||||
export interface MaintenanceScope {
|
||||
@@ -73,7 +73,7 @@ export const existingMaintenanceScope = async (
|
||||
const client = supported.find(({ name }) => name === registration.client);
|
||||
if (client === undefined) continue;
|
||||
const parsed = parseClientConfiguration(
|
||||
await readRegularFileText(client.configPath),
|
||||
await readFile(client.configPath, "utf8"),
|
||||
client.format,
|
||||
);
|
||||
const enabled = z
|
||||
|
||||
@@ -4,7 +4,6 @@ import { isAbsolute } from "node:path";
|
||||
import { z } from "zod";
|
||||
|
||||
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
|
||||
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
|
||||
import { publishWebScripts } from "../browser/assets/PublishWebScripts.js";
|
||||
import { selectScriptCapture } from "../browser/assets/ScriptCaptureAdapters.js";
|
||||
import {
|
||||
@@ -15,11 +14,9 @@ import {
|
||||
} from "../domain/analysisErrorCore.js";
|
||||
import type { AnalysisError } from "../domain/analysisErrorBase.js";
|
||||
import { createEvidence, type Evidence } from "../domain/evidence.js";
|
||||
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
|
||||
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
|
||||
import { jsonValueSchema } from "../domain/jsonValue.js";
|
||||
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
|
||||
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { safeParseJson } from "../domain/safeJson.js";
|
||||
import {
|
||||
@@ -29,41 +26,19 @@ import {
|
||||
import { WebScriptExportError } from "../domain/webScriptExportError.js";
|
||||
import type { ExecutionOptions } from "./AnalysisProvider.js";
|
||||
import { WEB_SCRIPT_EXPORT_PROVIDER as PROVIDER } from "./InvestigationProviders.js";
|
||||
import {
|
||||
readRegularFile,
|
||||
RegularFileCleanupFailure,
|
||||
retryRegularFileCleanup,
|
||||
} from "./RegularFileRead.js";
|
||||
import {
|
||||
NonRegularFileReadError,
|
||||
RegularFileChangedError,
|
||||
} from "../filesystem/RegularFile.js";
|
||||
import { readRegularFile } from "./RegularFileRead.js";
|
||||
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
|
||||
|
||||
const OPERATION = "export_web_scripts";
|
||||
|
||||
interface CaptureRead {
|
||||
readonly bytes: Buffer;
|
||||
readonly cleanup?: {
|
||||
readonly reason: string;
|
||||
readonly incomplete: boolean;
|
||||
readonly cause: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
interface PreparedCapture {
|
||||
readonly bytes: Buffer;
|
||||
readonly loaded: ReturnType<typeof selectScriptCapture>;
|
||||
}
|
||||
|
||||
/** Export one local capture through the shared CLI/MCP application workflow. */
|
||||
export const exportWebScripts = async (
|
||||
rawInput: unknown,
|
||||
resources: ArtifactResourceScope,
|
||||
options: ExecutionOptions = {},
|
||||
): Promise<Result<Evidence, AnalysisError>> => {
|
||||
const input = exportWebScriptsInputSchema.safeParse(rawInput);
|
||||
return input.success
|
||||
? exportWebScriptsValidated(input.data, resources, options)
|
||||
? exportWebScriptsValidated(input.data, options)
|
||||
: err(
|
||||
analysisInputErrorFromIssues(OPERATION, input.error.issues, rawInput),
|
||||
);
|
||||
@@ -72,7 +47,6 @@ export const exportWebScripts = async (
|
||||
/** Publish input already parsed by a named adapter contract. */
|
||||
export const exportWebScriptsValidated = async (
|
||||
input: ExportWebScriptsInput,
|
||||
resources: ArtifactResourceScope,
|
||||
options: ExecutionOptions = {},
|
||||
): Promise<Result<Evidence, AnalysisError>> => {
|
||||
if (!isAbsolute(input.capture_path) || !isAbsolute(input.output_directory))
|
||||
@@ -88,19 +62,19 @@ export const exportWebScriptsValidated = async (
|
||||
);
|
||||
try {
|
||||
options.signal?.throwIfAborted();
|
||||
const prepared = await prepareCapture(
|
||||
input.capture_path,
|
||||
resources,
|
||||
options.signal,
|
||||
);
|
||||
if (!prepared.ok) return prepared;
|
||||
const { bytes, loaded } = prepared.value;
|
||||
const read = await readCapture(input.capture_path, options.signal);
|
||||
if (!read.ok) return read;
|
||||
const bytes = read.value;
|
||||
const loaded = parseCapture(bytes);
|
||||
if (!loaded.ok) return loaded;
|
||||
options.signal?.throwIfAborted();
|
||||
const sha256 = createHash("sha256").update(bytes).digest("hex");
|
||||
const result = await publishWebScripts(input, loaded, sha256, {
|
||||
resources,
|
||||
...(options.signal === undefined ? {} : { signal: options.signal }),
|
||||
});
|
||||
const result = await publishWebScripts(
|
||||
input,
|
||||
loaded.value,
|
||||
sha256,
|
||||
options.signal,
|
||||
);
|
||||
return ok(
|
||||
createEvidence(
|
||||
{ path: input.capture_path, sha256, format: "file" },
|
||||
@@ -119,109 +93,55 @@ export const exportWebScriptsValidated = async (
|
||||
limitations: result.limitations,
|
||||
locations: [{ kind: "artifact-path", path: result.manifest.path }],
|
||||
evidenceLinks:
|
||||
loaded.sourceEvidenceId === null ? [] : [loaded.sourceEvidenceId],
|
||||
loaded.value.sourceEvidenceId === null
|
||||
? []
|
||||
: [loaded.value.sourceEvidenceId],
|
||||
},
|
||||
),
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
return err(mapExportFailure(cause, input, options.signal));
|
||||
}
|
||||
};
|
||||
|
||||
const prepareCapture = async (
|
||||
path: string,
|
||||
resources: ArtifactResourceScope,
|
||||
signal: AbortSignal | undefined,
|
||||
): Promise<Result<PreparedCapture, AnalysisError>> => {
|
||||
const read = await resources.run(() => readCapture(path, resources, signal));
|
||||
if (!read.ok) return read;
|
||||
const cleanup = read.value.cleanup;
|
||||
const loaded = parseCapture(
|
||||
read.value.bytes,
|
||||
cleanup === undefined || cleanup.incomplete
|
||||
? undefined
|
||||
: { cause: cleanup.cause },
|
||||
);
|
||||
if (!loaded.ok)
|
||||
if (cause instanceof WebScriptExportError) return err(cause);
|
||||
if (options.signal?.aborted === true)
|
||||
return err(new AnalysisCancelledError(OPERATION));
|
||||
if (cause instanceof ArtifactReaderFailure)
|
||||
return err(
|
||||
new WebScriptExportError(
|
||||
cause.reason,
|
||||
input.output_directory,
|
||||
cause.message,
|
||||
),
|
||||
);
|
||||
if (cause instanceof z.ZodError)
|
||||
return err(new AnalysisOutputError(OPERATION, cause.message, { cause }));
|
||||
if (
|
||||
cause instanceof Error &&
|
||||
"code" in cause &&
|
||||
typeof cause.code === "string"
|
||||
)
|
||||
return err(
|
||||
new WebScriptExportError(
|
||||
"io",
|
||||
`${input.capture_path} → ${input.output_directory}`,
|
||||
cause.message,
|
||||
),
|
||||
);
|
||||
return err(
|
||||
cleanup?.incomplete === true
|
||||
? incompleteCaptureCleanup(loaded.error, path, cleanup)
|
||||
: loaded.error,
|
||||
);
|
||||
if (cleanup !== undefined) {
|
||||
const primary = new WebScriptExportError(
|
||||
"io",
|
||||
path,
|
||||
`Capture handle close failed: ${cleanup.reason}`,
|
||||
{ cause: cleanup.cause },
|
||||
);
|
||||
return err(
|
||||
cleanup.incomplete
|
||||
? incompleteCaptureCleanup(primary, path, cleanup)
|
||||
: primary,
|
||||
new ProviderAdapterError(PROVIDER.id, OPERATION, {
|
||||
cause,
|
||||
diagnostics: {
|
||||
capture_path: input.capture_path,
|
||||
output_directory: input.output_directory,
|
||||
error_name: cause instanceof Error ? cause.name : "UnknownError",
|
||||
error_message:
|
||||
cause instanceof Error
|
||||
? cause.message
|
||||
: typeof cause === "string"
|
||||
? cause
|
||||
: "Unknown script export failure",
|
||||
},
|
||||
}),
|
||||
);
|
||||
}
|
||||
return ok({ bytes: read.value.bytes, loaded: loaded.value });
|
||||
};
|
||||
|
||||
const mapExportFailure = (
|
||||
cause: unknown,
|
||||
input: ExportWebScriptsInput,
|
||||
signal: AbortSignal | undefined,
|
||||
): AnalysisError => {
|
||||
if (cause instanceof WebScriptExportError) return cause;
|
||||
if (cause instanceof ArtifactReaderFailure && cause.cleanup !== undefined)
|
||||
return analysisErrorWithCleanupFailure(
|
||||
new WebScriptExportError(
|
||||
cause.reason,
|
||||
input.output_directory,
|
||||
cause.message,
|
||||
{ cause },
|
||||
),
|
||||
new ProviderCleanupError(
|
||||
PROVIDER.id,
|
||||
cause.cleanup.resources,
|
||||
{ reason: cause.cleanup.reason },
|
||||
{ cause, operation: OPERATION },
|
||||
),
|
||||
OPERATION,
|
||||
);
|
||||
if (signal?.aborted === true)
|
||||
return new AnalysisCancelledError(OPERATION, { cause });
|
||||
if (cause instanceof ArtifactReaderFailure)
|
||||
return new WebScriptExportError(
|
||||
cause.reason,
|
||||
input.output_directory,
|
||||
cause.message,
|
||||
{ cause },
|
||||
);
|
||||
if (cause instanceof z.ZodError)
|
||||
return new AnalysisOutputError(OPERATION, cause.message, { cause });
|
||||
if (
|
||||
cause instanceof Error &&
|
||||
"code" in cause &&
|
||||
typeof cause.code === "string"
|
||||
)
|
||||
return new WebScriptExportError(
|
||||
"io",
|
||||
`${input.capture_path} → ${input.output_directory}`,
|
||||
cause.message,
|
||||
{ cause },
|
||||
);
|
||||
return new ProviderAdapterError(PROVIDER.id, OPERATION, {
|
||||
cause,
|
||||
diagnostics: {
|
||||
capture_path: input.capture_path,
|
||||
output_directory: input.output_directory,
|
||||
error_name: cause instanceof Error ? cause.name : "UnknownError",
|
||||
error_message:
|
||||
cause instanceof Error
|
||||
? cause.message
|
||||
: typeof cause === "string"
|
||||
? cause
|
||||
: "Unknown script export failure",
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -230,115 +150,44 @@ const mapExportFailure = (
|
||||
*/
|
||||
const readCapture = async (
|
||||
path: string,
|
||||
resources: ArtifactResourceScope,
|
||||
signal: AbortSignal | undefined,
|
||||
): Promise<Result<CaptureRead, AnalysisError>> => {
|
||||
): Promise<Result<Buffer, AnalysisError>> => {
|
||||
try {
|
||||
return ok({ bytes: await readRegularFile(path, { signal }) });
|
||||
return ok(await readRegularFile(path, signal));
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof RegularFileCleanupFailure) {
|
||||
const retry = await retryRegularFileCleanup(cause, resources);
|
||||
const cleanup = {
|
||||
reason: retry.cleanup?.reason ?? errorMessage(cause.cleanupCause),
|
||||
incomplete: retry.cleanup !== undefined,
|
||||
cause: cause.cleanupCause,
|
||||
};
|
||||
if (cause.outcome.kind === "completed")
|
||||
return ok({ bytes: cause.outcome.value as Buffer, cleanup });
|
||||
const primary = captureReadFailure(
|
||||
path,
|
||||
cause.outcome.cause,
|
||||
signal,
|
||||
retry.cleanup === undefined ? { cause: cause.cleanupCause } : undefined,
|
||||
);
|
||||
const code =
|
||||
cause instanceof Error && "code" in cause ? String(cause.code) : "";
|
||||
if (code === "EACCES" || code === "EPERM")
|
||||
return err(
|
||||
retry.cleanup === undefined
|
||||
? primary
|
||||
: incompleteCaptureCleanup(primary, path, cleanup),
|
||||
new AnalysisAccessDeniedError(OPERATION, path, code, { cause }),
|
||||
);
|
||||
}
|
||||
return err(captureReadFailure(path, cause, signal));
|
||||
if (
|
||||
cause instanceof NonRegularFileReadError ||
|
||||
code === "ENOENT" ||
|
||||
code === "ENOTDIR" ||
|
||||
code === "EISDIR" ||
|
||||
code === "ENXIO"
|
||||
)
|
||||
return err(
|
||||
new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: ["capture_path"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
cause instanceof NonRegularFileReadError && code !== "EISDIR"
|
||||
? cause.message
|
||||
: code === "EISDIR"
|
||||
? `Selected capture is a directory, not a file: ${path}`
|
||||
: `Selected capture could not be read (${code}): ${path}`,
|
||||
},
|
||||
]),
|
||||
);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
|
||||
const captureReadFailure = (
|
||||
path: string,
|
||||
cause: unknown,
|
||||
signal: AbortSignal | undefined,
|
||||
closeFailure?: { readonly cause: unknown },
|
||||
): AnalysisError => {
|
||||
const causeWithClose =
|
||||
closeFailure === undefined
|
||||
? cause
|
||||
: new AggregateError(
|
||||
[cause, closeFailure.cause],
|
||||
"Capture read and file-handle close both failed",
|
||||
{ cause },
|
||||
);
|
||||
if (signal?.aborted === true)
|
||||
return new AnalysisCancelledError(OPERATION, { cause: causeWithClose });
|
||||
const code =
|
||||
cause instanceof Error && "code" in cause ? String(cause.code) : "";
|
||||
if (code === "EACCES" || code === "EPERM")
|
||||
return new AnalysisAccessDeniedError(OPERATION, path, code, {
|
||||
cause: causeWithClose,
|
||||
});
|
||||
if (
|
||||
cause instanceof NonRegularFileReadError ||
|
||||
cause instanceof RegularFileChangedError ||
|
||||
code === "ENOENT" ||
|
||||
code === "ENOTDIR" ||
|
||||
code === "EISDIR" ||
|
||||
code === "ENXIO"
|
||||
)
|
||||
return new AnalysisInputError(OPERATION, { cause: causeWithClose }, [
|
||||
{
|
||||
path: ["capture_path"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
cause instanceof NonRegularFileReadError && code !== "EISDIR"
|
||||
? cause.message
|
||||
: code === "EISDIR"
|
||||
? `Selected capture is a directory, not a file: ${path}`
|
||||
: `Selected capture could not be read (${code}): ${path}`,
|
||||
},
|
||||
]);
|
||||
if (code !== "")
|
||||
return new WebScriptExportError("io", path, errorMessage(cause), {
|
||||
cause: causeWithClose,
|
||||
});
|
||||
return new ProviderAdapterError(PROVIDER.id, OPERATION, {
|
||||
cause: causeWithClose,
|
||||
diagnostics: {
|
||||
capture_path: path,
|
||||
error_name: cause instanceof Error ? cause.name : "UnknownError",
|
||||
error_message: cause instanceof Error ? cause.message : String(cause),
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
const incompleteCaptureCleanup = (
|
||||
primary: AnalysisError,
|
||||
path: string,
|
||||
cleanup: NonNullable<CaptureRead["cleanup"]>,
|
||||
): AnalysisError =>
|
||||
analysisErrorWithCleanupFailure(
|
||||
primary,
|
||||
new ProviderCleanupError(
|
||||
PROVIDER.id,
|
||||
[path],
|
||||
{ reason: cleanup.reason },
|
||||
{ cause: cleanup.cause, operation: OPERATION },
|
||||
),
|
||||
OPERATION,
|
||||
);
|
||||
|
||||
const errorMessage = (cause: unknown): string =>
|
||||
cause instanceof Error ? cause.message : String(cause);
|
||||
|
||||
const parseCapture = (
|
||||
bytes: Buffer,
|
||||
closeFailure?: { readonly cause: unknown },
|
||||
): Result<ReturnType<typeof selectScriptCapture>, AnalysisError> => {
|
||||
try {
|
||||
const text = new TextDecoder("utf-8", {
|
||||
@@ -349,16 +198,8 @@ const parseCapture = (
|
||||
if (!json.ok) throw new TypeError(json.error);
|
||||
return ok(selectScriptCapture(json.value));
|
||||
} catch (cause: unknown) {
|
||||
const causeWithClose =
|
||||
closeFailure === undefined
|
||||
? cause
|
||||
: new AggregateError(
|
||||
[cause, closeFailure.cause],
|
||||
"Capture parse and file-handle close both failed",
|
||||
{ cause },
|
||||
);
|
||||
return err(
|
||||
new AnalysisInputError(OPERATION, { cause: causeWithClose }, [
|
||||
new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: ["capture_path"],
|
||||
reason: "invalid_format",
|
||||
|
||||
@@ -14,7 +14,7 @@ import type { Result } from "../../domain/result.js";
|
||||
export interface AdbDeviceAnalysisPort {
|
||||
/** Probe the selected adb binary without contacting devices. */
|
||||
inspectAvailability(signal?: AbortSignal): Promise<ProviderAvailability>;
|
||||
/** Terminally stop admission, drain work, and retry retained package rollback. */
|
||||
/** The provider holds no long-lived resources; safe to call always. */
|
||||
close(): Promise<void>;
|
||||
execute(
|
||||
request: AdbRequest,
|
||||
|
||||
@@ -11,7 +11,6 @@ import {
|
||||
} from "../../../tests/fixtures/analysisView.js";
|
||||
import {
|
||||
inspectAnalysisView,
|
||||
inspectAnalysisViewValidated,
|
||||
summarizeRetainedAnalysis,
|
||||
} from "./AnalysisViewService.js";
|
||||
import type { Evidence } from "../../domain/evidence.js";
|
||||
@@ -59,35 +58,6 @@ it.each(["/fixtures/unknown.exe", ""])(
|
||||
},
|
||||
);
|
||||
|
||||
it("reports inline Evidence identity failures at the complete source path", () => {
|
||||
const parent = analysisViewLayoutEvidence();
|
||||
const input = {
|
||||
source: {
|
||||
kind: "inline",
|
||||
evidence: { ...parent, evidence_id: `ev_${"0".repeat(64)}` },
|
||||
},
|
||||
view: { kind: "summary" },
|
||||
} as const;
|
||||
|
||||
for (const result of [
|
||||
inspectAnalysisView(input),
|
||||
inspectAnalysisViewValidated(input),
|
||||
])
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
_tag: "AnalysisInputError",
|
||||
issues: [
|
||||
{
|
||||
path: ["source", "evidence", "evidence_id"],
|
||||
reason: "invalid_value",
|
||||
message: "Evidence semantic identifier does not match its record",
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("projects a small native view from an authenticated >10 MiB retained dossier", () => {
|
||||
const original = functionDossierSchema.parse(ghidraFunctionDossier());
|
||||
const parent = createEvidence(
|
||||
|
||||
@@ -132,11 +132,8 @@ const resolveAnalysisViewSource = (
|
||||
return err(
|
||||
analysisInputErrorFromIssues(
|
||||
INSPECT_ANALYSIS_VIEW_OPERATION,
|
||||
cause.issues.map((issue) => ({
|
||||
...issue,
|
||||
path: ["source", "evidence", ...issue.path],
|
||||
})),
|
||||
{ source },
|
||||
cause.issues,
|
||||
source.evidence,
|
||||
{ cause },
|
||||
),
|
||||
);
|
||||
|
||||
@@ -2,10 +2,9 @@ import type { ApplicationNode } from "../../domain/javascript/javascriptApplicat
|
||||
import { sha256Text } from "../../domain/javascript/javascriptStaticAnalysisHelpers.js";
|
||||
import type { JavaScriptSourceRange } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
||||
import {
|
||||
sourceNodeFor,
|
||||
type JavaScriptArtifactGraphContext,
|
||||
type JavaScriptArtifactGraphCoverage,
|
||||
import type {
|
||||
JavaScriptArtifactGraphContext,
|
||||
JavaScriptArtifactGraphCoverage,
|
||||
} from "./JavaScriptArtifactGraphContext.js";
|
||||
import {
|
||||
astObservationEvidence,
|
||||
@@ -39,11 +38,10 @@ export const electronFindingSourceNode = (
|
||||
context: JavaScriptArtifactGraphContext,
|
||||
file: JavaScriptArtifactFile,
|
||||
moduleKey: string | null,
|
||||
runtime?: string,
|
||||
): ApplicationNode | undefined =>
|
||||
moduleKey === null
|
||||
? (context.assetNodes.get(file.path) ?? context.fileNodes.get(file.path))
|
||||
: sourceNodeFor(context, file.path, moduleKey, runtime);
|
||||
: context.moduleNodes.get(`${file.path}\0${moduleKey}`);
|
||||
|
||||
/** Observation-scoped identity shared safely across different artifact files. */
|
||||
export const electronObservationIdentity = (
|
||||
|
||||
@@ -73,7 +73,6 @@ const addIpcRecord = (
|
||||
context,
|
||||
record.file,
|
||||
record.finding.module_key,
|
||||
record.finding.module_runtime,
|
||||
);
|
||||
if (source === undefined) return;
|
||||
state.sources.set(record.key, source);
|
||||
@@ -256,7 +255,6 @@ const addValidationObservations = (state: IpcGraphState): void => {
|
||||
state.context,
|
||||
file,
|
||||
validation.module_key,
|
||||
validation.module_runtime,
|
||||
)
|
||||
: state.handlers.get(handlerRecord.key);
|
||||
if (target !== undefined)
|
||||
|
||||
@@ -38,12 +38,7 @@ export const addElectronNativeBoundaries = (
|
||||
|
||||
const addNativeBinding = (input: NativeBindingInput): void => {
|
||||
const { context, file, value, coverage } = input;
|
||||
const source = electronFindingSourceNode(
|
||||
context,
|
||||
file,
|
||||
value.module_key,
|
||||
value.module_runtime,
|
||||
);
|
||||
const source = electronFindingSourceNode(context, file, value.module_key);
|
||||
if (source === undefined) return;
|
||||
const resolved = resolveArtifactPath(
|
||||
value.specifier,
|
||||
|
||||
@@ -39,32 +39,17 @@ export const addElectronWindowBoundaries = (
|
||||
if (javascript === null) continue;
|
||||
const coverage = javascriptAnalysisCoverage(javascript);
|
||||
for (const value of javascript.electron.browser_windows) {
|
||||
const source = electronFindingSourceNode(
|
||||
context,
|
||||
file,
|
||||
value.module_key,
|
||||
value.module_runtime,
|
||||
);
|
||||
const source = electronFindingSourceNode(context, file, value.module_key);
|
||||
if (source !== undefined)
|
||||
addBrowserWindow({ context, file, source, value, coverage });
|
||||
}
|
||||
for (const value of javascript.electron.context_bridge_apis) {
|
||||
const source = electronFindingSourceNode(
|
||||
context,
|
||||
file,
|
||||
value.module_key,
|
||||
value.module_runtime,
|
||||
);
|
||||
const source = electronFindingSourceNode(context, file, value.module_key);
|
||||
if (source !== undefined)
|
||||
addContextBridge({ context, file, source, value, coverage });
|
||||
}
|
||||
for (const value of javascript.electron.utility_processes) {
|
||||
const source = electronFindingSourceNode(
|
||||
context,
|
||||
file,
|
||||
value.module_key,
|
||||
value.module_runtime,
|
||||
);
|
||||
const source = electronFindingSourceNode(context, file, value.module_key);
|
||||
if (source !== undefined)
|
||||
addUtilityProcess({ context, file, source, value, coverage });
|
||||
}
|
||||
|
||||
+13
-24
@@ -2,11 +2,11 @@ import { setImmediate } from "node:timers";
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js";
|
||||
import { analyzeJavaScriptApplication } from "../../support/javascriptApplicationScope.js";
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
describe("JavaScript application cancellation", () => {
|
||||
describe("JavaScript analysis cancellation before publication", () => {
|
||||
it("interrupts semantic graph commitment before result validation", async () => {
|
||||
const root = await createTestTempDirectory("rea-js-cancel-semantic-");
|
||||
await writeFile(
|
||||
@@ -33,9 +33,7 @@ describe("JavaScript application cancellation", () => {
|
||||
},
|
||||
);
|
||||
if (result.ok)
|
||||
throw new Error(
|
||||
"Cancelled commitment must not publish a successful analysis",
|
||||
);
|
||||
throw new Error("Cancelled commitment must not publish Evidence");
|
||||
expect(projectAnalysisError(result.error)).toMatchObject({
|
||||
code: "cancelled",
|
||||
details: { reason: "cancelled" },
|
||||
@@ -44,7 +42,7 @@ describe("JavaScript application cancellation", () => {
|
||||
expect(phases).not.toContain("validate_javascript_application_result");
|
||||
});
|
||||
|
||||
it("honors control turns and cancellation while preparing one large source for owned analysis (#1462)", async () => {
|
||||
it("handles control turns and cancellation inside one large source file (#1462)", async () => {
|
||||
const root = await createTestTempDirectory("rea-js-cancel-in-file-");
|
||||
await writeFile(
|
||||
join(root, "index.js"),
|
||||
@@ -76,7 +74,7 @@ describe("JavaScript application cancellation", () => {
|
||||
},
|
||||
},
|
||||
);
|
||||
if (result.ok) throw new Error("Cancelled analysis must not succeed");
|
||||
if (result.ok) throw new Error("Cancelled analysis must not publish");
|
||||
expect(projectAnalysisError(result.error)).toMatchObject({
|
||||
code: "cancelled",
|
||||
});
|
||||
@@ -120,21 +118,12 @@ describe("JavaScript application cancellation", () => {
|
||||
},
|
||||
);
|
||||
expect(requested).toBe(true);
|
||||
if (result.ok) throw new Error("Cancelled analysis must return a failure");
|
||||
const failure = projectAnalysisError(result.error);
|
||||
expect(failure).toMatchObject({ code: "cancelled" });
|
||||
if (phase === "parse_javascript_source")
|
||||
expect(failure).toMatchObject({
|
||||
details: {
|
||||
operation: "analyze_javascript_application",
|
||||
cleanup: "complete",
|
||||
partial_observation: {
|
||||
normalized_result: {
|
||||
statistics: { relevant_files: 1, parse_failures: 0 },
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
if (result.ok)
|
||||
throw new Error("Cancelled analysis must not publish Evidence");
|
||||
expect(projectAnalysisError(result.error)).toMatchObject({
|
||||
code: "cancelled",
|
||||
details: { reason: "cancelled" },
|
||||
});
|
||||
expect(terminal).toEqual([]);
|
||||
});
|
||||
|
||||
@@ -10,7 +10,7 @@ import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascri
|
||||
import { matchJavaScriptApplicationVersions } from "../../domain/javascript/javascriptApplicationVersionKeys.js";
|
||||
import { findApplicationFeatureSeeds } from "../../domain/javascript/javascriptFeatureSeed.js";
|
||||
import { traceApplicationFeatureInputSchema } from "../../domain/javascript/javascriptFeatureTraceSchemas.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
const analyzeSource = async (
|
||||
source: string,
|
||||
|
||||
@@ -5,7 +5,7 @@ import { expect, it } from "vitest";
|
||||
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
it.each([
|
||||
{
|
||||
|
||||
@@ -4,11 +4,9 @@ import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { z } from "zod";
|
||||
|
||||
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
|
||||
import { scanCanonicalArtifactInventory } from "../../../tests/fixtures/artifactInventory.js";
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
describe("JavaScript application failure diagnostics", () => {
|
||||
it("identifies the rejected result field in caller-visible diagnostics", async () => {
|
||||
@@ -155,41 +153,3 @@ describe("JavaScript application failure diagnostics", () => {
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe("JavaScript application cleanup diagnostics", () => {
|
||||
it("preserves cleanup and a complete inventory when translating reader failures", async () => {
|
||||
const inputPath = await createTestTempDirectory("rea-js-cleanup-failure-");
|
||||
await writeFile(join(inputPath, "main.js"), "export const value = 1;\n");
|
||||
const inventory = await scanCanonicalArtifactInventory(inputPath);
|
||||
const cleanup = {
|
||||
reason: "reader remains open",
|
||||
resources: [`JavaScript artifact reader for ${inputPath}`],
|
||||
};
|
||||
const partialObservation = {
|
||||
kind: "artifact-inventory" as const,
|
||||
inventory,
|
||||
};
|
||||
const failure = new ArtifactReaderFailure("io", "Reader cleanup failed", {
|
||||
cleanup,
|
||||
partialObservation,
|
||||
});
|
||||
|
||||
const result = await analyzeJavaScriptApplication(
|
||||
{ input_path: inputPath, format: "directory" },
|
||||
{
|
||||
progress: {
|
||||
report: async () => {
|
||||
throw failure;
|
||||
},
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
if (result.ok) throw new Error("Expected reader failure");
|
||||
expect(result.error).toMatchObject({
|
||||
_tag: "ArtifactOperationError",
|
||||
cleanup,
|
||||
partialObservation,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,14 +1,18 @@
|
||||
import { z } from "zod";
|
||||
|
||||
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
|
||||
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
|
||||
import {
|
||||
analyzeJavaScriptApplicationInputSchema,
|
||||
parseOwnedJavaScriptApplicationAnalysisSteps,
|
||||
} from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { AnalysisOutputError } from "../../domain/analysisErrorCore.js";
|
||||
import {
|
||||
AnalysisAccessDeniedError,
|
||||
AnalysisInputError,
|
||||
AnalysisOutputError,
|
||||
AnalysisUnsupportedTargetError,
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import { ArtifactOperationError } from "../../domain/artifactOperationError.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { type AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import type { Evidence } from "../../domain/evidence.js";
|
||||
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
@@ -28,7 +32,6 @@ const OPERATION = "analyze_javascript_application" as const;
|
||||
/** Statically analyze one local JavaScript/Electron application. */
|
||||
export const analyzeJavaScriptApplication = async (
|
||||
rawInput: unknown,
|
||||
scope: ArtifactResourceScope,
|
||||
options: ExecutionOptions = {},
|
||||
): Promise<Result<Evidence, AnalysisError>> => {
|
||||
const parsed = analyzeJavaScriptApplicationInputSchema.safeParse(rawInput);
|
||||
@@ -36,13 +39,12 @@ export const analyzeJavaScriptApplication = async (
|
||||
return err(
|
||||
analysisInputErrorFromIssues(OPERATION, parsed.error.issues, rawInput),
|
||||
);
|
||||
return analyzeJavaScriptApplicationValidated(parsed.data, scope, options);
|
||||
return analyzeJavaScriptApplicationValidated(parsed.data, options);
|
||||
};
|
||||
|
||||
/** Analyze input already parsed by a trusted adapter boundary. */
|
||||
export const analyzeJavaScriptApplicationValidated = async (
|
||||
input: z.output<typeof analyzeJavaScriptApplicationInputSchema>,
|
||||
scope: ArtifactResourceScope,
|
||||
options: ExecutionOptions = {},
|
||||
): Promise<Result<Evidence, AnalysisError>> => {
|
||||
try {
|
||||
@@ -59,14 +61,7 @@ export const analyzeJavaScriptApplicationValidated = async (
|
||||
input_path: input.input_path,
|
||||
format: input.format,
|
||||
integrity_policy: input.integrity_policy,
|
||||
...(input.max_heap_mb === undefined
|
||||
? {}
|
||||
: { max_heap_mb: input.max_heap_mb }),
|
||||
...(input.analysis_timeout_ms === undefined
|
||||
? {}
|
||||
: { analysis_timeout_ms: input.analysis_timeout_ms }),
|
||||
},
|
||||
scope,
|
||||
options.signal,
|
||||
options.progress,
|
||||
);
|
||||
@@ -111,10 +106,22 @@ export const analyzeJavaScriptApplicationValidated = async (
|
||||
assertJavaScriptAnalysisActive(options.signal);
|
||||
return ok(evidence);
|
||||
} catch (cause: unknown) {
|
||||
// Preserve execution constraints and their completed partial Evidence.
|
||||
if (cause instanceof AnalysisError) return err(cause);
|
||||
// Input selection failures are already typed by their cause.
|
||||
if (
|
||||
cause instanceof AnalysisInputError ||
|
||||
cause instanceof AnalysisUnsupportedTargetError ||
|
||||
cause instanceof AnalysisAccessDeniedError
|
||||
)
|
||||
return err(cause);
|
||||
if (cause instanceof ArtifactReaderFailure)
|
||||
return err(artifactFailureToOperationError(cause));
|
||||
return err(
|
||||
new ArtifactOperationError(
|
||||
OPERATION,
|
||||
cause.reason,
|
||||
cause.details,
|
||||
cause.message,
|
||||
),
|
||||
);
|
||||
if (cause instanceof z.ZodError)
|
||||
return err(
|
||||
new AnalysisOutputError(OPERATION, describeResultSchemaFailure(cause), {
|
||||
@@ -146,23 +153,6 @@ export const analyzeJavaScriptApplicationValidated = async (
|
||||
}
|
||||
};
|
||||
|
||||
const artifactFailureToOperationError = (
|
||||
cause: ArtifactReaderFailure,
|
||||
): ArtifactOperationError =>
|
||||
new ArtifactOperationError(
|
||||
OPERATION,
|
||||
cause.reason,
|
||||
cause.details,
|
||||
cause.message,
|
||||
{
|
||||
cause,
|
||||
...(cause.cleanup === undefined ? {} : { cleanup: cause.cleanup }),
|
||||
...(cause.partialObservation === undefined
|
||||
? {}
|
||||
: { partialObservation: cause.partialObservation }),
|
||||
},
|
||||
);
|
||||
|
||||
const describeResultSchemaFailure = (cause: z.ZodError): string => {
|
||||
const issue = cause.issues[0];
|
||||
if (issue === undefined) return "Result schema rejected an unknown issue";
|
||||
|
||||
@@ -7,7 +7,7 @@ import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirect
|
||||
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { javaScriptSemanticQueryResultSchema } from "../../domain/javascript/javascriptSemanticQuerySchemas.js";
|
||||
import { queryJavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticQuery.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
it.each([
|
||||
{
|
||||
|
||||
@@ -5,7 +5,7 @@ import { expect, it } from "vitest";
|
||||
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
const graphIdentities = async (source: string) => {
|
||||
const inputPath = await createTestTempDirectory("rea-js-template-literal-");
|
||||
@@ -31,16 +31,7 @@ const graphIdentities = async (source: string) => {
|
||||
...(identity.strategy === "artifact-local-key"
|
||||
? {
|
||||
namespace: identity.namespace,
|
||||
// Module-export identity is a JSON tuple whose first entry
|
||||
// binds the whole container; only that digest changes here.
|
||||
key:
|
||||
identity.namespace === "module-export"
|
||||
? JSON.stringify(
|
||||
(JSON.parse(identity.key) as unknown[]).slice(1),
|
||||
)
|
||||
: identity.namespace === "source-module"
|
||||
? identity.key.replace(/^[0-9a-f]{64}:/u, "")
|
||||
: identity.key,
|
||||
key: identity.key.replace(/^[0-9a-f]{64}:/u, ""),
|
||||
}
|
||||
: {}),
|
||||
}))
|
||||
|
||||
@@ -7,7 +7,7 @@ import { canonicalJson } from "../../domain/comparisonSemantics.js";
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
type ApplicationAnalysis = ReturnType<
|
||||
typeof javascriptApplicationAnalysisResultSchema.parse
|
||||
|
||||
@@ -7,6 +7,7 @@ import type { ApplicationVersionComparisonResult } from "../../domain/javascript
|
||||
import type { JavaScriptExportShapeComparisonResult } from "../../domain/javascript/javascriptExportShapeComparisonSchemas.js";
|
||||
import type { ApplicationFeatureTraceResult } from "../../domain/javascript/javascriptFeatureTraceSchemas.js";
|
||||
import type { JavaScriptSemanticTraceResult } from "../../domain/javascript/javascriptSemanticTraceSchemas.js";
|
||||
import { jsonValueSchema } from "../../domain/jsonValue.js";
|
||||
import type { SourceToBundleComparisonResult } from "../../domain/javascript/sourceToBundleComparisonSchemas.js";
|
||||
import { JAVASCRIPT_APPLICATION_WORKFLOW_PROVIDER } from "../InvestigationProviders.js";
|
||||
|
||||
@@ -19,7 +20,7 @@ export const createApplicationFeatureTraceEvidence = (
|
||||
predicateType: "rea.application-feature-trace",
|
||||
operation: "trace_application_feature",
|
||||
parameters,
|
||||
result,
|
||||
result: jsonValueSchema.parse(result),
|
||||
rawResult: null,
|
||||
confidence: "inferred",
|
||||
authority: "analyst-inference",
|
||||
@@ -37,7 +38,7 @@ export const createJavaScriptSemanticTraceEvidence = (
|
||||
predicateType: "rea.javascript-semantic-trace",
|
||||
operation: "trace_javascript_semantics",
|
||||
parameters,
|
||||
result,
|
||||
result: jsonValueSchema.parse(result),
|
||||
rawResult: null,
|
||||
confidence: "inferred",
|
||||
authority: "analyst-inference",
|
||||
@@ -55,7 +56,7 @@ export const createApplicationVersionComparisonEvidence = (
|
||||
predicateType: "rea.application-version-comparison",
|
||||
operation: "compare_application_versions",
|
||||
parameters,
|
||||
result,
|
||||
result: jsonValueSchema.parse(result),
|
||||
rawResult: null,
|
||||
confidence: "inferred",
|
||||
authority: "analyst-inference",
|
||||
@@ -73,7 +74,7 @@ export const createSourceToBundleComparisonEvidence = (
|
||||
predicateType: "rea.source-to-bundle-comparison",
|
||||
operation: "compare_source_to_bundle",
|
||||
parameters,
|
||||
result,
|
||||
result: jsonValueSchema.parse(result),
|
||||
rawResult: null,
|
||||
confidence: "inferred",
|
||||
authority: "analyst-inference",
|
||||
@@ -91,7 +92,7 @@ export const createJavaScriptExportShapeComparisonEvidence = (
|
||||
predicateType: "rea.javascript-export-shape-comparison",
|
||||
operation: "compare_javascript_export_shapes",
|
||||
parameters,
|
||||
result,
|
||||
result: jsonValueSchema.parse(result),
|
||||
rawResult: null,
|
||||
confidence: "inferred",
|
||||
authority: "analyst-inference",
|
||||
|
||||
@@ -8,7 +8,6 @@ import {
|
||||
classifyParsedJavaScriptOpenReceiversSteps,
|
||||
} from "../../domain/javascript/javascriptSemanticAnalysis.js";
|
||||
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
|
||||
import { projectJavaScriptModuleSemantics } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
|
||||
import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js";
|
||||
import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js";
|
||||
@@ -36,15 +35,6 @@ import type {
|
||||
import { analyzeJavaScriptJsonModule } from "./JavaScriptJsonModules.js";
|
||||
import { completeJavaScriptAnalysisSteps } from "./JavaScriptAnalysisControl.js";
|
||||
import { htmlArtifactReferences } from "../../domain/javascript/htmlArtifactReferences.js";
|
||||
import type { JavaScriptSourceAnalysisPort } from "../../domain/javascript/javascriptSourceAnalysis.js";
|
||||
import type { JavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
||||
import {
|
||||
AnalysisCancelledError,
|
||||
AnalysisResourceConstraintError,
|
||||
AnalysisTimeoutError,
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
|
||||
interface MutableArtifactAnalysis<
|
||||
SemanticIr extends JavaScriptModuleSemanticIr,
|
||||
@@ -105,7 +95,27 @@ export const analyzeAndProjectJavaScriptArtifactFiles = async (
|
||||
): Generator<void, JavaScriptModuleSemanticIr> {
|
||||
const projection = projectSemantics(file, ir);
|
||||
if (projection !== undefined) while (projection.next().done !== true) yield;
|
||||
return projectJavaScriptModuleSemantics(ir);
|
||||
const programScopes = ir.scopes.filter(({ kind }) => kind === "program");
|
||||
const programScopeId = programScopes[0]?.scopeId;
|
||||
const localNames = new Set(
|
||||
ir.moduleLinks.map(({ localName }) => localName),
|
||||
);
|
||||
const callableIds = new Set(
|
||||
ir.moduleLinks.map(({ callableId }) => callableId),
|
||||
);
|
||||
return {
|
||||
scopes: programScopes,
|
||||
bindings: ir.bindings.filter(
|
||||
({ name, scopeId }) =>
|
||||
scopeId === programScopeId && localNames.has(name),
|
||||
),
|
||||
callables: ir.callables.filter(({ callableId }) =>
|
||||
callableIds.has(callableId),
|
||||
),
|
||||
moduleLinks: ir.moduleLinks,
|
||||
coverage: ir.coverage,
|
||||
limitations: ir.limitations,
|
||||
};
|
||||
};
|
||||
const context = { state, projectSemantics: projectFileSemantics };
|
||||
for (const [index, file] of fileSet.files.entries()) {
|
||||
@@ -120,174 +130,6 @@ export const analyzeAndProjectJavaScriptArtifactFiles = async (
|
||||
return finalizeArtifactAnalysis(state);
|
||||
};
|
||||
|
||||
/** Analyze sources behind a hard heap boundary, retaining collected facts on interruption. */
|
||||
export const analyzeAndAdoptJavaScriptArtifactFiles = async (
|
||||
fileSet: JavaScriptArtifactFileSet,
|
||||
worker: Pick<JavaScriptSourceAnalysisPort, "analyze">,
|
||||
projection: JavaScriptSemanticGraphProjection,
|
||||
beforeFile: (
|
||||
file: JavaScriptArtifactFile,
|
||||
completed: number,
|
||||
total: number,
|
||||
) => Promise<void>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<{
|
||||
readonly analysis: JavaScriptModuleArtifactAnalysis;
|
||||
readonly interruption: AnalysisError | null;
|
||||
}> => {
|
||||
const state = emptyArtifactAnalysis<JavaScriptModuleSemanticIr>();
|
||||
let interruption: AnalysisError | null = null;
|
||||
for (const [index, file] of fileSet.files.entries()) {
|
||||
try {
|
||||
await setImmediate();
|
||||
await beforeFile(file, index, fileSet.files.length);
|
||||
if (signal?.aborted === true)
|
||||
throw new AnalysisCancelledError("analyze_javascript_application");
|
||||
if (file.kind !== "javascript" || !file.text.included) {
|
||||
const steps = analyzeArtifactFileSteps(file, {
|
||||
state,
|
||||
projectSemantics: function* (_file, ir) {
|
||||
yield;
|
||||
return projectJavaScriptModuleSemantics(ir);
|
||||
},
|
||||
});
|
||||
await completeJavaScriptAnalysisSteps(steps, signal);
|
||||
continue;
|
||||
}
|
||||
const result = await worker.analyze(
|
||||
file,
|
||||
projection.remainingFileNodeBudget(),
|
||||
signal,
|
||||
);
|
||||
const facts = result.ok ? result.value : result.error;
|
||||
const applicationProjectionFailure = facts.applicationProjectionFailure;
|
||||
if (facts.projection !== null)
|
||||
projection.adoptFileProjection(facts.projection);
|
||||
const error = result.ok ? null : result.error.error;
|
||||
const limits =
|
||||
error instanceof AnalysisResourceConstraintError
|
||||
? error.reportedLimits
|
||||
: error instanceof AnalysisTimeoutError
|
||||
? { worker_timeout_ms: error.timeoutMs }
|
||||
: null;
|
||||
const failure =
|
||||
error === null
|
||||
? undefined
|
||||
: {
|
||||
reason: `${file.path}: ${error.message}`,
|
||||
error: projectAnalysisError(error),
|
||||
limits,
|
||||
};
|
||||
state.files.push({
|
||||
file,
|
||||
javascript: facts.javascript,
|
||||
semantic: facts.module === null ? null : { ir: facts.module },
|
||||
...(failure === undefined ? {} : { analysis_failure: failure }),
|
||||
...(applicationProjectionFailure === undefined
|
||||
? {}
|
||||
: {
|
||||
application_projection_failure: {
|
||||
reason: applicationProjectionFailure.message,
|
||||
error: projectAnalysisError(applicationProjectionFailure),
|
||||
limits: applicationProjectionFailure.reportedLimits,
|
||||
},
|
||||
}),
|
||||
});
|
||||
if (failure !== undefined && facts.projection === null)
|
||||
projection.recordUnavailableFile(
|
||||
file,
|
||||
failure.reason,
|
||||
failure.limits,
|
||||
error instanceof AnalysisResourceConstraintError ||
|
||||
error instanceof AnalysisTimeoutError
|
||||
? "resource-limit"
|
||||
: "incomplete-module",
|
||||
);
|
||||
if (facts.javascript !== null) {
|
||||
state.visitedNodes += facts.javascript.visited_ast_nodes;
|
||||
state.findings +=
|
||||
findingCount(facts.javascript) +
|
||||
(facts.module?.moduleLinks.length ?? 0);
|
||||
state.modules += facts.javascript.bundler_registrations.reduce(
|
||||
(count, item) => count + item.modules.length,
|
||||
0,
|
||||
);
|
||||
if (facts.javascript.parse_status === "failed")
|
||||
state.parseFailures += 1;
|
||||
}
|
||||
if (
|
||||
error !== null &&
|
||||
(!(error instanceof AnalysisResourceConstraintError) ||
|
||||
error.reportedLimits?.parent_result_budget_exhausted === true)
|
||||
) {
|
||||
interruption = error;
|
||||
break;
|
||||
}
|
||||
} catch (cause: unknown) {
|
||||
if (signal?.aborted !== true && !(cause instanceof AnalysisError))
|
||||
throw cause;
|
||||
interruption =
|
||||
signal?.aborted === true
|
||||
? new AnalysisCancelledError("analyze_javascript_application")
|
||||
: cause instanceof AnalysisError
|
||||
? cause
|
||||
: null;
|
||||
if (interruption !== null) {
|
||||
state.files.push({
|
||||
file,
|
||||
javascript: null,
|
||||
semantic: null,
|
||||
analysis_failure: {
|
||||
reason: `${file.path}: ${interruption.message}`,
|
||||
error: projectAnalysisError(interruption),
|
||||
limits: null,
|
||||
},
|
||||
});
|
||||
projection.recordUnavailableFile(
|
||||
file,
|
||||
interruption.message,
|
||||
null,
|
||||
"incomplete-module",
|
||||
);
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (interruption !== null) {
|
||||
const analyzedPaths = new Set(state.files.map(({ file }) => file.path));
|
||||
const failure = {
|
||||
reason: `Source analysis was interrupted: ${interruption.message}`,
|
||||
error: projectAnalysisError(interruption),
|
||||
limits:
|
||||
interruption instanceof AnalysisResourceConstraintError
|
||||
? interruption.reportedLimits
|
||||
: interruption instanceof AnalysisTimeoutError
|
||||
? { worker_timeout_ms: interruption.timeoutMs }
|
||||
: null,
|
||||
};
|
||||
for (const file of fileSet.files) {
|
||||
if (analyzedPaths.has(file.path)) continue;
|
||||
state.files.push({
|
||||
file,
|
||||
javascript: null,
|
||||
semantic: null,
|
||||
analysis_failure: failure,
|
||||
});
|
||||
if (file.kind === "javascript")
|
||||
projection.recordUnavailableFile(
|
||||
file,
|
||||
failure.reason,
|
||||
failure.limits,
|
||||
interruption instanceof AnalysisResourceConstraintError ||
|
||||
interruption instanceof AnalysisTimeoutError
|
||||
? "resource-limit"
|
||||
: "incomplete-module",
|
||||
);
|
||||
}
|
||||
}
|
||||
return { analysis: finalizeArtifactAnalysis(state), interruption };
|
||||
};
|
||||
|
||||
const analyzeArtifactFiles = <SemanticIr extends JavaScriptModuleSemanticIr>(
|
||||
fileSet: JavaScriptArtifactFileSet,
|
||||
projectSemantics: ArtifactAnalysisContext<SemanticIr>["projectSemantics"],
|
||||
@@ -319,12 +161,6 @@ const finalizeArtifactAnalysis = <
|
||||
limitations: [
|
||||
"JavaScript and HTML were parsed as inert text; bundle bootstrap code was never executed.",
|
||||
"Static paths and relationships may remain unresolved when expressions are dynamic or obfuscated.",
|
||||
...state.files.flatMap(
|
||||
({ analysis_failure, application_projection_failure }) =>
|
||||
[analysis_failure, application_projection_failure].flatMap(
|
||||
(failure) => (failure === undefined ? [] : [failure.reason]),
|
||||
),
|
||||
),
|
||||
],
|
||||
};
|
||||
};
|
||||
|
||||
@@ -4,9 +4,7 @@ import type {
|
||||
JavaScriptStaticAnalysis,
|
||||
} from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
|
||||
import type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
|
||||
interface JavaScriptStructuredObservation {
|
||||
readonly path: string;
|
||||
@@ -91,7 +89,15 @@ export type JavaScriptJsonModuleObservation = JavaScriptStructuredObservation &
|
||||
);
|
||||
|
||||
/** Module facts needed after file-local semantic projection has finished. */
|
||||
export type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
|
||||
export type JavaScriptModuleSemanticIr = Pick<
|
||||
JavaScriptSemanticIr,
|
||||
| "scopes"
|
||||
| "bindings"
|
||||
| "callables"
|
||||
| "moduleLinks"
|
||||
| "coverage"
|
||||
| "limitations"
|
||||
>;
|
||||
|
||||
/** One relevant file plus optional AST-only JavaScript facts. */
|
||||
export interface AnalyzedJavaScriptArtifactFile<
|
||||
@@ -102,18 +108,6 @@ export interface AnalyzedJavaScriptArtifactFile<
|
||||
readonly semantic: {
|
||||
readonly ir: SemanticIr;
|
||||
} | null;
|
||||
/** Operational incompleteness, distinct from a malformed source file. */
|
||||
readonly analysis_failure?: {
|
||||
readonly reason: string;
|
||||
readonly error: JsonValue;
|
||||
readonly limits: Readonly<Record<string, JsonValue>> | null;
|
||||
};
|
||||
/** Facts retained inline when their expanded graph exceeds available memory. */
|
||||
readonly application_projection_failure?: {
|
||||
readonly reason: string;
|
||||
readonly error: JsonValue;
|
||||
readonly limits: Readonly<Record<string, JsonValue>> | null;
|
||||
};
|
||||
}
|
||||
|
||||
/** Complete static-analysis projection before graph construction. */
|
||||
|
||||
@@ -12,7 +12,7 @@ export class JavaScriptArtifactGraphAccumulator {
|
||||
readonly #nodes = new Map<string, ApplicationNode>();
|
||||
readonly #edges = new Map<string, ApplicationEdge>();
|
||||
|
||||
/** Create or merge one canonical node, retaining a stable live construction reference. */
|
||||
/** Create or merge one canonical node. */
|
||||
addNode<Input extends DisplayableNodeInput>(input: Input): ApplicationNode {
|
||||
const created = createJavaScriptApplicationNode(displayableNode(input));
|
||||
const existing = this.#nodes.get(created.node_id);
|
||||
@@ -23,23 +23,19 @@ export class JavaScriptArtifactGraphAccumulator {
|
||||
const bySemanticId = new Map(
|
||||
[...existing.observations, ...created.observations].map((observation) => [
|
||||
observation.observation_id,
|
||||
observation,
|
||||
observationInput(observation),
|
||||
]),
|
||||
);
|
||||
const observations = [...bySemanticId.values()].sort((left, right) =>
|
||||
left.observation_id < right.observation_id
|
||||
? -1
|
||||
: left.observation_id > right.observation_id
|
||||
? 1
|
||||
: 0,
|
||||
);
|
||||
// Both factories already validated and identified these observations.
|
||||
// Replacing the node retained every previous observation array through
|
||||
// path/chunk/IPC indexes when multiple paths shared an entity identity.
|
||||
existing.observations.length = 0;
|
||||
for (const observation of observations)
|
||||
existing.observations.push(observation);
|
||||
return existing;
|
||||
const observations = [...bySemanticId.entries()]
|
||||
.sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0))
|
||||
.map(([, observation]) => observation);
|
||||
const merged = createJavaScriptApplicationNode({
|
||||
kind: existing.kind,
|
||||
identity: existing.identity,
|
||||
observations,
|
||||
});
|
||||
this.#nodes.set(merged.node_id, merged);
|
||||
return merged;
|
||||
}
|
||||
|
||||
/** Create or replace one canonical edge by semantic identifier. */
|
||||
@@ -60,6 +56,17 @@ export class JavaScriptArtifactGraphAccumulator {
|
||||
}
|
||||
}
|
||||
|
||||
const observationInput = (
|
||||
observation: ApplicationNode["observations"][number],
|
||||
) => {
|
||||
const {
|
||||
observation_id: _observationId,
|
||||
identifier_strategy: _strategy,
|
||||
...input
|
||||
} = observation;
|
||||
return input;
|
||||
};
|
||||
|
||||
/** Node fields whose display form is normalized before validation. */
|
||||
interface DisplayableNodeInput {
|
||||
readonly observations: readonly { readonly label: string | null }[];
|
||||
|
||||
@@ -107,33 +107,19 @@ function* buildJavaScriptArtifactGraphInputSteps(
|
||||
yield;
|
||||
addJavaScriptArtifactFiles(context);
|
||||
yield;
|
||||
// Raw observations remain on the file nodes. Only admitted expansions feed
|
||||
// the relationship builders, including their Electron and bundler callers.
|
||||
const projectionContext = {
|
||||
...context,
|
||||
analysis: {
|
||||
...analysis,
|
||||
files: analysis.files.map((file) =>
|
||||
file.application_projection_failure === undefined
|
||||
? file
|
||||
: { ...file, javascript: null, semantic: null },
|
||||
),
|
||||
},
|
||||
};
|
||||
const packageRoots = addJavaScriptPackageNodes(projectionContext);
|
||||
const packageRoots = addJavaScriptPackageNodes(context);
|
||||
yield;
|
||||
addJavaScriptSourceModules(projectionContext);
|
||||
addJavaScriptSourceModules(context);
|
||||
yield;
|
||||
const bundlerLimitations = addJavaScriptBundlerNodes(projectionContext);
|
||||
const bundlerLimitations = addJavaScriptBundlerNodes(context);
|
||||
const relationshipOmissions =
|
||||
yield* addJavaScriptModuleRelationshipsSteps(projectionContext);
|
||||
const findingLimitations =
|
||||
yield* addJavaScriptStaticFindingsSteps(projectionContext);
|
||||
addElectronBoundaries(projectionContext);
|
||||
yield* addJavaScriptModuleRelationshipsSteps(context);
|
||||
const findingLimitations = yield* addJavaScriptStaticFindingsSteps(context);
|
||||
addElectronBoundaries(context);
|
||||
yield;
|
||||
addJavaScriptHtmlRoles(projectionContext);
|
||||
addJavaScriptHtmlRoles(context);
|
||||
yield;
|
||||
addJavaScriptSourceMapOriginals(projectionContext);
|
||||
addJavaScriptSourceMapOriginals(context);
|
||||
yield;
|
||||
const coverage = graphCoverage(context);
|
||||
return {
|
||||
@@ -167,34 +153,14 @@ const graphCoverage = (context: JavaScriptArtifactGraphContext) => {
|
||||
javascript !== null && javascript.parse_status === "partial",
|
||||
);
|
||||
const unknownGap =
|
||||
context.analysis.files.some(
|
||||
({ analysis_failure, application_projection_failure }) =>
|
||||
analysis_failure !== undefined ||
|
||||
application_projection_failure !== undefined,
|
||||
) ||
|
||||
context.analysis.parse_failures > 0 ||
|
||||
context.fileSet.invalid_utf8_files > 0 ||
|
||||
sourceMapPolicyGap ||
|
||||
malformedStructuredData ||
|
||||
partialJavaScript;
|
||||
const executionLimits = context.analysis.files.flatMap(
|
||||
({ analysis_failure, application_projection_failure }) =>
|
||||
[analysis_failure, application_projection_failure].flatMap((failure) =>
|
||||
failure === undefined
|
||||
? []
|
||||
: javaScriptAnalysisCoverageLimits(failure.limits),
|
||||
),
|
||||
);
|
||||
if (resourceLimits.length > 0 || executionLimits.length > 0)
|
||||
if (resourceLimits.length > 0)
|
||||
return partialApplicationCoverage(
|
||||
[
|
||||
...new Map(
|
||||
[
|
||||
...semanticResourceLimitCoverage(resourceLimits),
|
||||
...executionLimits,
|
||||
].map((limit) => [JSON.stringify(limit), limit]),
|
||||
).values(),
|
||||
],
|
||||
semanticResourceLimitCoverage(resourceLimits),
|
||||
null,
|
||||
);
|
||||
if (unknownGap) return partialApplicationCoverage([], null);
|
||||
@@ -294,4 +260,3 @@ const graphLimitations = (
|
||||
]),
|
||||
];
|
||||
};
|
||||
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||
|
||||
@@ -161,7 +161,7 @@ const addBundlerModuleNodes = (
|
||||
],
|
||||
});
|
||||
context.moduleNodes.set(
|
||||
moduleLookupKey(file.path, registration.runtime, moduleValue.module_key),
|
||||
moduleLookupKey(file.path, moduleValue.module_key),
|
||||
module,
|
||||
);
|
||||
addAstContainsEdge(context, {
|
||||
@@ -195,7 +195,7 @@ const addBundlerRuntimeEdges = (
|
||||
const { file, registration, chunk, coverage } = record;
|
||||
for (const moduleKey of registration.entry_module_keys) {
|
||||
const resolved = context.moduleNodes.get(
|
||||
moduleLookupKey(file.path, registration.runtime, moduleKey),
|
||||
moduleLookupKey(file.path, moduleKey),
|
||||
);
|
||||
const target =
|
||||
resolved ?? unresolvedBundlerModuleNode(context, record, moduleKey);
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import { compositeKey } from "../../domain/unicodeCodePointOrder.js";
|
||||
import {
|
||||
completeApplicationCoverage,
|
||||
@@ -100,8 +99,6 @@ export interface UnavailableParseInput {
|
||||
readonly asset: ApplicationNode;
|
||||
readonly operation: string;
|
||||
readonly limitation: string;
|
||||
readonly failure?: JsonValue;
|
||||
readonly limits?: JavaScriptArtifactGraphCoverage["limits"];
|
||||
}
|
||||
|
||||
/** Add a direct artifact-containment edge. */
|
||||
@@ -185,15 +182,12 @@ export const addUnavailableStaticParseScope = (
|
||||
path: input.file.path,
|
||||
operation: input.operation,
|
||||
limitation: input.limitation,
|
||||
...(input.failure === undefined
|
||||
? {}
|
||||
: { analysis_failure: input.failure }),
|
||||
},
|
||||
evidence: unavailableAstEvidence({
|
||||
sha256: input.file.sha256,
|
||||
path: input.file.path,
|
||||
operation: input.operation,
|
||||
coverage: partialApplicationCoverage(input.limits ?? [], null),
|
||||
coverage: partialApplicationCoverage([], null),
|
||||
limitation: input.limitation,
|
||||
}),
|
||||
},
|
||||
@@ -329,11 +323,8 @@ export const selfReferenceOmissions = (
|
||||
];
|
||||
|
||||
/** Deterministic key for one recovered module inside a bundle asset. */
|
||||
export const moduleLookupKey = (
|
||||
path: string,
|
||||
runtime: string,
|
||||
moduleKey: string,
|
||||
): string => compositeKey([path, runtime, moduleKey]);
|
||||
export const moduleLookupKey = (path: string, moduleKey: string): string =>
|
||||
`${path}\0${moduleKey}`;
|
||||
|
||||
/** Deterministic key for one recovered bundler chunk inside a bundle asset. */
|
||||
export const chunkLookupKey = (
|
||||
@@ -362,10 +353,7 @@ export const sourceNodeFor = (
|
||||
context: JavaScriptArtifactGraphContext,
|
||||
path: string,
|
||||
moduleKey: string | null,
|
||||
runtime?: string,
|
||||
): ApplicationNode | undefined =>
|
||||
moduleKey === null
|
||||
? context.sourceModuleNodes.get(path)
|
||||
: runtime === undefined
|
||||
? undefined
|
||||
: context.moduleNodes.get(moduleLookupKey(path, runtime, moduleKey));
|
||||
: context.moduleNodes.get(moduleLookupKey(path, moduleKey));
|
||||
|
||||
@@ -78,12 +78,8 @@ const addReference = (
|
||||
omissions: { selfReferences: number },
|
||||
): void => {
|
||||
const source =
|
||||
sourceNodeFor(
|
||||
context,
|
||||
input.file.path,
|
||||
input.value.module_key,
|
||||
input.value.module_runtime,
|
||||
) ?? input.asset;
|
||||
sourceNodeFor(context, input.file.path, input.value.module_key) ??
|
||||
input.asset;
|
||||
const resolved =
|
||||
input.value.specifier === null
|
||||
? null
|
||||
@@ -166,12 +162,8 @@ const addEndpoint = (
|
||||
input: FindingInput<StaticEndpoint>,
|
||||
): void => {
|
||||
const source =
|
||||
sourceNodeFor(
|
||||
context,
|
||||
input.file.path,
|
||||
input.value.module_key,
|
||||
input.value.module_runtime,
|
||||
) ?? input.asset;
|
||||
sourceNodeFor(context, input.file.path, input.value.module_key) ??
|
||||
input.asset;
|
||||
const key = compactStringIdentityKey(input.value.value);
|
||||
const label = `${input.value.kind} endpoint`;
|
||||
const node = context.accumulator.addNode({
|
||||
@@ -212,12 +204,8 @@ const addStorage = (
|
||||
input: FindingInput<StaticStorage>,
|
||||
): void => {
|
||||
const source =
|
||||
sourceNodeFor(
|
||||
context,
|
||||
input.file.path,
|
||||
input.value.module_key,
|
||||
input.value.module_runtime,
|
||||
) ?? input.asset;
|
||||
sourceNodeFor(context, input.file.path, input.value.module_key) ??
|
||||
input.asset;
|
||||
const node = context.accumulator.addNode({
|
||||
kind: "storage",
|
||||
identity: artifactLocalIdentity(
|
||||
@@ -274,12 +262,8 @@ const addDiscoveredRole = (
|
||||
coverage: input.coverage,
|
||||
});
|
||||
const source =
|
||||
sourceNodeFor(
|
||||
context,
|
||||
input.file.path,
|
||||
input.value.module_key,
|
||||
input.value.module_runtime,
|
||||
) ?? input.asset;
|
||||
sourceNodeFor(context, input.file.path, input.value.module_key) ??
|
||||
input.asset;
|
||||
addStaticInferenceEdge(context, {
|
||||
source,
|
||||
target: role,
|
||||
@@ -375,12 +359,9 @@ const resolveReference = (
|
||||
if (specifier === null) return null;
|
||||
const chunk = resolveBundlerChunkReference(context, sourcePath, reference);
|
||||
if (chunk !== null) return chunk;
|
||||
const module =
|
||||
reference.module_runtime === undefined
|
||||
? undefined
|
||||
: context.moduleNodes.get(
|
||||
moduleLookupKey(sourcePath, reference.module_runtime, specifier),
|
||||
);
|
||||
const module = context.moduleNodes.get(
|
||||
moduleLookupKey(sourcePath, specifier),
|
||||
);
|
||||
if (module !== undefined) {
|
||||
const file = context.filesByPath.get(sourcePath);
|
||||
return file === undefined
|
||||
@@ -426,11 +407,10 @@ const resolveBundlerChunkReference = (
|
||||
if (file === undefined) return null;
|
||||
const registrations = context.analysis.files
|
||||
.find(({ file: analyzedFile }) => analyzedFile.path === sourcePath)
|
||||
?.javascript?.bundler_registrations.filter(({ runtime, modules }) =>
|
||||
reference.module_runtime === undefined
|
||||
? reference.module_key === null
|
||||
: runtime === reference.module_runtime &&
|
||||
modules.some(
|
||||
?.javascript?.bundler_registrations.filter(({ modules }) =>
|
||||
reference.module_key === null
|
||||
? true
|
||||
: modules.some(
|
||||
({ module_key: moduleKey }) => moduleKey === reference.module_key,
|
||||
),
|
||||
);
|
||||
|
||||
@@ -2,9 +2,6 @@ import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySn
|
||||
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
|
||||
import { completeApplicationCoverage } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
|
||||
import type { JavaScriptModuleArtifactAnalysis } from "./JavaScriptArtifactAnalysisTypes.js";
|
||||
import { jsonValueSchema } from "../../domain/jsonValue.js";
|
||||
import { freezeJsonSnapshot } from "../../domain/immutableJson.js";
|
||||
import type { JavaScriptSourceRange } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
|
||||
import type {
|
||||
JavaScriptArtifactContainer,
|
||||
JavaScriptArtifactFile,
|
||||
@@ -15,14 +12,12 @@ import {
|
||||
addUnavailableStaticParseScope,
|
||||
artifactFileNodeKind,
|
||||
artifactLocalIdentity,
|
||||
javascriptAnalysisCoverage,
|
||||
createElectronRoleNode,
|
||||
linkElectronRoleToAsset,
|
||||
type JavaScriptArtifactGraphContext,
|
||||
} from "./JavaScriptArtifactGraphContext.js";
|
||||
import {
|
||||
artifactObservationEvidence,
|
||||
astObservationEvidence,
|
||||
staticInferenceEvidence,
|
||||
} from "./JavaScriptArtifactGraphEvidence.js";
|
||||
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
|
||||
@@ -120,52 +115,9 @@ export const addJavaScriptArtifactContainers = (
|
||||
export const addJavaScriptArtifactFiles = (
|
||||
context: JavaScriptArtifactGraphContext,
|
||||
): void => {
|
||||
const packagesByPath = indexFirstByPath(context.analysis.packages);
|
||||
const jsonModulesByPath = indexFirstByPath(context.analysis.json_modules);
|
||||
const sourceMapsByPath = indexFirstByPath(context.analysis.source_maps);
|
||||
for (const analyzed of context.analysis.files) {
|
||||
const { file } = analyzed;
|
||||
const jsonValue = jsonModulesByPath.get(file.path);
|
||||
const target = createFileTarget(
|
||||
context,
|
||||
file,
|
||||
analyzed.javascript,
|
||||
jsonValue,
|
||||
);
|
||||
if (
|
||||
analyzed.application_projection_failure !== undefined &&
|
||||
analyzed.javascript !== null
|
||||
)
|
||||
context.accumulator.addNode({
|
||||
kind: target.kind,
|
||||
identity: target.identity,
|
||||
observations: [
|
||||
{
|
||||
label: file.path,
|
||||
properties: {
|
||||
static_analysis: freezeJsonSnapshot(
|
||||
jsonValueSchema.parse(analyzed.javascript),
|
||||
),
|
||||
semantic_module_analysis:
|
||||
analyzed.semantic === null
|
||||
? null
|
||||
: freezeJsonSnapshot(
|
||||
jsonValueSchema.parse(analyzed.semantic.ir),
|
||||
),
|
||||
application_projection_failure:
|
||||
analyzed.application_projection_failure.error,
|
||||
},
|
||||
evidence: astObservationEvidence({
|
||||
sha256: file.sha256,
|
||||
path: file.path,
|
||||
operation: "retain-unexpanded-static-findings",
|
||||
range: sourceFileRange(file),
|
||||
coverage: javascriptAnalysisCoverage(analyzed.javascript),
|
||||
limitations: [analyzed.application_projection_failure.reason],
|
||||
}),
|
||||
},
|
||||
],
|
||||
});
|
||||
const target = createFileTarget(context, file, analyzed.javascript);
|
||||
context.fileNodes.set(file.path, target);
|
||||
if (file.kind === "javascript") context.assetNodes.set(file.path, target);
|
||||
const entry = createAsarEntry(context, file);
|
||||
@@ -198,18 +150,7 @@ export const addJavaScriptArtifactFiles = (
|
||||
}),
|
||||
});
|
||||
}
|
||||
if (analyzed.analysis_failure !== undefined)
|
||||
addUnavailableStaticParseScope(context, {
|
||||
file,
|
||||
asset: target,
|
||||
operation: "analyze-javascript-source",
|
||||
limitation: analyzed.analysis_failure.reason,
|
||||
failure: analyzed.analysis_failure.error,
|
||||
limits: javaScriptAnalysisCoverageLimits(
|
||||
analyzed.analysis_failure.limits,
|
||||
),
|
||||
});
|
||||
else if (
|
||||
if (
|
||||
file.kind === "javascript" &&
|
||||
(analyzed.javascript === null ||
|
||||
analyzed.javascript.parse_status === "failed")
|
||||
@@ -222,18 +163,9 @@ export const addJavaScriptArtifactFiles = (
|
||||
? "JavaScript syntax could not be parsed."
|
||||
: `JavaScript text was unavailable: ${file.text.reason}.`,
|
||||
});
|
||||
if (analyzed.application_projection_failure !== undefined)
|
||||
addUnavailableStaticParseScope(context, {
|
||||
file,
|
||||
asset: target,
|
||||
operation: "project-javascript-application",
|
||||
limitation: analyzed.application_projection_failure.reason,
|
||||
failure: analyzed.application_projection_failure.error,
|
||||
limits: javaScriptAnalysisCoverageLimits(
|
||||
analyzed.application_projection_failure.limits,
|
||||
),
|
||||
});
|
||||
const packageValue = packagesByPath.get(file.path);
|
||||
const packageValue = context.analysis.packages.find(
|
||||
({ path }) => path === file.path,
|
||||
);
|
||||
if (packageValue !== undefined && packageValue.status !== "included")
|
||||
addUnavailableStaticParseScope(context, {
|
||||
file,
|
||||
@@ -241,6 +173,9 @@ export const addJavaScriptArtifactFiles = (
|
||||
operation: "parse-package-json",
|
||||
limitation: packageValue.limitation,
|
||||
});
|
||||
const jsonValue = context.analysis.json_modules.find(
|
||||
({ path }) => path === file.path,
|
||||
);
|
||||
if (jsonValue !== undefined && jsonValue.status !== "included")
|
||||
addUnavailableStaticParseScope(context, {
|
||||
file,
|
||||
@@ -248,7 +183,9 @@ export const addJavaScriptArtifactFiles = (
|
||||
operation: "parse-json-module",
|
||||
limitation: jsonValue.limitation,
|
||||
});
|
||||
const sourceMap = sourceMapsByPath.get(file.path);
|
||||
const sourceMap = context.analysis.source_maps.find(
|
||||
({ path }) => path === file.path,
|
||||
);
|
||||
if (sourceMap !== undefined && sourceMap.status === "invalid")
|
||||
addUnavailableStaticParseScope(context, {
|
||||
file,
|
||||
@@ -259,23 +196,6 @@ export const addJavaScriptArtifactFiles = (
|
||||
}
|
||||
};
|
||||
|
||||
const sourceFileRange = (
|
||||
file: JavaScriptArtifactFile,
|
||||
): JavaScriptSourceRange => {
|
||||
if (!file.text.included)
|
||||
throw new TypeError("Retained static facts require their source text");
|
||||
let line = 1;
|
||||
let lastLineStart = 0;
|
||||
for (const match of file.text.value.matchAll(/\r\n|[\n\r\u2028\u2029]/gu)) {
|
||||
line += 1;
|
||||
lastLineStart = match.index + match[0].length;
|
||||
}
|
||||
return {
|
||||
start: { line: 1, column: 0 },
|
||||
end: { line, column: file.text.value.length - lastLineStart },
|
||||
};
|
||||
};
|
||||
|
||||
/** Project package metadata and its declared Electron roles. */
|
||||
export const addJavaScriptPackageNodes = (
|
||||
context: JavaScriptArtifactGraphContext,
|
||||
@@ -350,9 +270,11 @@ const createFileTarget = (
|
||||
context: JavaScriptArtifactGraphContext,
|
||||
file: JavaScriptArtifactFile,
|
||||
javascript: JavaScriptModuleArtifactAnalysis["files"][number]["javascript"],
|
||||
json: JavaScriptModuleArtifactAnalysis["json_modules"][number] | undefined,
|
||||
): ApplicationNode => {
|
||||
const kind = artifactFileNodeKind(file.kind);
|
||||
const json = context.analysis.json_modules.find(
|
||||
({ path }) => path === file.path,
|
||||
);
|
||||
return context.accumulator.addNode({
|
||||
kind,
|
||||
identity: {
|
||||
@@ -387,16 +309,6 @@ const createFileTarget = (
|
||||
});
|
||||
};
|
||||
|
||||
const indexFirstByPath = <Value extends { readonly path: string }>(
|
||||
values: readonly Value[],
|
||||
): ReadonlyMap<string, Value> => {
|
||||
const index = new Map<string, Value>();
|
||||
for (const value of values) {
|
||||
if (!index.has(value.path)) index.set(value.path, value);
|
||||
}
|
||||
return index;
|
||||
};
|
||||
|
||||
const createAsarEntry = (
|
||||
context: JavaScriptArtifactGraphContext,
|
||||
file: JavaScriptArtifactFile,
|
||||
@@ -485,4 +397,3 @@ const containerProperties = (container: JavaScriptArtifactContainer) => ({
|
||||
bytes: container.bytes,
|
||||
inventory_artifact_id: container.inventory_artifact_id,
|
||||
});
|
||||
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||
|
||||
@@ -234,8 +234,7 @@ const resolvePackageSpecifier = (
|
||||
if (rawExports === undefined || rawExports === null) {
|
||||
if (subpath === ".")
|
||||
return resolveDirectory(input, directory, metadata, new Set());
|
||||
// Preserve the trailing slash for a package directory request.
|
||||
const candidate = confineCandidate(`${directory}/${subpath.slice(2)}`);
|
||||
const candidate = confineCandidate(posix.join(directory, subpath.slice(2)));
|
||||
return typeof candidate === "string"
|
||||
? resolveCandidate(input, candidate, new Set())
|
||||
: candidate;
|
||||
|
||||
@@ -11,22 +11,15 @@ import { createJavaScriptArtifactReader as createReader } from "../../artifacts/
|
||||
import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js";
|
||||
import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js";
|
||||
import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { parseOwnedJavaScriptApplicationAnalysisSteps } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
|
||||
import type { ArtifactInventoryPartialObservation } from "../../domain/artifactPartialObservation.js";
|
||||
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
|
||||
import { analyzeAndAdoptJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js";
|
||||
import { createJavaScriptSourceAnalysis } from "../../artifacts/javascript/JavaScriptSourceAnalysis.js";
|
||||
import { JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||
import { createOwnedJavaScriptApplicationEvidenceCooperatively } from "./JavaScriptApplicationEvidence.js";
|
||||
import { analyzeAndProjectJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js";
|
||||
import { readJavaScriptArtifactFiles } from "../../artifacts/javascript/JavaScriptArtifactFiles.js";
|
||||
import { buildImmutableJavaScriptArtifactGraphSteps } from "./JavaScriptArtifactGraphBuilder.js";
|
||||
import {
|
||||
javascriptArtifactReconstructionInputSchema,
|
||||
type JavaScriptArtifactReconstructionInput,
|
||||
} from "./JavaScriptArtifactReconstructionInput.js";
|
||||
import { scanCanonicalArtifactInventoryInScope } from "../../artifacts/inventory/scanCanonical.js";
|
||||
import { scanCanonicalArtifactInventory } from "../../artifacts/inventory/scanCanonical.js";
|
||||
import { summarizeElectronBoundaries } from "./ElectronBoundaryAnalysis.js";
|
||||
import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
|
||||
import type { ProgressReporter } from "../ProgressReporter.js";
|
||||
@@ -63,241 +56,118 @@ export interface JavaScriptArtifactReconstructionResult {
|
||||
/** Reconstruct one local ASAR or extracted directory without executing code. */
|
||||
export const reconstructJavaScriptArtifact = async (
|
||||
rawInput: unknown,
|
||||
scope: ArtifactResourceScope,
|
||||
signal?: AbortSignal,
|
||||
progress?: ProgressReporter,
|
||||
): Promise<JavaScriptArtifactReconstructionResult> =>
|
||||
scope.run(async () => {
|
||||
let effectiveSignal = signal;
|
||||
let effectiveProgress = progress;
|
||||
const reportPhase = async (
|
||||
phase: string,
|
||||
message: string,
|
||||
): Promise<void> => {
|
||||
await effectiveProgress?.report({
|
||||
phase,
|
||||
completed: 0,
|
||||
total: 1,
|
||||
message,
|
||||
});
|
||||
await checkpointJavaScriptAnalysis(effectiveSignal);
|
||||
};
|
||||
const input = javascriptArtifactReconstructionInputSchema.parse(rawInput);
|
||||
abortIfNeeded(signal);
|
||||
const path = await resolveSelectedInput(input.input_path);
|
||||
const format = await resolveFormat(path, input);
|
||||
const snapshot = await scanCanonicalArtifactInventoryInScope(path, {
|
||||
resourceScope: scope,
|
||||
signal,
|
||||
integrity: { mode: input.integrity_policy },
|
||||
});
|
||||
if (snapshot.manifest.root_format !== format)
|
||||
throw new ArtifactReaderFailure(
|
||||
"format",
|
||||
`Artifact inventory classified ${path} as ${snapshot.manifest.root_format}, not ${format}`,
|
||||
);
|
||||
const reader = createReader(path, format);
|
||||
const readerOwner = {
|
||||
kind: "reader" as const,
|
||||
reader,
|
||||
resource: `JavaScript artifact reader for ${path}`,
|
||||
};
|
||||
let readerOpen = true;
|
||||
let reconstructionOutcome:
|
||||
| {
|
||||
readonly kind: "completed";
|
||||
readonly result: JavaScriptArtifactReconstructionResult;
|
||||
}
|
||||
| { readonly kind: "failed"; readonly cause: unknown };
|
||||
try {
|
||||
await reportPhase(
|
||||
"read_javascript_artifacts",
|
||||
"Reading inventoried JavaScript application sources",
|
||||
);
|
||||
const files = await readJavaScriptArtifactFiles(
|
||||
reader,
|
||||
snapshot,
|
||||
scope,
|
||||
signal,
|
||||
);
|
||||
// Release acquisition before analysis consumes these owned file facts.
|
||||
const earlyCleanup = await scope.release(readerOwner);
|
||||
readerOpen = false;
|
||||
if (earlyCleanup.kind === "failed")
|
||||
throw ArtifactReaderFailure.withCleanup(
|
||||
earlyCleanup.cause,
|
||||
ArtifactReaderFailure.cleanupObservation(
|
||||
earlyCleanup.cause,
|
||||
readerOwner.resource,
|
||||
),
|
||||
{ kind: "artifact-inventory", inventory: snapshot },
|
||||
);
|
||||
await reportPhase(
|
||||
"parse_javascript_sources",
|
||||
`Parsing and projecting ${String(files.files.length)} application source files`,
|
||||
);
|
||||
const semanticProjection = createJavaScriptSemanticGraphProjection();
|
||||
const worker = createJavaScriptSourceAnalysis(
|
||||
{
|
||||
heapMb:
|
||||
input.max_heap_mb ?? JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS.heapMb,
|
||||
timeoutMs:
|
||||
input.analysis_timeout_ms ??
|
||||
JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS.timeoutMs,
|
||||
},
|
||||
files.files.filter(
|
||||
(file) => file.kind === "javascript" && file.text.included,
|
||||
).length,
|
||||
);
|
||||
let outcome;
|
||||
let cleanupFailure: AnalysisError | undefined;
|
||||
try {
|
||||
outcome = await analyzeAndAdoptJavaScriptArtifactFiles(
|
||||
files,
|
||||
worker,
|
||||
semanticProjection,
|
||||
async (file, completed, total) => {
|
||||
abortIfNeeded(signal);
|
||||
await progress?.report({
|
||||
phase: "parse_javascript_source",
|
||||
completed: 0,
|
||||
total: 1,
|
||||
message: `Parsing and projecting ${file.path} (${String(completed + 1)}/${String(total)})`,
|
||||
});
|
||||
abortIfNeeded(signal);
|
||||
},
|
||||
signal,
|
||||
);
|
||||
} finally {
|
||||
try {
|
||||
const cleanup = await scope.release({
|
||||
kind: "javascript-source-analysis",
|
||||
analysis: worker,
|
||||
resource: `JavaScript source analyzer for ${path}`,
|
||||
});
|
||||
if (cleanup.kind === "failed") throw cleanup.cause;
|
||||
} catch (cause: unknown) {
|
||||
if (!(cause instanceof AnalysisError)) throw cause;
|
||||
cleanupFailure = cause;
|
||||
}
|
||||
}
|
||||
const analysis = outcome.analysis;
|
||||
const interruption = cleanupFailure ?? outcome.interruption;
|
||||
if (interruption !== null && interruption !== undefined) {
|
||||
effectiveSignal = undefined;
|
||||
effectiveProgress = undefined;
|
||||
}
|
||||
await checkpointJavaScriptAnalysis(effectiveSignal);
|
||||
abortIfNeeded(effectiveSignal);
|
||||
await reportPhase(
|
||||
"build_javascript_application_graph",
|
||||
"Constructing application and Electron boundary relationships",
|
||||
);
|
||||
const applicationGraphSteps = buildImmutableJavaScriptArtifactGraphSteps(
|
||||
snapshot,
|
||||
files,
|
||||
analysis,
|
||||
);
|
||||
await reportPhase(
|
||||
"seal_javascript_application_graph",
|
||||
"Sealing the validated application graph",
|
||||
);
|
||||
const graph = await completeJavaScriptAnalysisSteps(
|
||||
applicationGraphSteps,
|
||||
effectiveSignal,
|
||||
);
|
||||
await reportPhase(
|
||||
"build_javascript_semantic_graph",
|
||||
"Binding and validating static semantic relationships",
|
||||
);
|
||||
const semanticGraphSteps = semanticProjection.finishImmutableSteps(
|
||||
snapshot.manifest.root_sha256,
|
||||
graph,
|
||||
);
|
||||
await reportPhase(
|
||||
"seal_javascript_semantic_graph",
|
||||
"Validating and sealing the semantic graph",
|
||||
);
|
||||
const semanticGraph = await completeJavaScriptAnalysisSteps(
|
||||
semanticGraphSteps,
|
||||
effectiveSignal,
|
||||
);
|
||||
await checkpointJavaScriptAnalysis(effectiveSignal);
|
||||
const completed = {
|
||||
input_path: path,
|
||||
format,
|
||||
root_artifact_sha256: snapshot.manifest.root_sha256,
|
||||
inventory_manifest_id: snapshot.manifest.manifest_id,
|
||||
inventory_graph_sha256: snapshot.manifest.graph_sha256,
|
||||
integrity_contradictions: snapshot.integrity_contradictions,
|
||||
graph,
|
||||
semantic_graph: semanticGraph,
|
||||
electron_summary: summarizeElectronBoundaries(analysis),
|
||||
statistics: {
|
||||
relevant_files: files.files.length,
|
||||
nested_asar_containers: files.containers.length,
|
||||
text_bytes_read: files.text_bytes_read,
|
||||
invalid_utf8_files: files.invalid_utf8_files,
|
||||
parsed_javascript_files: analysis.files.filter(
|
||||
({ javascript }) => javascript !== null,
|
||||
).length,
|
||||
visited_ast_nodes: analysis.visited_ast_nodes,
|
||||
findings: analysis.findings,
|
||||
modules: analysis.modules,
|
||||
parse_failures: analysis.parse_failures,
|
||||
},
|
||||
limitations: analysis.limitations,
|
||||
};
|
||||
if (interruption !== null && interruption !== undefined) {
|
||||
const { electron_summary: summary, ...application } = completed;
|
||||
const partial = await completeJavaScriptAnalysisSteps(
|
||||
parseOwnedJavaScriptApplicationAnalysisSteps({
|
||||
...application,
|
||||
summary,
|
||||
limitations: completed.graph.limitations,
|
||||
}),
|
||||
);
|
||||
const { max_heap_mb, analysis_timeout_ms, ...coreInput } = input;
|
||||
const parameters = {
|
||||
...coreInput,
|
||||
...(max_heap_mb === undefined ? {} : { max_heap_mb }),
|
||||
...(analysis_timeout_ms === undefined ? {} : { analysis_timeout_ms }),
|
||||
};
|
||||
const evidence =
|
||||
await createOwnedJavaScriptApplicationEvidenceCooperatively(
|
||||
parameters,
|
||||
partial,
|
||||
);
|
||||
throw interruption.retainPartialObservation(evidence);
|
||||
}
|
||||
reconstructionOutcome = { kind: "completed", result: completed };
|
||||
} catch (cause: unknown) {
|
||||
reconstructionOutcome = { kind: "failed", cause };
|
||||
}
|
||||
if (readerOpen) {
|
||||
const cleanupAttempt = await scope.release(readerOwner);
|
||||
if (cleanupAttempt.kind === "failed") {
|
||||
const cleanup = ArtifactReaderFailure.cleanupObservation(
|
||||
cleanupAttempt.cause,
|
||||
readerOwner.resource,
|
||||
);
|
||||
const partialObservation: ArtifactInventoryPartialObservation = {
|
||||
kind: "artifact-inventory",
|
||||
inventory: snapshot,
|
||||
};
|
||||
throw ArtifactReaderFailure.withCleanup(
|
||||
reconstructionOutcome.kind === "failed"
|
||||
? reconstructionOutcome.cause
|
||||
: cleanupAttempt.cause,
|
||||
cleanup,
|
||||
partialObservation,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (reconstructionOutcome.kind === "failed")
|
||||
throw reconstructionOutcome.cause;
|
||||
return reconstructionOutcome.result;
|
||||
): Promise<JavaScriptArtifactReconstructionResult> => {
|
||||
const reportPhase = async (phase: string, message: string): Promise<void> => {
|
||||
await progress?.report({ phase, completed: 0, total: 1, message });
|
||||
await checkpointJavaScriptAnalysis(signal);
|
||||
};
|
||||
const input = javascriptArtifactReconstructionInputSchema.parse(rawInput);
|
||||
abortIfNeeded(signal);
|
||||
const path = await resolveSelectedInput(input.input_path);
|
||||
const format = await resolveFormat(path, input);
|
||||
const snapshot = await scanCanonicalArtifactInventory(path, {
|
||||
signal,
|
||||
integrity: { mode: input.integrity_policy },
|
||||
});
|
||||
if (snapshot.manifest.root_format !== format)
|
||||
throw new ArtifactReaderFailure(
|
||||
"format",
|
||||
`Artifact inventory classified ${path} as ${snapshot.manifest.root_format}, not ${format}`,
|
||||
);
|
||||
const reader = createReader(path, format);
|
||||
try {
|
||||
await reportPhase(
|
||||
"read_javascript_artifacts",
|
||||
"Reading inventoried JavaScript application sources",
|
||||
);
|
||||
const files = await readJavaScriptArtifactFiles(reader, snapshot, signal);
|
||||
await reportPhase(
|
||||
"parse_javascript_sources",
|
||||
`Parsing and projecting ${String(files.files.length)} application source files`,
|
||||
);
|
||||
const semanticProjection = createJavaScriptSemanticGraphProjection();
|
||||
const analysis = await analyzeAndProjectJavaScriptArtifactFiles(
|
||||
files,
|
||||
semanticProjection.projectFileSteps,
|
||||
async (file, completed, total) => {
|
||||
abortIfNeeded(signal);
|
||||
await progress?.report({
|
||||
phase: "parse_javascript_source",
|
||||
completed: 0,
|
||||
total: 1,
|
||||
message: `Parsing and projecting ${file.path} (${String(completed + 1)}/${String(total)})`,
|
||||
});
|
||||
abortIfNeeded(signal);
|
||||
},
|
||||
signal,
|
||||
);
|
||||
await checkpointJavaScriptAnalysis(signal);
|
||||
abortIfNeeded(signal);
|
||||
await reportPhase(
|
||||
"build_javascript_application_graph",
|
||||
"Constructing application and Electron boundary relationships",
|
||||
);
|
||||
const applicationGraphSteps = buildImmutableJavaScriptArtifactGraphSteps(
|
||||
snapshot,
|
||||
files,
|
||||
analysis,
|
||||
);
|
||||
await reportPhase(
|
||||
"seal_javascript_application_graph",
|
||||
"Sealing the validated application graph",
|
||||
);
|
||||
const graph = await completeJavaScriptAnalysisSteps(
|
||||
applicationGraphSteps,
|
||||
signal,
|
||||
);
|
||||
await reportPhase(
|
||||
"build_javascript_semantic_graph",
|
||||
"Binding and validating static semantic relationships",
|
||||
);
|
||||
const semanticGraphSteps = semanticProjection.finishImmutableSteps(
|
||||
snapshot.manifest.root_sha256,
|
||||
graph,
|
||||
);
|
||||
await reportPhase(
|
||||
"seal_javascript_semantic_graph",
|
||||
"Validating and sealing the semantic graph",
|
||||
);
|
||||
const semanticGraph = await completeJavaScriptAnalysisSteps(
|
||||
semanticGraphSteps,
|
||||
signal,
|
||||
);
|
||||
await checkpointJavaScriptAnalysis(signal);
|
||||
return {
|
||||
input_path: path,
|
||||
format,
|
||||
root_artifact_sha256: snapshot.manifest.root_sha256,
|
||||
inventory_manifest_id: snapshot.manifest.manifest_id,
|
||||
inventory_graph_sha256: snapshot.manifest.graph_sha256,
|
||||
integrity_contradictions: snapshot.integrity_contradictions,
|
||||
graph,
|
||||
semantic_graph: semanticGraph,
|
||||
electron_summary: summarizeElectronBoundaries(analysis),
|
||||
statistics: {
|
||||
relevant_files: files.files.length,
|
||||
nested_asar_containers: files.containers.length,
|
||||
text_bytes_read: files.text_bytes_read,
|
||||
invalid_utf8_files: files.invalid_utf8_files,
|
||||
parsed_javascript_files: analysis.files.filter(
|
||||
({ javascript }) => javascript !== null,
|
||||
).length,
|
||||
visited_ast_nodes: analysis.visited_ast_nodes,
|
||||
findings: analysis.findings,
|
||||
modules: analysis.modules,
|
||||
parse_failures: analysis.parse_failures,
|
||||
},
|
||||
limitations: analysis.limitations,
|
||||
};
|
||||
} finally {
|
||||
await reader.close();
|
||||
}
|
||||
};
|
||||
|
||||
const OPERATION = "analyze_javascript_application";
|
||||
|
||||
|
||||
@@ -1,13 +1,11 @@
|
||||
import { z } from "zod";
|
||||
import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js";
|
||||
import { javaScriptAnalysisResourceControlsSchema } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||
|
||||
/** Local ASAR/directory reconstruction request. */
|
||||
export const javascriptArtifactReconstructionInputSchema = z.strictObject({
|
||||
input_path: z.string().min(1),
|
||||
format: z.enum(["auto", "asar", "directory"]).default("auto"),
|
||||
integrity_policy: artifactIntegrityPolicySchema,
|
||||
...javaScriptAnalysisResourceControlsSchema.shape,
|
||||
});
|
||||
|
||||
/** Parsed local reconstruction request. */
|
||||
|
||||
@@ -8,7 +8,7 @@ import { describe, expect, it } from "vitest";
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { staticPath } from "../../domain/javascript/javascriptStaticAnalysisHelpers.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
const urlCases = [
|
||||
["./preload%20name.cjs", "preload name.cjs", "main.mjs"],
|
||||
|
||||
@@ -7,7 +7,7 @@ import { promisify } from "node:util";
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
|
||||
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
|
||||
const filesFor = (paths: readonly string[]) =>
|
||||
|
||||
@@ -5,7 +5,7 @@ import { expect, it } from "vitest";
|
||||
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
|
||||
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
|
||||
import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
|
||||
|
||||
it("classifies overloaded open calls from lexical receiver facts", async () => {
|
||||
const inputPath = await createTestTempDirectory(
|
||||
|
||||
@@ -355,7 +355,7 @@ const semanticNodeFor = (roleKey: string) => {
|
||||
unpacked: false,
|
||||
text: { included: true, value: "" },
|
||||
};
|
||||
const state = createSemanticGraphProjectionState();
|
||||
const state = createSemanticGraphProjectionState({ nodes: [] });
|
||||
return constructSemanticGraphNode(
|
||||
file,
|
||||
{
|
||||
@@ -565,7 +565,7 @@ it("does not mark a file truncated when it exactly fills its budget", () => {
|
||||
it("only records node loss when the budget rejects a new identity", () => {
|
||||
// The truncation signal must record blocked node creation, not a zero
|
||||
// remaining budget: a file using exactly its share drops nothing.
|
||||
const state = createSemanticGraphProjectionState();
|
||||
const state = createSemanticGraphProjectionState({ nodes: [] });
|
||||
state.fileNodeBudget = 2;
|
||||
state.fileNodesDropped = false;
|
||||
const first = addSemanticGraphNode(state, semanticNodeFor("first"));
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import {
|
||||
createJavaScriptSemanticGraph,
|
||||
createJavaScriptSemanticGraphUnknown,
|
||||
sealTransferredJavaScriptSemanticGraphSteps,
|
||||
type JavaScriptSemanticGraph,
|
||||
} from "../../domain/javascript/javascriptSemanticGraph.js";
|
||||
@@ -11,7 +10,6 @@ import type {
|
||||
import {
|
||||
JAVASCRIPT_SEMANTIC_RELATION_FAMILIES,
|
||||
JAVASCRIPT_SEMANTIC_RELATION_FAMILY,
|
||||
JAVASCRIPT_SEMANTIC_RELATIONS,
|
||||
} from "../../domain/javascript/javascriptSemanticGraphSchemas.js";
|
||||
import type {
|
||||
JavaScriptSemanticCallArgument,
|
||||
@@ -35,13 +33,6 @@ import { projectSemanticResources } from "./JavaScriptSemanticGraphResourceProje
|
||||
import { projectSemanticObjects } from "./JavaScriptSemanticGraphObjectProjection.js";
|
||||
import { projectSemanticValues } from "./JavaScriptSemanticGraphValueProjection.js";
|
||||
import { inferredSemanticEvidenceAt } from "./JavaScriptSemanticGraphEvidence.js";
|
||||
import { observedSemanticEvidence } from "./JavaScriptSemanticGraphEvidence.js";
|
||||
import {
|
||||
partialApplicationCoverage,
|
||||
type ApplicationCoverage,
|
||||
} from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
|
||||
import type { JsonValue } from "../../domain/jsonValue.js";
|
||||
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
|
||||
import { projectSemanticFunctionFingerprints } from "./JavaScriptSemanticGraphFingerprintProjection.js";
|
||||
import {
|
||||
projectSemanticClosureCaptures,
|
||||
@@ -64,7 +55,6 @@ import {
|
||||
bindSemanticGraphApplicationNodes,
|
||||
type SemanticGraphProjectionState as BuilderState,
|
||||
} from "./JavaScriptSemanticGraphConstruction.js";
|
||||
import type { JavaScriptSemanticFileProjection } from "../../domain/javascript/javascriptSemanticFileProjection.js";
|
||||
|
||||
interface BuilderInput {
|
||||
readonly rootArtifactSha256: string;
|
||||
@@ -136,19 +126,6 @@ export const buildJavaScriptSemanticGraph = ({
|
||||
|
||||
/** File-local semantic projection that does not retain consumed source IR. */
|
||||
export interface JavaScriptSemanticGraphProjection {
|
||||
/** Node capacity offered to the next independently analyzed source file. */
|
||||
readonly remainingFileNodeBudget: () => number;
|
||||
/** Adopt decoded file-local facts; final validation and proof remain in this realm. */
|
||||
readonly adoptFileProjection: (
|
||||
projection: JavaScriptSemanticFileProjection,
|
||||
) => void;
|
||||
/** Preserve source-bound unknowns for an execution or result-retention failure. */
|
||||
readonly recordUnavailableFile: (
|
||||
file: JavaScriptArtifactFile,
|
||||
reason: string,
|
||||
limits: Readonly<Record<string, JsonValue>> | null,
|
||||
unknownReason: "resource-limit" | "incomplete-module",
|
||||
) => void;
|
||||
readonly projectFile: (
|
||||
file: JavaScriptArtifactFile,
|
||||
ir: JavaScriptSemanticIr,
|
||||
@@ -171,15 +148,9 @@ export interface JavaScriptSemanticGraphProjection {
|
||||
/** Accumulate semantic nodes while each source IR is still file-local. */
|
||||
export const createJavaScriptSemanticGraphProjection =
|
||||
(): JavaScriptSemanticGraphProjection => {
|
||||
const state = emptyState();
|
||||
const state = emptyState({ nodes: [] });
|
||||
const fingerprints: JavaScriptSemanticFingerprint[] = [];
|
||||
let truncatedFiles = 0;
|
||||
let unavailableFiles = 0;
|
||||
let resourceTruncatedFiles = 0;
|
||||
const executionLimits = new Map<
|
||||
string,
|
||||
ApplicationCoverage["limits"][number]
|
||||
>();
|
||||
const projectSourceSteps = function* (
|
||||
file: JavaScriptArtifactFile,
|
||||
ir: JavaScriptSemanticIr,
|
||||
@@ -236,15 +207,12 @@ export const createJavaScriptSemanticGraphProjection =
|
||||
fingerprints: [...fingerprints],
|
||||
unknowns,
|
||||
coverage: {
|
||||
status:
|
||||
truncatedFiles > 0 || unavailableFiles > 0 ? "partial" : "unknown",
|
||||
truncated: truncatedFiles > 0 || resourceTruncatedFiles > 0,
|
||||
omitted_nodes: truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
|
||||
omitted_relations:
|
||||
truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
|
||||
limits: [
|
||||
...executionLimits.values(),
|
||||
...(truncatedFiles > 0
|
||||
status: truncatedFiles > 0 ? "partial" : "unknown",
|
||||
truncated: truncatedFiles > 0,
|
||||
omitted_nodes: truncatedFiles > 0 ? null : 0,
|
||||
omitted_relations: truncatedFiles > 0 ? null : 0,
|
||||
limits:
|
||||
truncatedFiles > 0
|
||||
? [
|
||||
{
|
||||
name: "semantic_graph_node_ceiling",
|
||||
@@ -252,14 +220,12 @@ export const createJavaScriptSemanticGraphProjection =
|
||||
unit: "items" as const,
|
||||
},
|
||||
]
|
||||
: []),
|
||||
],
|
||||
: [],
|
||||
families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({
|
||||
family,
|
||||
status: semanticFamilyStatus(family),
|
||||
retained_relations: retainedByFamily.get(family) ?? 0,
|
||||
omitted_relations:
|
||||
truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
|
||||
omitted_relations: truncatedFiles > 0 ? null : 0,
|
||||
unknown_ids: unknownIdsByFamily.get(family) ?? [],
|
||||
})),
|
||||
},
|
||||
@@ -286,87 +252,6 @@ export const createJavaScriptSemanticGraphProjection =
|
||||
return graph;
|
||||
};
|
||||
return {
|
||||
remainingFileNodeBudget: () =>
|
||||
Math.min(
|
||||
SEMANTIC_GRAPH_FILE_NODE_CEILING,
|
||||
Math.max(0, SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size),
|
||||
),
|
||||
adoptFileProjection: (projection) => {
|
||||
const newNodes = projection.nodes.filter(
|
||||
({ node_id }) => !state.nodes.has(node_id),
|
||||
);
|
||||
if (
|
||||
newNodes.length >
|
||||
Math.min(
|
||||
SEMANTIC_GRAPH_FILE_NODE_CEILING,
|
||||
SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size,
|
||||
)
|
||||
)
|
||||
throw new RangeError(
|
||||
"Transferred semantic nodes exceed the offered tree capacity",
|
||||
);
|
||||
for (const context of projection.evidenceContexts)
|
||||
state.evidenceContexts.adopt(context);
|
||||
for (const node of newNodes) state.nodes.set(node.node_id, node);
|
||||
for (const relation of projection.relations)
|
||||
state.relations.set(relation.relation_id, relation);
|
||||
for (const unknown of projection.unknowns)
|
||||
state.unknowns.set(unknown.unknown_id, unknown);
|
||||
for (const root of projection.roots) state.roots.add(root);
|
||||
fingerprints.push(...projection.fingerprints);
|
||||
if (projection.truncated) truncatedFiles += 1;
|
||||
},
|
||||
recordUnavailableFile: (file, reason, limits, unknownReason) => {
|
||||
const retainedLimits = javaScriptAnalysisCoverageLimits(limits);
|
||||
for (const limit of retainedLimits) {
|
||||
executionLimits.set(
|
||||
JSON.stringify([limit.name, limit.value, limit.unit]),
|
||||
limit,
|
||||
);
|
||||
}
|
||||
state.fileNodeBudget = Math.min(
|
||||
SEMANTIC_GRAPH_FILE_NODE_CEILING,
|
||||
Math.max(0, SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size),
|
||||
);
|
||||
const module = retainNode(state, file, {
|
||||
kind: "module",
|
||||
roleKey: "module",
|
||||
location: null,
|
||||
label: file.path,
|
||||
functionNodeId: null,
|
||||
});
|
||||
state.fileNodeBudget = null;
|
||||
if (module !== null) state.roots.add(module.node_id);
|
||||
const evidence = {
|
||||
...observedSemanticEvidence(file, null),
|
||||
authority: "unknown" as const,
|
||||
state: "unknown" as const,
|
||||
confidence: "unknown" as const,
|
||||
coverage: partialApplicationCoverage(retainedLimits, null),
|
||||
limitations: [reason],
|
||||
};
|
||||
for (const family of JAVASCRIPT_SEMANTIC_RELATION_FAMILIES) {
|
||||
const unknown = createJavaScriptSemanticGraphUnknown(
|
||||
{
|
||||
node_id: module?.node_id ?? null,
|
||||
family,
|
||||
relation_kinds: JAVASCRIPT_SEMANTIC_RELATIONS.filter(
|
||||
(relation) =>
|
||||
JAVASCRIPT_SEMANTIC_RELATION_FAMILY[relation] === family,
|
||||
),
|
||||
reason: unknownReason,
|
||||
detail: reason,
|
||||
candidate_node_ids: [],
|
||||
evidence,
|
||||
},
|
||||
state.evidenceContexts,
|
||||
);
|
||||
state.unknowns.set(unknown.unknown_id, unknown);
|
||||
}
|
||||
unavailableFiles += 1;
|
||||
if (unknownReason === "resource-limit" && retainedLimits.length > 0)
|
||||
resourceTruncatedFiles += 1;
|
||||
},
|
||||
projectFile: (file, ir) => {
|
||||
const steps = projectSourceSteps(file, ir);
|
||||
while (steps.next().done !== true);
|
||||
@@ -387,35 +272,6 @@ export const createJavaScriptSemanticGraphProjection =
|
||||
};
|
||||
};
|
||||
|
||||
/** Project one source with the parent's remaining node capacity, retaining no full IR. */
|
||||
export function* projectJavaScriptSemanticFileSteps(
|
||||
file: JavaScriptArtifactFile,
|
||||
ir: JavaScriptSemanticIr,
|
||||
nodeBudget: number,
|
||||
): Generator<void, JavaScriptSemanticFileProjection> {
|
||||
if (
|
||||
!Number.isSafeInteger(nodeBudget) ||
|
||||
nodeBudget < 0 ||
|
||||
nodeBudget > SEMANTIC_GRAPH_FILE_NODE_CEILING
|
||||
)
|
||||
throw new RangeError(
|
||||
"File semantic projection requires a valid offered node capacity",
|
||||
);
|
||||
const state = emptyState();
|
||||
state.fileNodeBudget = nodeBudget;
|
||||
state.fileNodesDropped = false;
|
||||
const fingerprints = yield* projectFileSteps(file, ir, state);
|
||||
return {
|
||||
roots: [...state.roots],
|
||||
evidenceContexts: state.evidenceContexts.contexts,
|
||||
nodes: [...state.nodes.values()],
|
||||
relations: [...state.relations.values()],
|
||||
unknowns: [...state.unknowns.values()],
|
||||
fingerprints,
|
||||
truncated: Boolean(state.fileNodesDropped),
|
||||
};
|
||||
}
|
||||
|
||||
function* projectFileSteps(
|
||||
file: JavaScriptArtifactFile,
|
||||
ir: JavaScriptSemanticIr,
|
||||
|
||||
@@ -28,6 +28,7 @@ export interface SemanticGraphProjectionState {
|
||||
readonly unknowns: Map<string, JavaScriptSemanticGraphUnknown>;
|
||||
readonly evidenceContexts: JavaScriptSemanticEvidenceContextRegistry;
|
||||
readonly roots: Set<string>;
|
||||
readonly applicationNodeIdsByLocation: ReadonlyMap<string, readonly string[]>;
|
||||
/**
|
||||
* Remaining node budget for the file currently being projected.
|
||||
*
|
||||
@@ -68,29 +69,32 @@ export interface SemanticRelationConstructionInput {
|
||||
readonly properties?: Readonly<Record<string, JsonValue>>;
|
||||
}
|
||||
|
||||
/** Create empty projection state; application-node links are bound at finish. */
|
||||
export const createSemanticGraphProjectionState =
|
||||
(): SemanticGraphProjectionState => ({
|
||||
/** Create empty projection state with structural application-node mappings. */
|
||||
export const createSemanticGraphProjectionState = (
|
||||
applicationGraph: Pick<JavaScriptApplicationGraph, "nodes">,
|
||||
): SemanticGraphProjectionState => {
|
||||
const index = indexApplicationNodes(applicationGraph.nodes);
|
||||
return {
|
||||
nodes: new Map(),
|
||||
relations: new Map(),
|
||||
unknowns: new Map(),
|
||||
evidenceContexts: new JavaScriptSemanticEvidenceContextRegistry(),
|
||||
roots: new Set(),
|
||||
});
|
||||
applicationNodeIdsByLocation: index.identifiers,
|
||||
};
|
||||
};
|
||||
|
||||
/** Bind retained semantic identities after all application observations exist. */
|
||||
export const bindSemanticGraphApplicationNodes = (
|
||||
state: SemanticGraphProjectionState,
|
||||
applicationGraph: Pick<JavaScriptApplicationGraph, "nodes">,
|
||||
): void => {
|
||||
const applicationNodeIdsByLocation = indexApplicationNodes(
|
||||
applicationGraph.nodes,
|
||||
);
|
||||
const index = indexApplicationNodes(applicationGraph.nodes).identifiers;
|
||||
for (const [identifier, node] of state.nodes) {
|
||||
state.nodes.set(identifier, {
|
||||
...node,
|
||||
application_node_ids: [
|
||||
...(applicationNodeIdsByLocation.get(
|
||||
...(index.get(
|
||||
applicationLocationKey(
|
||||
node.identity.artifact_sha256,
|
||||
node.identity.module_path,
|
||||
@@ -118,7 +122,7 @@ export const constructSemanticGraphNode = (
|
||||
role_key: input.roleKey,
|
||||
},
|
||||
function_node_id: input.functionNodeId,
|
||||
application_node_ids: [],
|
||||
application_node_ids: matchingApplicationNodeIds(file, input, state),
|
||||
label: input.label === null ? null : javascriptDisplayText(input.label),
|
||||
properties: input.properties ?? {},
|
||||
evidence: observedSemanticEvidence(file, input.location),
|
||||
@@ -243,7 +247,9 @@ export const addSemanticFallbackRoot = (
|
||||
|
||||
const indexApplicationNodes = (
|
||||
nodes: JavaScriptApplicationGraph["nodes"],
|
||||
): ReadonlyMap<string, readonly string[]> => {
|
||||
): {
|
||||
readonly identifiers: ReadonlyMap<string, readonly string[]>;
|
||||
} => {
|
||||
const identifiers = new Map<string, Set<string>>();
|
||||
for (const node of nodes) {
|
||||
for (const observation of node.observations) {
|
||||
@@ -274,9 +280,19 @@ const indexApplicationNodes = (
|
||||
return [key, sorted];
|
||||
}),
|
||||
);
|
||||
return indexed;
|
||||
return { identifiers: indexed };
|
||||
};
|
||||
|
||||
const matchingApplicationNodeIds = (
|
||||
file: JavaScriptArtifactFile,
|
||||
input: SemanticNodeConstructionInput,
|
||||
state: SemanticGraphProjectionState,
|
||||
): string[] => [
|
||||
...(state.applicationNodeIdsByLocation.get(
|
||||
applicationLocationKey(file.sha256, file.path, input.location),
|
||||
) ?? []),
|
||||
];
|
||||
|
||||
const addApplicationNodeIdentifier = (
|
||||
identifiers: Map<string, Set<string>>,
|
||||
key: string,
|
||||
|
||||
@@ -325,7 +325,8 @@ const resolveUiAction = (
|
||||
(node) =>
|
||||
["control", "view", "view_controller", "scene"].includes(node.kind) &&
|
||||
(node.id === query ||
|
||||
node.attributes.interface_builder_object_id === query),
|
||||
node.attributes.interface_builder_object_id === query ||
|
||||
node.id.endsWith(`:object:${query}`)),
|
||||
);
|
||||
const actionIds = new Set(
|
||||
graph.edges.flatMap((edge) =>
|
||||
|
||||
@@ -7,7 +7,6 @@ import {
|
||||
import { ghidraFunctionDossier } from "../../domain/ghidraValues.fixture.js";
|
||||
import { functionDossierSchema } from "../../domain/hopperValues.js";
|
||||
import { jsonValueSchema } from "../../domain/jsonValue.js";
|
||||
import { nativeValueTraceSchema } from "../../domain/native/nativeValueTrace.js";
|
||||
import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js";
|
||||
import { ok, err } from "../../domain/result.js";
|
||||
|
||||
@@ -103,132 +102,6 @@ const analysis: AnalysisOperationPort = {
|
||||
);
|
||||
},
|
||||
};
|
||||
describe("native value parameter-use node membership", () => {
|
||||
it("retains only parameter uses whose parameter nodes fit the node budget", async () => {
|
||||
const operationCount = 3_000;
|
||||
const parameterUseCount = 12_000;
|
||||
const base = functionDossierSchema.parse(ghidraFunctionDossier());
|
||||
const procedureAddress = base.procedure.address;
|
||||
const largeFlow = {
|
||||
available: true as const,
|
||||
provenance: "ghidra-high-pcode" as const,
|
||||
operations: Array.from({ length: operationCount }, (_, index) => ({
|
||||
...node(`operation-${String(index)}`, procedureAddress, "MULTIEQUAL"),
|
||||
inputs: [scalar, scalar, scalar, scalar],
|
||||
})),
|
||||
def_use: [],
|
||||
effects: [],
|
||||
parameters: [{ ordinal: 0, name: "value", data_type: "/int" }],
|
||||
parameter_uses: Array.from({ length: parameterUseCount }, (_, index) => ({
|
||||
ordinal: 0,
|
||||
use: `operation-${String(index % operationCount)}`,
|
||||
input_index: Math.floor(index / operationCount),
|
||||
})),
|
||||
truncated: false,
|
||||
omitted_operations_lower_bound: 0,
|
||||
known_omitted_inputs: 0,
|
||||
known_omitted_edges: 0,
|
||||
limitations: [],
|
||||
};
|
||||
const largeDossier = {
|
||||
...base,
|
||||
native_value_flow: largeFlow,
|
||||
};
|
||||
const largeAnalysis: AnalysisOperationPort = {
|
||||
execute: async () =>
|
||||
ok(createAnalysisExecution(largeDossier, provider, { subject })),
|
||||
};
|
||||
|
||||
const includedParameter = await traceNativeValues(largeAnalysis, {
|
||||
procedure: procedureAddress,
|
||||
max_nodes: 20_000,
|
||||
max_edges: 40_000,
|
||||
offset: operationCount,
|
||||
limit: 1,
|
||||
});
|
||||
if (!includedParameter.ok) throw includedParameter.error;
|
||||
const includedResult = nativeValueTraceSchema.parse(
|
||||
includedParameter.value,
|
||||
);
|
||||
expect(includedResult).toMatchObject({
|
||||
total_nodes: operationCount + 1,
|
||||
total_edges: parameterUseCount,
|
||||
nodes: [
|
||||
expect.objectContaining({
|
||||
id: `${procedureAddress}/parameter:0`,
|
||||
kind: "parameter",
|
||||
}),
|
||||
],
|
||||
truncated: true,
|
||||
});
|
||||
expect(includedResult.edges).toHaveLength(parameterUseCount);
|
||||
|
||||
const omittedParameter = await traceNativeValues(largeAnalysis, {
|
||||
procedure: procedureAddress,
|
||||
max_nodes: operationCount,
|
||||
max_edges: 40_000,
|
||||
offset: operationCount - 1,
|
||||
limit: 1,
|
||||
});
|
||||
if (!omittedParameter.ok) throw omittedParameter.error;
|
||||
const omittedResult = nativeValueTraceSchema.parse(omittedParameter.value);
|
||||
expect(omittedResult).toMatchObject({
|
||||
total_nodes: operationCount,
|
||||
total_edges: 0,
|
||||
nodes: [expect.objectContaining({ kind: "operation" })],
|
||||
edges: [],
|
||||
truncated: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves operation IDs that collide with generated parameter IDs", async () => {
|
||||
const base = functionDossierSchema.parse(ghidraFunctionDossier());
|
||||
const address = base.procedure.address;
|
||||
const execution = createAnalysisExecution(
|
||||
{
|
||||
...base,
|
||||
native_value_flow: {
|
||||
available: true,
|
||||
provenance: "ghidra-high-pcode",
|
||||
operations: [
|
||||
node("parameter:0", address, "COPY"),
|
||||
node("use", address, "COPY"),
|
||||
],
|
||||
def_use: [],
|
||||
effects: [],
|
||||
parameters: [{ ordinal: 0, name: "value", data_type: "/int" }],
|
||||
parameter_uses: [{ ordinal: 0, use: "use", input_index: 1 }],
|
||||
truncated: false,
|
||||
omitted_operations_lower_bound: 0,
|
||||
known_omitted_inputs: 0,
|
||||
known_omitted_edges: 0,
|
||||
limitations: [],
|
||||
},
|
||||
},
|
||||
provider,
|
||||
{ subject },
|
||||
);
|
||||
const result = await traceNativeValues(
|
||||
{ execute: async () => ok(execution) },
|
||||
{ procedure: address, max_nodes: 2 },
|
||||
);
|
||||
|
||||
if (!result.ok) throw result.error;
|
||||
const parsedResult = nativeValueTraceSchema.parse(result.value);
|
||||
expect(parsedResult).toMatchObject({
|
||||
total_nodes: 2,
|
||||
total_edges: 1,
|
||||
edges: [
|
||||
{
|
||||
source: `${address}/parameter:0`,
|
||||
target: `${address}/use`,
|
||||
kind: "parameter-use",
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("bounded native value dependency composition", () => {
|
||||
it("reports work truncation and stable pagination without traversing ambiguous calls", async () => {
|
||||
const bounded = await traceNativeValues(analysis, {
|
||||
|
||||
@@ -21,7 +21,6 @@ export const traceNativeValues = async (
|
||||
): EnhancedResult => {
|
||||
const input = nativeValueTraceInputSchema.parse(parameters);
|
||||
const nodes: typeof nativeValueTraceSchema._output.nodes = [];
|
||||
const nodeIds = new Set<string>();
|
||||
const edges: typeof nativeValueTraceSchema._output.edges = [];
|
||||
const procedures: typeof nativeValueTraceSchema._output.procedures = [];
|
||||
const unknowns: typeof nativeValueTraceSchema._output.unknowns = [];
|
||||
@@ -155,28 +154,25 @@ export const traceNativeValues = async (
|
||||
bytes += size;
|
||||
ids.add(operation.id);
|
||||
nodes.push(node);
|
||||
nodeIds.add(node.id);
|
||||
}
|
||||
for (const parameter of flow.parameters) {
|
||||
if (nodes.length >= input.max_nodes) {
|
||||
truncated = true;
|
||||
break;
|
||||
}
|
||||
const id = `${address}/parameter:${parameter.ordinal}`;
|
||||
nodes.push({
|
||||
id,
|
||||
id: `${address}/parameter:${parameter.ordinal}`,
|
||||
procedure: address,
|
||||
evidence_id: evidenceId,
|
||||
kind: "parameter",
|
||||
operation: null,
|
||||
parameter,
|
||||
});
|
||||
nodeIds.add(id);
|
||||
}
|
||||
for (const use of flow.parameter_uses) {
|
||||
if (
|
||||
!ids.has(use.use) ||
|
||||
!nodeIds.has(`${address}/parameter:${use.ordinal}`)
|
||||
!nodes.some((node) => node.id === `${address}/parameter:${use.ordinal}`)
|
||||
)
|
||||
continue;
|
||||
if (edges.length >= input.max_edges) {
|
||||
|
||||
@@ -60,30 +60,6 @@ describe("English Unicode case collisions", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("retains every spelling in a high-cardinality folded sibling group", () => {
|
||||
const count = 8_192;
|
||||
const variants = caseVariants("abcdefghijklm", count);
|
||||
const { limitation, byPath } = annotate(variants);
|
||||
const ordered = [...variants].sort();
|
||||
const first = ordered[0];
|
||||
const second = ordered[1];
|
||||
const original = variants[0];
|
||||
if (first === undefined || second === undefined || original === undefined) {
|
||||
throw new Error("Expected at least two case variants");
|
||||
}
|
||||
expect(limitation).toBe(ENGLISH_UNICODE_CASE_INVENTORY_LIMITATION);
|
||||
expect(byPath.size).toBe(count);
|
||||
expect(byPath.get(first)).toEqual([
|
||||
`Logical path ${first} collides under English (en-US) Unicode case folding with ${second} and ${String(count - 2)} other spellings; ${NOTE}`,
|
||||
]);
|
||||
expect(byPath.get(original)?.[0]).toContain(
|
||||
`with ${first} and ${String(count - 2)} other spellings;`,
|
||||
);
|
||||
expect(
|
||||
[...byPath.values()].every((limitations) => limitations.length === 1),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("stays linear for large case-variant subtrees", () => {
|
||||
const count = 5_000;
|
||||
const paths = Array.from({ length: count }, (_, index) => [
|
||||
@@ -105,12 +81,3 @@ describe("English Unicode case collisions", () => {
|
||||
expect(byPath.get(deep)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
const caseVariants = (lowercase: string, count: number): string[] =>
|
||||
Array.from({ length: count }, (_, mask) =>
|
||||
[...lowercase]
|
||||
.map((character, index) =>
|
||||
(mask & (1 << index)) === 0 ? character : character.toUpperCase(),
|
||||
)
|
||||
.join(""),
|
||||
);
|
||||
|
||||
@@ -177,12 +177,12 @@ const englishUnicodeCaseCollisions = (
|
||||
if (path.length === 0 || path === ".") continue;
|
||||
let node = root;
|
||||
for (const part of path.split("/")) {
|
||||
const folded = englishCaseFold(part);
|
||||
const spellings = node.spellingsByFold.get(folded);
|
||||
if (spellings === undefined) node.spellingsByFold.set(folded, [part]);
|
||||
else if (!spellings.includes(part)) spellings.push(part);
|
||||
let child = node.bySpelling.get(part);
|
||||
if (child === undefined) {
|
||||
const folded = englishCaseFold(part);
|
||||
const spellings = node.spellingsByFold.get(folded);
|
||||
if (spellings === undefined) node.spellingsByFold.set(folded, [part]);
|
||||
else spellings.push(part);
|
||||
child = emptyCaseNode();
|
||||
node.bySpelling.set(part, child);
|
||||
}
|
||||
|
||||
@@ -33,12 +33,10 @@ import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import { interfaceBuilderLimitsSchema } from "../domain/apple/interfaceBuilderGraph.js";
|
||||
import { err, ok, type Result } from "../domain/result.js";
|
||||
import { ArtifactReaderFailure } from "./ArtifactReader.js";
|
||||
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
||||
import { artifactCapabilities } from "./ArtifactProviderMetadata.js";
|
||||
import { ARTIFACT_GRAPH_PROVIDER } from "../application/InvestigationProviders.js";
|
||||
import { createEvidence } from "../domain/evidence.js";
|
||||
import { createArtifactInspection } from "../domain/artifactInspection.js";
|
||||
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
|
||||
|
||||
/** Read-only inventory and exclusively owned extraction provider. */
|
||||
export class ArtifactProvider implements AnalysisProvider {
|
||||
@@ -69,8 +67,6 @@ export class ArtifactProvider implements AnalysisProvider {
|
||||
}
|
||||
|
||||
class ArtifactClient implements AnalysisClient {
|
||||
readonly #resourceScope = new ArtifactResourceScope();
|
||||
|
||||
constructor(
|
||||
private readonly target: BinaryTarget,
|
||||
private readonly environment: Readonly<NodeJS.ProcessEnv>,
|
||||
@@ -188,7 +184,6 @@ class ArtifactClient implements AnalysisClient {
|
||||
outputRoot: parsed.output_root,
|
||||
environment: this.environment,
|
||||
integrityPolicy: parsed.integrity_policy,
|
||||
resourceScope: this.#resourceScope,
|
||||
},
|
||||
options?.signal,
|
||||
);
|
||||
@@ -225,25 +220,7 @@ class ArtifactClient implements AnalysisClient {
|
||||
}
|
||||
|
||||
close(): Promise<Result<null, AnalysisError>> {
|
||||
return this.#resourceScope.close().then(
|
||||
() => ok(null),
|
||||
(cause: unknown) => {
|
||||
const failure =
|
||||
cause instanceof ArtifactReaderFailure ? cause : undefined;
|
||||
return err(
|
||||
new ProviderCleanupError(
|
||||
ARTIFACT_GRAPH_PROVIDER.id,
|
||||
failure?.cleanup?.resources ?? ["artifact resources"],
|
||||
{
|
||||
reason:
|
||||
failure?.cleanup?.reason ??
|
||||
(cause instanceof Error ? cause.message : String(cause)),
|
||||
},
|
||||
{ cause, operation: "close_binary" },
|
||||
),
|
||||
);
|
||||
},
|
||||
);
|
||||
return Promise.resolve(ok(null));
|
||||
}
|
||||
|
||||
/** The active target's kind is outside this operation's supported targets. */
|
||||
@@ -376,7 +353,6 @@ class ArtifactClient implements AnalysisClient {
|
||||
options?: ExecutionOptions,
|
||||
) {
|
||||
return inventoryArtifact(this.target.sourcePath ?? this.target.path, {
|
||||
resourceScope: this.#resourceScope,
|
||||
environment: this.environment,
|
||||
...(options?.signal === undefined ? {} : { signal: options.signal }),
|
||||
integrity: { mode: parsed.integrity_policy },
|
||||
|
||||
@@ -31,37 +31,6 @@ export interface ArtifactEntry {
|
||||
};
|
||||
}
|
||||
|
||||
/** Snapshot mutable entry metadata before the producer later validates open(). */
|
||||
export const copyArtifactEntry = (entry: ArtifactEntry): ArtifactEntry => ({
|
||||
...entry,
|
||||
limitations: [...entry.limitations],
|
||||
...(entry.sourceIdentity === undefined
|
||||
? {}
|
||||
: { sourceIdentity: { ...entry.sourceIdentity } }),
|
||||
});
|
||||
|
||||
/** Compare every portable and adapter-owned fact supplied with an entry. */
|
||||
export const sameArtifactEntry = (
|
||||
entry: ArtifactEntry,
|
||||
produced: ArtifactEntry,
|
||||
): boolean =>
|
||||
entry.path === produced.path &&
|
||||
entry.kind === produced.kind &&
|
||||
entry.declaredSize === produced.declaredSize &&
|
||||
entry.compressedSize === produced.compressedSize &&
|
||||
entry.executable === produced.executable &&
|
||||
entry.encrypted === produced.encrypted &&
|
||||
entry.byteOffset === produced.byteOffset &&
|
||||
entry.declaredSha256 === produced.declaredSha256 &&
|
||||
entry.unpacked === produced.unpacked &&
|
||||
entry.adapterKey === produced.adapterKey &&
|
||||
entry.limitations.length === produced.limitations.length &&
|
||||
entry.limitations.every(
|
||||
(limitation, index) => limitation === produced.limitations[index],
|
||||
) &&
|
||||
entry.sourceIdentity?.device === produced.sourceIdentity?.device &&
|
||||
entry.sourceIdentity?.inode === produced.sourceIdentity?.inode;
|
||||
|
||||
/** Read-only adapter over one directory, archive, or virtual container. */
|
||||
export interface ArtifactReader {
|
||||
readonly format: "directory" | ZipPackageFormat | "asar" | "file";
|
||||
|
||||
@@ -1,114 +0,0 @@
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import { err } from "../domain/result.js";
|
||||
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
|
||||
import type { ArtifactReader } from "./ArtifactReader.js";
|
||||
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
||||
|
||||
const deferred = <T>() => {
|
||||
let resolve: (value: T) => void = () => {};
|
||||
const promise = new Promise<T>((resolvePromise) => {
|
||||
resolve = resolvePromise;
|
||||
});
|
||||
return { promise, resolve };
|
||||
};
|
||||
|
||||
it("waits for admitted operations and shares concurrent close", async () => {
|
||||
const scope = new ArtifactResourceScope();
|
||||
const started = deferred<void>();
|
||||
const finish = deferred<void>();
|
||||
const operation = scope.run(async () => {
|
||||
started.resolve();
|
||||
await finish.promise;
|
||||
});
|
||||
await started.promise;
|
||||
|
||||
const close = scope.close();
|
||||
expect(scope.close()).toBe(close);
|
||||
await expect(scope.run(async () => undefined)).rejects.toMatchObject({
|
||||
reason: "unavailable",
|
||||
message: "Artifact resource scope is closed",
|
||||
});
|
||||
let closed = false;
|
||||
void close.then(() => {
|
||||
closed = true;
|
||||
});
|
||||
await Promise.resolve();
|
||||
expect(closed).toBe(false);
|
||||
|
||||
finish.resolve();
|
||||
await operation;
|
||||
await close;
|
||||
expect(closed).toBe(true);
|
||||
});
|
||||
|
||||
it("retains the same failed reader owner and retries before admitting work", async () => {
|
||||
const scope = new ArtifactResourceScope();
|
||||
let closeAttempts = 0;
|
||||
const reader = {
|
||||
async close() {
|
||||
closeAttempts += 1;
|
||||
if (closeAttempts < 4) throw new Error(`close attempt ${closeAttempts}`);
|
||||
},
|
||||
} as unknown as ArtifactReader;
|
||||
const owner = { kind: "reader" as const, resource: "fixture reader", reader };
|
||||
|
||||
expect(await scope.release(owner)).toMatchObject({ kind: "failed" });
|
||||
expect(closeAttempts).toBe(1);
|
||||
|
||||
await expect(scope.run(async () => "must not run")).rejects.toMatchObject({
|
||||
reason: "unavailable",
|
||||
message: "Artifact cleanup remains incomplete: fixture reader",
|
||||
});
|
||||
expect(closeAttempts).toBe(2);
|
||||
await expect(scope.close()).rejects.toMatchObject({
|
||||
reason: "unavailable",
|
||||
});
|
||||
expect(closeAttempts).toBe(3);
|
||||
await scope.close();
|
||||
expect(closeAttempts).toBe(4);
|
||||
});
|
||||
|
||||
it("retains a failed source analyzer until shutdown can verify its release", async () => {
|
||||
const scope = new ArtifactResourceScope();
|
||||
let releaseAllowed = false;
|
||||
const failure = new AnalysisOutputError(
|
||||
"analyze_javascript_application",
|
||||
"Owned worker is still running",
|
||||
);
|
||||
const owner = {
|
||||
kind: "javascript-source-analysis" as const,
|
||||
resource: "fixture source analyzer",
|
||||
analysis: {
|
||||
async analyze() {
|
||||
return err({
|
||||
error: failure,
|
||||
javascript: null,
|
||||
module: null,
|
||||
projection: null,
|
||||
});
|
||||
},
|
||||
async close() {
|
||||
if (!releaseAllowed) throw failure;
|
||||
},
|
||||
},
|
||||
};
|
||||
expect(await scope.release(owner)).toEqual({
|
||||
kind: "failed",
|
||||
cause: failure,
|
||||
});
|
||||
let admitted = false;
|
||||
await expect(
|
||||
scope.run(async () => {
|
||||
admitted = true;
|
||||
}),
|
||||
).rejects.toMatchObject({
|
||||
cleanup: { resources: ["fixture source analyzer"] },
|
||||
});
|
||||
expect(admitted).toBe(false);
|
||||
await expect(scope.close()).rejects.toMatchObject({
|
||||
cleanup: { resources: ["fixture source analyzer"] },
|
||||
});
|
||||
releaseAllowed = true;
|
||||
await scope.close();
|
||||
});
|
||||
@@ -1,156 +0,0 @@
|
||||
import type { JavaScriptSourceAnalysisPort } from "../domain/javascript/javascriptSourceAnalysis.js";
|
||||
import type { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
|
||||
import type { OwnedDirectoryHandle } from "../filesystem/OwnedDirectoryHandle.js";
|
||||
|
||||
import type { ArtifactReader } from "./ArtifactReader.js";
|
||||
import { ArtifactReaderFailure } from "./ArtifactReader.js";
|
||||
import type { SafeOutputTree } from "./SafeOutputTree.js";
|
||||
|
||||
/** One artifact resource and its format-specific cleanup owner. */
|
||||
export type ArtifactResourceOwner =
|
||||
| {
|
||||
readonly kind: "javascript-source-analysis";
|
||||
readonly resource: string;
|
||||
readonly analysis: JavaScriptSourceAnalysisPort;
|
||||
}
|
||||
| {
|
||||
readonly kind: "directory-handle";
|
||||
readonly resource: string;
|
||||
readonly handle: OwnedDirectoryHandle;
|
||||
}
|
||||
| {
|
||||
readonly kind: "reader";
|
||||
readonly resource: string;
|
||||
readonly reader: ArtifactReader;
|
||||
}
|
||||
| {
|
||||
readonly kind: "file-handle";
|
||||
readonly resource: string;
|
||||
readonly handle: OwnedFileHandle;
|
||||
}
|
||||
| {
|
||||
readonly kind: "output-tree";
|
||||
readonly resource: string;
|
||||
readonly tree: SafeOutputTree;
|
||||
};
|
||||
|
||||
export type ArtifactCleanupAttempt =
|
||||
| { readonly kind: "released" }
|
||||
| { readonly kind: "failed"; readonly cause: unknown };
|
||||
|
||||
/** Caller-owned scope that retains failed artifact cleanup for a later close. */
|
||||
export class ArtifactResourceScope {
|
||||
readonly #pending = new Set<ArtifactResourceOwner>();
|
||||
readonly #active = new Set<Promise<unknown>>();
|
||||
#draining: Promise<void> | undefined;
|
||||
#closed = false;
|
||||
|
||||
/** Admit a resource operation while the scope remains open. */
|
||||
async run<T>(operation: () => Promise<T>): Promise<T> {
|
||||
while (true) {
|
||||
if (this.#closed)
|
||||
throw new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
"Artifact resource scope is closed",
|
||||
);
|
||||
if (this.#draining !== undefined) await this.#draining;
|
||||
if (this.#closed)
|
||||
throw new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
"Artifact resource scope is closed",
|
||||
);
|
||||
if (this.#pending.size > 0) {
|
||||
await this.#retryCleanup();
|
||||
continue;
|
||||
}
|
||||
const running = Promise.resolve().then(operation);
|
||||
this.#active.add(running);
|
||||
try {
|
||||
return await running;
|
||||
} finally {
|
||||
this.#active.delete(running);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Attempt cleanup now and retain the same owner if it fails. */
|
||||
async release(owner: ArtifactResourceOwner): Promise<ArtifactCleanupAttempt> {
|
||||
try {
|
||||
await cleanup(owner);
|
||||
this.#pending.delete(owner);
|
||||
return { kind: "released" };
|
||||
} catch (cause: unknown) {
|
||||
this.#pending.add(owner);
|
||||
return { kind: "failed", cause };
|
||||
}
|
||||
}
|
||||
|
||||
/** Retry retained owners without closing resource admission. */
|
||||
#retryCleanup(): Promise<void> {
|
||||
if (this.#draining !== undefined) return this.#draining;
|
||||
this.#draining = this.#drainAndRetry().finally(() => {
|
||||
this.#draining = undefined;
|
||||
});
|
||||
return this.#draining;
|
||||
}
|
||||
|
||||
/** Stop admission, wait for active operations, and retry every retained owner. */
|
||||
close(): Promise<void> {
|
||||
this.#closed = true;
|
||||
return this.#retryCleanup();
|
||||
}
|
||||
|
||||
async #drainAndRetry(): Promise<void> {
|
||||
while (this.#active.size > 0) await Promise.allSettled(this.#active);
|
||||
const failures: {
|
||||
readonly owner: ArtifactResourceOwner;
|
||||
readonly cause: unknown;
|
||||
}[] = [];
|
||||
for (const owner of this.#pending) {
|
||||
try {
|
||||
await cleanup(owner);
|
||||
this.#pending.delete(owner);
|
||||
} catch (cause: unknown) {
|
||||
failures.push({ owner, cause });
|
||||
}
|
||||
}
|
||||
if (failures.length === 0) return;
|
||||
const observations = failures.map(({ owner, cause }) =>
|
||||
ArtifactReaderFailure.cleanupObservation(cause, owner.resource),
|
||||
);
|
||||
const resources = [
|
||||
...new Set(observations.flatMap(({ resources }) => resources)),
|
||||
];
|
||||
throw new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`Artifact cleanup remains incomplete: ${resources.join(", ")}`,
|
||||
{
|
||||
cause: new AggregateError(
|
||||
failures.map(({ cause }) => cause),
|
||||
"One or more artifact resources remain owned",
|
||||
),
|
||||
cleanup: {
|
||||
reason: observations.map(({ reason }) => reason).join("; "),
|
||||
resources,
|
||||
},
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const cleanup = async (owner: ArtifactResourceOwner): Promise<void> => {
|
||||
switch (owner.kind) {
|
||||
case "javascript-source-analysis":
|
||||
await owner.analysis.close();
|
||||
return;
|
||||
case "reader":
|
||||
await owner.reader.close();
|
||||
return;
|
||||
case "file-handle":
|
||||
case "directory-handle":
|
||||
await owner.handle.close();
|
||||
return;
|
||||
case "output-tree":
|
||||
await owner.tree.rollback();
|
||||
}
|
||||
};
|
||||
@@ -19,19 +19,15 @@ import { isAbsolute, join, relative, sep } from "node:path";
|
||||
import { Readable, Transform } from "node:stream";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
|
||||
import { listPackage, statFile, uncache } from "@electron/asar";
|
||||
import { getRawHeader, listPackage, statFile, uncache } from "@electron/asar";
|
||||
|
||||
import {
|
||||
ArtifactReaderFailure,
|
||||
copyArtifactEntry,
|
||||
sameArtifactEntry,
|
||||
type ArtifactEntry,
|
||||
type ArtifactReader,
|
||||
} from "./ArtifactReader.js";
|
||||
import { closeAsarHandle, readValidatedAsarEntry } from "./AsarEntryStream.js";
|
||||
import { admitAsarHeader } from "./AsarHeader.js";
|
||||
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
|
||||
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
|
||||
|
||||
/**
|
||||
* Official Electron ASAR adapter with range-streamed member reads.
|
||||
@@ -45,13 +41,6 @@ export class AsarArtifactReader implements ArtifactReader {
|
||||
#archiveSize: number | undefined;
|
||||
#headerSize: number | undefined;
|
||||
readonly #entries = new Map<string, AsarEntryState>();
|
||||
readonly #unpackedHandles = new Map<
|
||||
UnpackedFileHandle,
|
||||
{
|
||||
readonly path: string;
|
||||
readonly owner: OwnedFileHandle<UnpackedFileHandle>;
|
||||
}
|
||||
>();
|
||||
#snapshot: { readonly path: string; readonly sha256: string } | undefined;
|
||||
#snapshotRoot: string | undefined;
|
||||
|
||||
@@ -76,7 +65,6 @@ export class AsarArtifactReader implements ArtifactReader {
|
||||
try {
|
||||
const admitted = await admitAsarHeader(headerHandle);
|
||||
this.#archiveSize = admitted.archiveBytes;
|
||||
this.#headerSize = admitted.headerBytes;
|
||||
} finally {
|
||||
await headerHandle.close();
|
||||
}
|
||||
@@ -84,6 +72,7 @@ export class AsarArtifactReader implements ArtifactReader {
|
||||
paths = listPackage(snapshotPath, { isPack: false }).sort((left, right) =>
|
||||
left.localeCompare(right, "en"),
|
||||
);
|
||||
this.#headerSize = getRawHeader(snapshotPath).headerSize;
|
||||
const archiveMetadata = await lstat(snapshotPath);
|
||||
if (!archiveMetadata.isFile() || archiveMetadata.isSymbolicLink())
|
||||
throw new ArtifactReaderFailure(
|
||||
@@ -167,8 +156,32 @@ export class AsarArtifactReader implements ArtifactReader {
|
||||
);
|
||||
const { metadata } = state;
|
||||
const producedEntry = state.entry;
|
||||
if (producedEntry.unpacked)
|
||||
return this.#openUnpackedStream(producedEntry, signal);
|
||||
if (producedEntry.unpacked) {
|
||||
const handle = await this.#openUnpackedEntry(producedEntry);
|
||||
try {
|
||||
const observed = await handle.stat();
|
||||
abortIfNeeded(signal);
|
||||
if (!observed.isFile())
|
||||
throw new ArtifactReaderFailure(
|
||||
"path",
|
||||
`ASAR unpacked entry is not a regular file: ${producedEntry.path}`,
|
||||
);
|
||||
} catch (cause: unknown) {
|
||||
await handle.close().catch(() => undefined);
|
||||
throw asarFailure(this.path, `read ${producedEntry.path}`, cause);
|
||||
}
|
||||
const source = handle.createReadStream({
|
||||
start: 0,
|
||||
autoClose: true,
|
||||
});
|
||||
return readValidatedAsarEntry(
|
||||
source,
|
||||
undefined,
|
||||
producedEntry.path,
|
||||
this.path,
|
||||
signal,
|
||||
);
|
||||
}
|
||||
const archiveSize = this.#archiveSize;
|
||||
const headerSize = this.#headerSize;
|
||||
const offset =
|
||||
@@ -302,39 +315,22 @@ export class AsarArtifactReader implements ArtifactReader {
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
let cleanupFailure: ArtifactReaderFailure | undefined;
|
||||
for (const [handle, { path }] of this.#unpackedHandles) {
|
||||
try {
|
||||
await this.#closeUnpackedHandle(handle);
|
||||
} catch (cause: unknown) {
|
||||
cleanupFailure = ArtifactReaderFailure.withCleanup(
|
||||
cleanupFailure ?? cause,
|
||||
ArtifactReaderFailure.cleanupObservation(cause, path),
|
||||
);
|
||||
}
|
||||
}
|
||||
const root = this.#snapshotRoot;
|
||||
if (root !== undefined) {
|
||||
try {
|
||||
await this.removeSnapshot(root);
|
||||
if (this.#snapshotRoot === root) this.#snapshotRoot = undefined;
|
||||
} catch (cause: unknown) {
|
||||
cleanupFailure = ArtifactReaderFailure.withCleanup(
|
||||
cleanupFailure ?? cause,
|
||||
{
|
||||
reason: `Could not remove ASAR snapshot: ${root}`,
|
||||
resources: [root],
|
||||
},
|
||||
);
|
||||
throw ArtifactReaderFailure.withCleanup(cause, {
|
||||
reason: `Could not remove ASAR snapshot: ${root}`,
|
||||
resources: [root],
|
||||
});
|
||||
}
|
||||
this.#snapshotRoot = undefined;
|
||||
}
|
||||
if (root === undefined || this.#snapshotRoot === undefined) {
|
||||
if (this.#snapshot !== undefined) uncache(this.#snapshot.path);
|
||||
this.#snapshot = undefined;
|
||||
this.#resetArchiveState();
|
||||
this.#entries.clear();
|
||||
}
|
||||
if (cleanupFailure !== undefined) throw cleanupFailure;
|
||||
if (this.#snapshot !== undefined) uncache(this.#snapshot.path);
|
||||
this.#snapshot = undefined;
|
||||
this.#resetArchiveState();
|
||||
this.#entries.clear();
|
||||
}
|
||||
|
||||
#snapshotPath(): string {
|
||||
@@ -350,12 +346,8 @@ export class AsarArtifactReader implements ArtifactReader {
|
||||
return [];
|
||||
}
|
||||
|
||||
async #openUnpackedStream(
|
||||
entry: ArtifactEntry,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Readable> {
|
||||
async #openUnpackedEntry(entry: ArtifactEntry): Promise<UnpackedFileHandle> {
|
||||
const unpackedRoot = `${this.path}.unpacked`;
|
||||
let handle: UnpackedFileHandle | undefined;
|
||||
try {
|
||||
const rootMetadata = await lstat(unpackedRoot);
|
||||
if (!rootMetadata.isDirectory() || rootMetadata.isSymbolicLink())
|
||||
@@ -382,21 +374,22 @@ export class AsarArtifactReader implements ArtifactReader {
|
||||
"path",
|
||||
`ASAR unpacked entry is not a regular file: ${entry.path}`,
|
||||
);
|
||||
handle = await this.openUnpackedFile(
|
||||
const handle = await this.openUnpackedFile(
|
||||
canonical,
|
||||
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
|
||||
);
|
||||
const openedHandle = handle;
|
||||
this.#unpackedHandles.set(openedHandle, {
|
||||
path: canonical,
|
||||
owner: new OwnedFileHandle(openedHandle),
|
||||
});
|
||||
const openedMetadata = await openedHandle.stat();
|
||||
const openedMetadata = await handle
|
||||
.stat()
|
||||
.catch(async (cause: unknown) => {
|
||||
await handle.close().catch(() => undefined);
|
||||
throw cause;
|
||||
});
|
||||
if (
|
||||
!openedMetadata.isFile() ||
|
||||
openedMetadata.dev !== pathMetadata.dev ||
|
||||
openedMetadata.ino !== pathMetadata.ino
|
||||
) {
|
||||
await handle.close();
|
||||
throw new ArtifactReaderFailure(
|
||||
"integrity",
|
||||
`ASAR unpacked entry changed before read: ${entry.path}`,
|
||||
@@ -409,46 +402,25 @@ export class AsarArtifactReader implements ArtifactReader {
|
||||
},
|
||||
);
|
||||
}
|
||||
abortIfNeeded(signal);
|
||||
const source = readFileHandleChunks(openedHandle, {
|
||||
start: 0,
|
||||
});
|
||||
source.once("close", () => {
|
||||
void this.#closeUnpackedHandle(openedHandle).catch(() => undefined);
|
||||
});
|
||||
return readValidatedAsarEntry(
|
||||
source,
|
||||
undefined,
|
||||
entry.path,
|
||||
this.path,
|
||||
signal,
|
||||
);
|
||||
return handle;
|
||||
} catch (cause: unknown) {
|
||||
let failure = asarUnpackedEntryFailure(this.path, entry, cause);
|
||||
if (handle !== undefined && this.#unpackedHandles.has(handle)) {
|
||||
try {
|
||||
await this.#closeUnpackedHandle(handle);
|
||||
} catch (cleanupCause: unknown) {
|
||||
failure = ArtifactReaderFailure.withCleanup(
|
||||
failure,
|
||||
ArtifactReaderFailure.cleanupObservation(
|
||||
cleanupCause,
|
||||
this.#unpackedHandles.get(handle)?.path ?? entry.path,
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
throw failure;
|
||||
if (cause instanceof ArtifactReaderFailure) throw cause;
|
||||
if (entry.unpacked && isMissingFile(cause))
|
||||
throw new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`ASAR unpacked entry bytes are unavailable: ${entry.path}`,
|
||||
{ cause },
|
||||
{
|
||||
logicalPath: entry.path,
|
||||
declaredSha256: entry.declaredSha256,
|
||||
calculatedSha256: null,
|
||||
unpacked: true,
|
||||
},
|
||||
);
|
||||
throw asarFailure(this.path, `read ${entry.path}`, cause);
|
||||
}
|
||||
}
|
||||
|
||||
async #closeUnpackedHandle(handle: UnpackedFileHandle): Promise<void> {
|
||||
const owned = this.#unpackedHandles.get(handle);
|
||||
if (owned === undefined) return;
|
||||
await owned.owner.close();
|
||||
this.#unpackedHandles.delete(handle);
|
||||
}
|
||||
|
||||
#resetArchiveState(): void {
|
||||
this.#archiveSize = undefined;
|
||||
this.#headerSize = undefined;
|
||||
@@ -461,36 +433,43 @@ type AsarEntryState = {
|
||||
readonly entry: ArtifactEntry;
|
||||
};
|
||||
type UnpackedFileHandle = {
|
||||
readonly fd: number;
|
||||
stat(): Promise<Stats>;
|
||||
close(): Promise<void>;
|
||||
read: FileHandle["read"];
|
||||
createReadStream: FileHandle["createReadStream"];
|
||||
};
|
||||
type OpenAsarUnpackedFile = (
|
||||
path: string,
|
||||
flags: number,
|
||||
) => Promise<UnpackedFileHandle>;
|
||||
|
||||
const asarUnpackedEntryFailure = (
|
||||
archivePath: string,
|
||||
const copyArtifactEntry = (entry: ArtifactEntry): ArtifactEntry => ({
|
||||
...entry,
|
||||
limitations: [...entry.limitations],
|
||||
...(entry.sourceIdentity === undefined
|
||||
? {}
|
||||
: { sourceIdentity: { ...entry.sourceIdentity } }),
|
||||
});
|
||||
|
||||
const sameArtifactEntry = (
|
||||
entry: ArtifactEntry,
|
||||
cause: unknown,
|
||||
): ArtifactReaderFailure => {
|
||||
if (cause instanceof ArtifactReaderFailure) return cause;
|
||||
if (entry.unpacked && isMissingFile(cause))
|
||||
return new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`ASAR unpacked entry bytes are unavailable: ${entry.path}`,
|
||||
{ cause },
|
||||
{
|
||||
logicalPath: entry.path,
|
||||
declaredSha256: entry.declaredSha256,
|
||||
calculatedSha256: null,
|
||||
unpacked: true,
|
||||
},
|
||||
);
|
||||
return asarFailure(archivePath, `read ${entry.path}`, cause);
|
||||
};
|
||||
produced: ArtifactEntry,
|
||||
): boolean =>
|
||||
entry.path === produced.path &&
|
||||
entry.kind === produced.kind &&
|
||||
entry.declaredSize === produced.declaredSize &&
|
||||
entry.compressedSize === produced.compressedSize &&
|
||||
entry.executable === produced.executable &&
|
||||
entry.encrypted === produced.encrypted &&
|
||||
entry.byteOffset === produced.byteOffset &&
|
||||
entry.declaredSha256 === produced.declaredSha256 &&
|
||||
entry.unpacked === produced.unpacked &&
|
||||
entry.adapterKey === produced.adapterKey &&
|
||||
entry.limitations.length === produced.limitations.length &&
|
||||
entry.limitations.every(
|
||||
(limitation, index) => limitation === produced.limitations[index],
|
||||
) &&
|
||||
entry.sourceIdentity?.device === produced.sourceIdentity?.device &&
|
||||
entry.sourceIdentity?.inode === produced.sourceIdentity?.inode;
|
||||
|
||||
const isAsarFileMetadata = (
|
||||
metadata: ReturnType<typeof statFile>,
|
||||
|
||||
@@ -1,33 +1,23 @@
|
||||
import { constants, type Stats } from "node:fs";
|
||||
import {
|
||||
lstat,
|
||||
open,
|
||||
opendir,
|
||||
realpath,
|
||||
type FileHandle,
|
||||
} from "node:fs/promises";
|
||||
import { constants } from "node:fs";
|
||||
import { lstat, open, opendir, realpath } from "node:fs/promises";
|
||||
import { isAbsolute, join, relative, sep } from "node:path";
|
||||
import { Readable } from "node:stream";
|
||||
import type { Readable } from "node:stream";
|
||||
|
||||
import {
|
||||
ArtifactReaderFailure,
|
||||
type ArtifactEntry,
|
||||
type ArtifactReader,
|
||||
} from "./ArtifactReader.js";
|
||||
import { OwnedDirectoryHandle } from "../filesystem/OwnedDirectoryHandle.js";
|
||||
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
|
||||
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
|
||||
|
||||
/** Non-symlink-following directory reader. */
|
||||
export class DirectoryArtifactReader implements ArtifactReader {
|
||||
readonly format = "directory" as const;
|
||||
#rootPromise: Promise<string> | undefined;
|
||||
readonly #rootPromise: Promise<string>;
|
||||
readonly #rootPath: string;
|
||||
readonly #directories = new Set<OwnedDirectoryHandle>();
|
||||
readonly #files = new Map<OwnedFileHandle, string>();
|
||||
|
||||
constructor(root: string) {
|
||||
this.#rootPath = root;
|
||||
this.#rootPromise = realpath(root);
|
||||
}
|
||||
|
||||
/** Enumerate entries, optionally restricting descent while retaining directories. */
|
||||
@@ -35,7 +25,9 @@ export class DirectoryArtifactReader implements ArtifactReader {
|
||||
signal?: AbortSignal,
|
||||
descendInto?: (path: string) => boolean,
|
||||
): AsyncIterable<ArtifactEntry> {
|
||||
const root = await this.#resolveRoot();
|
||||
const root = await this.#rootPromise.catch((cause: unknown) => {
|
||||
throw directoryIoFailure("resolve artifact root", this.#rootPath, cause);
|
||||
});
|
||||
const pending = [root];
|
||||
while (pending.length > 0) {
|
||||
abortIfNeeded(signal);
|
||||
@@ -44,100 +36,62 @@ export class DirectoryArtifactReader implements ArtifactReader {
|
||||
await withDirectoryIoContext("inspect directory", directory, () =>
|
||||
assertContainedDirectory(root, directory),
|
||||
);
|
||||
for await (const entry of this.#readDirectory(
|
||||
directory,
|
||||
pending,
|
||||
signal,
|
||||
descendInto,
|
||||
))
|
||||
yield entry;
|
||||
const handle = await opendir(directory).catch((cause: unknown) => {
|
||||
throw directoryIoFailure("open directory", directory, cause);
|
||||
});
|
||||
const directories: string[] = [];
|
||||
try {
|
||||
for await (const child of handle) {
|
||||
abortIfNeeded(signal);
|
||||
const absolute = join(directory, child.name);
|
||||
const metadata = await lstat(absolute).catch((cause: unknown) => {
|
||||
throw directoryIoFailure("inspect entry", absolute, cause);
|
||||
});
|
||||
const path = relative(root, absolute).split(sep).join("/");
|
||||
const kind = metadata.isSymbolicLink()
|
||||
? "symlink"
|
||||
: metadata.isDirectory()
|
||||
? "directory"
|
||||
: metadata.isFile()
|
||||
? "file"
|
||||
: undefined;
|
||||
if (kind === undefined) continue;
|
||||
yield {
|
||||
path,
|
||||
kind,
|
||||
declaredSize: kind === "file" ? metadata.size : null,
|
||||
compressedSize: null,
|
||||
executable: (metadata.mode & 0o111) !== 0,
|
||||
encrypted: false,
|
||||
byteOffset: null,
|
||||
declaredSha256: null,
|
||||
unpacked: false,
|
||||
limitations:
|
||||
kind === "symlink"
|
||||
? ["Symlink target was not followed or disclosed."]
|
||||
: [],
|
||||
adapterKey: absolute,
|
||||
...(kind === "file"
|
||||
? {
|
||||
sourceIdentity: { device: metadata.dev, inode: metadata.ino },
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
if (kind === "directory" && (descendInto?.(path) ?? true))
|
||||
directories.push(absolute);
|
||||
}
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof ArtifactReaderFailure) throw cause;
|
||||
throw directoryIoFailure("read directory", directory, cause);
|
||||
} finally {
|
||||
// best-effort cleanup: directory-handle close must not mask traversal.
|
||||
await handle.close().catch(() => undefined);
|
||||
}
|
||||
await withDirectoryIoContext("inspect directory", directory, () =>
|
||||
assertContainedDirectory(root, directory),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async #resolveRoot(): Promise<string> {
|
||||
return (this.#rootPromise ??= realpath(this.#rootPath)).catch(
|
||||
(cause: unknown) => {
|
||||
throw directoryIoFailure(
|
||||
"resolve artifact root",
|
||||
this.#rootPath,
|
||||
cause,
|
||||
);
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
async *#readDirectory(
|
||||
directory: string,
|
||||
pending: string[],
|
||||
signal?: AbortSignal,
|
||||
descendInto?: (path: string) => boolean,
|
||||
): AsyncIterable<ArtifactEntry> {
|
||||
const root = await this.#resolveRoot();
|
||||
const handle = new OwnedDirectoryHandle(
|
||||
await opendir(directory).catch((cause: unknown) => {
|
||||
throw directoryIoFailure("open directory", directory, cause);
|
||||
}),
|
||||
);
|
||||
this.#directories.add(handle);
|
||||
const childDirectories: string[] = [];
|
||||
let traversal:
|
||||
| { readonly kind: "complete" }
|
||||
| { readonly kind: "failed"; readonly cause: unknown } = {
|
||||
kind: "complete",
|
||||
};
|
||||
try {
|
||||
for (;;) {
|
||||
const child = await handle.handle.read();
|
||||
if (child === null) break;
|
||||
abortIfNeeded(signal);
|
||||
const absolute = join(directory, child.name);
|
||||
const metadata = await lstat(absolute).catch((cause: unknown) => {
|
||||
throw directoryIoFailure("inspect entry", absolute, cause);
|
||||
});
|
||||
const path = relative(root, absolute).split(sep).join("/");
|
||||
const entry = artifactEntry(path, absolute, metadata);
|
||||
if (entry === undefined) continue;
|
||||
yield entry;
|
||||
if (entry.kind === "directory" && (descendInto?.(path) ?? true))
|
||||
childDirectories.push(absolute);
|
||||
}
|
||||
} catch (cause: unknown) {
|
||||
traversal = {
|
||||
kind: "failed",
|
||||
cause:
|
||||
cause instanceof ArtifactReaderFailure
|
||||
? cause
|
||||
: directoryIoFailure("read directory", directory, cause),
|
||||
};
|
||||
} finally {
|
||||
await this.#finishDirectory(handle, directory, traversal);
|
||||
}
|
||||
if (traversal.kind === "failed") throw traversal.cause;
|
||||
childDirectories.reverse();
|
||||
pending.push(...childDirectories);
|
||||
}
|
||||
|
||||
async #finishDirectory(
|
||||
handle: OwnedDirectoryHandle,
|
||||
path: string,
|
||||
traversal:
|
||||
| { readonly kind: "complete" }
|
||||
| { readonly kind: "failed"; readonly cause: unknown },
|
||||
): Promise<void> {
|
||||
try {
|
||||
await handle.close();
|
||||
this.#directories.delete(handle);
|
||||
} catch (cause: unknown) {
|
||||
const closeFailure = directoryIoFailure("close directory", path, cause);
|
||||
if (traversal.kind === "failed")
|
||||
throw ArtifactReaderFailure.withCleanup(
|
||||
traversal.cause,
|
||||
ArtifactReaderFailure.cleanupObservation(closeFailure, path),
|
||||
);
|
||||
throw closeFailure;
|
||||
directories.reverse();
|
||||
pending.push(...directories);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -148,16 +102,14 @@ export class DirectoryArtifactReader implements ArtifactReader {
|
||||
"format",
|
||||
"Only regular files can be opened",
|
||||
);
|
||||
const fileHandle = await open(
|
||||
const handle = await open(
|
||||
entry.adapterKey,
|
||||
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
|
||||
).catch((cause: unknown) => {
|
||||
throw directoryIoFailure("open entry", entry.adapterKey, cause);
|
||||
});
|
||||
const owner = new OwnedFileHandle(fileHandle);
|
||||
this.#files.set(owner, entry.adapterKey);
|
||||
try {
|
||||
const observed = await owner.handle.stat().catch((cause: unknown) => {
|
||||
const observed = await handle.stat().catch((cause: unknown) => {
|
||||
throw directoryIoFailure(
|
||||
"inspect opened entry",
|
||||
entry.adapterKey,
|
||||
@@ -175,76 +127,16 @@ export class DirectoryArtifactReader implements ArtifactReader {
|
||||
`Directory entry changed before open: ${entry.path}`,
|
||||
);
|
||||
}
|
||||
abortIfNeeded(signal);
|
||||
const source = Readable.from(
|
||||
mapFileReadCancellation(owner.handle, signal),
|
||||
{ objectMode: false },
|
||||
);
|
||||
source.once("close", () => {
|
||||
void this.#closeFile(owner).catch(() => undefined);
|
||||
});
|
||||
return source;
|
||||
return handle.createReadStream({ autoClose: true });
|
||||
} catch (cause: unknown) {
|
||||
try {
|
||||
await this.#closeFile(owner);
|
||||
} catch (closeCause: unknown) {
|
||||
throw ArtifactReaderFailure.withCleanup(
|
||||
cause,
|
||||
ArtifactReaderFailure.cleanupObservation(
|
||||
directoryIoFailure(
|
||||
"close opened entry",
|
||||
entry.adapterKey,
|
||||
closeCause,
|
||||
),
|
||||
entry.adapterKey,
|
||||
),
|
||||
);
|
||||
}
|
||||
// best-effort cleanup: file-handle close must not mask the open failure.
|
||||
await handle.close().catch(() => undefined);
|
||||
throw cause;
|
||||
}
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
const failures: unknown[] = [];
|
||||
for (const [owner, path] of this.#files) {
|
||||
try {
|
||||
await this.#closeFile(owner);
|
||||
} catch (cause: unknown) {
|
||||
failures.push(directoryIoFailure("close opened entry", path, cause));
|
||||
}
|
||||
}
|
||||
for (const owner of this.#directories) {
|
||||
try {
|
||||
await owner.close();
|
||||
this.#directories.delete(owner);
|
||||
} catch (cause: unknown) {
|
||||
failures.push(
|
||||
directoryIoFailure("close directory", owner.handle.path, cause),
|
||||
);
|
||||
}
|
||||
}
|
||||
if (failures.length > 0) {
|
||||
const reason = failures.map(failureMessage).join("; ");
|
||||
throw new ArtifactReaderFailure(
|
||||
"io",
|
||||
`Directory artifact cleanup failed: ${reason}`,
|
||||
{
|
||||
cause: new AggregateError(failures),
|
||||
cleanup: {
|
||||
reason,
|
||||
resources: [
|
||||
...this.#files.values(),
|
||||
...[...this.#directories].map((owner) => owner.handle.path),
|
||||
],
|
||||
},
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async #closeFile(owner: OwnedFileHandle<FileHandle>): Promise<void> {
|
||||
await owner.close();
|
||||
this.#files.delete(owner);
|
||||
close(): Promise<void> {
|
||||
return Promise.resolve();
|
||||
}
|
||||
|
||||
provenance(): readonly [] {
|
||||
@@ -252,42 +144,6 @@ export class DirectoryArtifactReader implements ArtifactReader {
|
||||
}
|
||||
}
|
||||
|
||||
const artifactEntry = (
|
||||
path: string,
|
||||
absolute: string,
|
||||
metadata: Stats,
|
||||
): ArtifactEntry | undefined => {
|
||||
const kind = metadata.isSymbolicLink()
|
||||
? "symlink"
|
||||
: metadata.isDirectory()
|
||||
? "directory"
|
||||
: metadata.isFile()
|
||||
? "file"
|
||||
: undefined;
|
||||
if (kind === undefined) return undefined;
|
||||
return {
|
||||
path,
|
||||
kind,
|
||||
declaredSize: kind === "file" ? metadata.size : null,
|
||||
compressedSize: null,
|
||||
executable: (metadata.mode & 0o111) !== 0,
|
||||
encrypted: false,
|
||||
byteOffset: null,
|
||||
declaredSha256: null,
|
||||
unpacked: false,
|
||||
limitations:
|
||||
kind === "symlink"
|
||||
? ["Symlink target was not followed or disclosed."]
|
||||
: [],
|
||||
adapterKey: absolute,
|
||||
...(kind === "file"
|
||||
? {
|
||||
sourceIdentity: { device: metadata.dev, inode: metadata.ino },
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
};
|
||||
|
||||
const directoryIoFailure = (
|
||||
operation: string,
|
||||
path: string,
|
||||
@@ -301,38 +157,13 @@ const directoryIoFailure = (
|
||||
typeof cause.code === "string"
|
||||
? ` (${cause.code})`
|
||||
: "";
|
||||
const detail = failureMessage(cause);
|
||||
return new ArtifactReaderFailure(
|
||||
"io",
|
||||
`Could not ${operation} at ${path}${code}: ${detail}`,
|
||||
`Could not ${operation} at ${path}${code}`,
|
||||
{ cause },
|
||||
);
|
||||
};
|
||||
|
||||
const failureMessage = (cause: unknown): string =>
|
||||
cause instanceof Error ? cause.message : String(cause);
|
||||
|
||||
const mapFileReadCancellation = async function* (
|
||||
handle: FileHandle,
|
||||
signal?: AbortSignal,
|
||||
): AsyncGenerator<Buffer> {
|
||||
try {
|
||||
for await (const chunk of readFileHandleChunks(handle, { signal })) {
|
||||
abortIfNeeded(signal);
|
||||
yield chunk;
|
||||
}
|
||||
abortIfNeeded(signal);
|
||||
} catch (cause: unknown) {
|
||||
if (signal?.aborted === true)
|
||||
throw new ArtifactReaderFailure(
|
||||
"cancelled",
|
||||
"Directory artifact read cancelled",
|
||||
{ cause },
|
||||
);
|
||||
throw cause;
|
||||
}
|
||||
};
|
||||
|
||||
const withDirectoryIoContext = async <T>(
|
||||
operation: string,
|
||||
path: string,
|
||||
|
||||
@@ -1,84 +1,41 @@
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import { describe, expect, it, onTestFinished } from "vitest";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
|
||||
import { CPU, fatImage, machoImage } from "./apple/MachoImage.fixture.js";
|
||||
import { MachOSliceArtifactReader } from "./MachOSliceArtifactReader.js";
|
||||
import { ok } from "../domain/result.js";
|
||||
import { openRegularFile } from "../filesystem/RegularFile.js";
|
||||
import type { NativeCommandRunner } from "../native/CommandRunner.js";
|
||||
|
||||
describe("MachOSliceArtifactReader", () => {
|
||||
it("keeps a borrowed source usable after a cancelled slice read", async () => {
|
||||
const root = await createTestTempDirectory("rea-slices-cancelled-");
|
||||
const binary = join(root, "fat");
|
||||
const x86 = machoImage({ cpu: CPU.x86_64 });
|
||||
const arm = Buffer.concat([
|
||||
Buffer.from(machoImage({ cpu: CPU.arm64 })),
|
||||
Buffer.alloc(160_000, 0x5a),
|
||||
]);
|
||||
await writeFile(
|
||||
binary,
|
||||
fatImage([
|
||||
{ cpu: CPU.x86_64, bytes: x86 },
|
||||
{ cpu: CPU.arm64, bytes: arm },
|
||||
]),
|
||||
);
|
||||
const handle = await openRegularFile(binary, { symlinks: "reject" });
|
||||
const source = { handle, initial: await handle.stat() };
|
||||
const runner: NativeCommandRunner = {
|
||||
run: () =>
|
||||
Promise.resolve(
|
||||
ok({
|
||||
tool: "lipo",
|
||||
executable: "/usr/bin/lipo",
|
||||
executableSha256: "1".repeat(64),
|
||||
toolVersion: null,
|
||||
versionReason: "fixture",
|
||||
arguments: ["-detailed_info", binary],
|
||||
stdout: `architecture x86_64\n cputype 16777223\n cpusubtype 3\n offset 4096\n size ${String(x86.length)}\n align 2^12 (4096)\narchitecture arm64\n cputype 16777228\n cpusubtype 0\n offset 8192\n size ${String(arm.length)}\n align 2^12 (4096)\n`,
|
||||
stderr: "",
|
||||
stdoutBytes: 1,
|
||||
stderrBytes: 0,
|
||||
exitCode: 0,
|
||||
signal: null,
|
||||
}),
|
||||
),
|
||||
};
|
||||
const reader = new MachOSliceArtifactReader(binary, {}, runner, source);
|
||||
onTestFinished(async () => {
|
||||
await reader.close();
|
||||
await handle.close();
|
||||
});
|
||||
|
||||
const entries = [];
|
||||
for await (const entry of reader.entries()) entries.push(entry);
|
||||
const armEntry = entries.find((entry) => entry.path === "slices/arm64");
|
||||
if (armEntry === undefined) throw new Error("Expected an arm64 slice");
|
||||
|
||||
const controller = new AbortController();
|
||||
const cancelledRead = await reader.open(armEntry, controller.signal);
|
||||
const iterator = cancelledRead[Symbol.asyncIterator]();
|
||||
const firstChunk = await iterator.next();
|
||||
expect(firstChunk.value).toHaveLength(64 * 1024);
|
||||
controller.abort();
|
||||
await expect(iterator.next()).rejects.toMatchObject({
|
||||
reason: "cancelled",
|
||||
});
|
||||
|
||||
const verifiedEntries = [];
|
||||
for await (const entry of reader.entries()) verifiedEntries.push(entry);
|
||||
const verifiedArmEntry = verifiedEntries.find(
|
||||
(entry) => entry.path === "slices/arm64",
|
||||
);
|
||||
if (verifiedArmEntry === undefined)
|
||||
throw new Error("Expected arm64 after re-verifying the borrowed source");
|
||||
const verifiedChunks: Buffer[] = [];
|
||||
const verifiedRead = await reader.open(verifiedArmEntry);
|
||||
for await (const chunk of verifiedRead)
|
||||
verifiedChunks.push(Buffer.from(chunk));
|
||||
expect(Buffer.concat(verifiedChunks)).toEqual(arm);
|
||||
/**
|
||||
* Read a 32-bit FAT table directly. The host's /bin/ls changes between macOS
|
||||
* releases, so the expected slice layout comes from the file itself.
|
||||
*/
|
||||
const fatSlices = async (
|
||||
path: string,
|
||||
): Promise<readonly (readonly [number, number])[]> => {
|
||||
const bytes = await readFile(path);
|
||||
expect(bytes.readUInt32BE(0)).toBe(0xca_fe_ba_be);
|
||||
return Array.from({ length: bytes.readUInt32BE(4) }, (_, index) => {
|
||||
const entry = 8 + index * 20;
|
||||
return [bytes.readUInt32BE(entry + 8), bytes.readUInt32BE(entry + 12)];
|
||||
});
|
||||
};
|
||||
|
||||
describe("Mach-O slice reader with system lipo", () => {
|
||||
it.skipIf(process.platform !== "darwin")(
|
||||
"checks real /bin/ls lipo CPU symbols and ptrauth capabilities against the FAT table",
|
||||
async () => {
|
||||
const reader = new MachOSliceArtifactReader("/bin/ls", {});
|
||||
const entries = [];
|
||||
for await (const entry of reader.entries()) entries.push(entry);
|
||||
expect(entries.map(({ path }) => path)).toEqual([
|
||||
"slices/x86_64",
|
||||
"slices/arm64e",
|
||||
]);
|
||||
expect(
|
||||
entries.map(({ byteOffset, declaredSize }) => [
|
||||
byteOffset,
|
||||
declaredSize,
|
||||
]),
|
||||
).toEqual(await fatSlices("/bin/ls"));
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user