Compare commits

..
402 changed files with 4050 additions and 22200 deletions
@@ -2,7 +2,7 @@
name: reverse-engineer-anything name: reverse-engineer-anything
description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.
metadata: metadata:
version: "37" version: "36"
--- ---
# REA # REA
@@ -25,18 +25,57 @@ a skill installed from repository main may describe capabilities absent from
an older npm release. Keep complete inline Evidence when that server does not an older npm release. Keep complete inline Evidence when that server does not
advertise retained references. advertise retained references.
If tools are absent, a call is blocked by the client, or a required provider is When tools are absent or registration is stale:
unavailable, read [connection and recovery](references/connection-and-recovery.md).
Diagnose the observed blocker, continue useful supported work, and resume the
original task after repair. Client approval, shell PATH, registration and engine
availability have different remedies; avoid repeating setup for all of them.
Prefer an existing `rea` CLI. When none is installed and REA package acquisition 1. Diagnose without changing files. For Codex, run
is within the authorized scope, use `npx -y rea-agents@latest` in its place. `npx -y rea-agents@latest doctor --client codex --json`. Substitute the current
Configuration repair starts with a read-only setup plan; show paths, backups and supported client: `claude_code`, `claude_desktop`, `codex`, `cursor`,
skill replacement before applying authorized changes. Preserve existing approval `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`, `copilot_cli`,
and target-execution boundaries. REA setup never installs or upgrades unrelated `commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`, `hermes`, or `grok_bot`. `grok_bot` has no local
software, including Node.js, npm, Homebrew, Java, Ghidra, IDA, JADX, Binwalk or Unblob. file registration: doctor reports the Grok Bot chat step, and setup does not
write `mcp.json`. If the client is unknown, use `doctor --json` and
inspect its registration results before choosing a setup scope.
2. Distinguish the reason. Missing, malformed, or stale registration needs a
scoped configuration repair. An aligned registration with no tools in the
active session needs a restart/reconnection; doctor cannot prove that the
current agent has connected. A missing provider affects only tasks requiring
that provider: static JavaScript inspection needs neither Hopper nor Ghidra,
and Android inspection has separate bring-your-own JADX/Java prerequisites.
3. For a configuration repair, prepare the read-only plan:
`npx -y rea-agents@latest setup --client codex --dry-run --json`.
Use the current client's ID, show its exact proposed paths, backups, and
changes, and obtain approval before setup writes configuration or installs
Hopper. Setup normally installs the matching bundled skill too; include that
replacement in the reviewed plan. After approval, apply the same scope with
`npx -y rea-agents@latest setup --client codex --yes`. Add `--install-hopper`
only if that separate installation was needed and explicitly approved.
4. Restart/reconnect the affected agent. Verify that REA tools actually appear
in the session, then resume the original investigation. If they remain
absent, inspect the client's MCP launch error rather than repeating setup.
REA setup never installs or upgrades Node.js, npm, Homebrew, Java, Ghidra, IDA,
JADX, Binwalk, or Unblob. Use existing prerequisites; do not install unrelated
software to repair
MCP registration. For an unsupported client, use manual stdio registration
with a version-pinned `rea-agents` package or continue through the CLI.
A concrete CLI fallback for an operator-supplied JavaScript tree or ASAR is:
```bash
npx -y rea-agents@latest analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
npx -y rea-agents@latest inspect-analysis-view app-view.json
```
No MCP registration or native engine is required. The CLI returns the complete
Evidence record directly, and a real application can produce hundreds of
megabytes, so save it to a file rather than reading it into context. Start from
the summary view, then request a module page
(`{"kind": "page", "collection": "modules", "offset": 0, "limit": 32}`) or one
module the same way, or pass the saved Evidence inline to
`trace-application-feature`. Read graph, limitations, and unknowns with the
same care as an MCP result. This fallback does not establish that MCP is
configured. Native CLI tasks still require their selected engine.
## Route the target first ## Route the target first
@@ -101,11 +140,9 @@ after opening; IDA does not supply `binary_overview`. Consult session availabili
when composing broader workflows. Modern direct callers and unsupported dossier when composing broader workflows. Modern direct callers and unsupported dossier
facets remain unknown; live IDA results are not replayed from snapshots. facets remain unknown; live IDA results are not replayed from snapshots.
If the user says only "my app" without a name, path, or target already selected If the app is missing, ask which app to inspect. Resolve a human-readable app
for this task, ask which app before searching for artifacts. Paths in unrelated name to one clear installed artifact when possible; ask only when matches are
saved reports and nearby test fixtures do not select the user's target. Resolve ambiguous. Never choose an example app on the user's behalf.
an actual human-readable app name to one clear installed artifact when possible;
ask when matches are ambiguous. Never choose an example app on the user's behalf.
In a target-free session, use `open_binary` to bind any archive/package or In a target-free session, use `open_binary` to bind any archive/package or
native target whose analysis tool operates on the active target. Do not call an native target whose analysis tool operates on the active target. Do not call an
@@ -120,11 +157,7 @@ For a JavaScript application or ELF layout whose size is unknown or large,
call `analyze_javascript_application` or `inspect_binary_layout` with call `analyze_javascript_application` or `inspect_binary_layout` with
`"detail": "summary"`, then inspect the retained `parent_evidence_id` with `"detail": "summary"`, then inspect the retained `parent_evidence_id` with
`inspect_analysis_view`. `inspect_analysis_view`.
Page module identities and select the returned `node_id` before requesting an Do not repeat an identical tool call. Make a focused follow-up only when the
item; a single file path can represent several graph roles. Do not repeat an
identical analysis or rejected call without a changed input or resolved blocker.
Recheck session state when a lifecycle change makes that status useful.
Make a focused follow-up only when the
returned result leaves a specific question unanswered. returned result leaves a specific question unanswered.
If MCP reports `resource_constraint` with `details.resource: "transport"`, call If MCP reports `resource_constraint` with `details.resource: "transport"`, call
@@ -135,14 +168,6 @@ session to a selected path. The analysis remains complete in the ledger; avoid
repeating it merely to request the same oversized response. Complete CLI JSON repeating it merely to request the same oversized response. Complete CLI JSON
output also streams without a single MCP frame. output also streams without a single MCP frame.
A client can also truncate a successful tool result without a REA transport
error. Treat that preview as incomplete; use the retained parent ID for smaller
views or export when the full record is needed. Do not parse the preview as
canonical JSON or repeat the producer. For JavaScript, read the
[module-view and trace recipes](references/javascript-applications.md) before
drilling into a large graph; a broad bidirectional trace can expand unrelated
containment relationships even after summary analysis.
Every conclusion must distinguish observations, inferences, and unknowns. Cite Every conclusion must distinguish observations, inferences, and unknowns. Cite
Evidence IDs, preserve limitations and incomplete coverage, and never imply Evidence IDs, preserve limitations and incomplete coverage, and never imply
that static analysis observed execution. Runtime requests execute the declared that static analysis observed execution. Runtime requests execute the declared
@@ -150,9 +175,7 @@ target and lifecycle; do not broaden the target or action beyond those fields.
Within the user's requested investigation, call available analysis tools Within the user's requested investigation, call available analysis tools
directly. REA does not require permission grants or per-call approval flags. directly. REA does not require permission grants or per-call approval flags.
The agent client can independently require approval. Follow the declared request Follow the declared request scope and the host's actual access requirements.
scope and the host's actual access requirements; use the recovery guide for an
observed client-policy rejection.
## Plan broader investigations ## Plan broader investigations
@@ -1,127 +0,0 @@
# Connection and task recovery
Read this guide only for an observed connection, command, approval or provider
blocker. Keep the user's target and question in view. Explain what prevented the
operation and the next useful step; after repair, continue the investigation.
Commands below use an installed `rea`. If none is available, use the package
invocation in `SKILL.md` when acquiring REA is authorized. Follow the connected
server's schemas and the installed CLI's help rather than assuming repository
main and an older release expose the same arguments.
## Choose the remedy from the observed failure
| Observation | Meaning and next step |
| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No REA tools in the active session | Inspect registration with `rea doctor --client codex --json`. A skill installation alone does not register MCP. |
| Missing, malformed or stale registration | Prepare a scoped setup plan; apply only authorized changes with backups. |
| Aligned registration, but no connected tools | Restart/reconnect the client. Doctor checks registration; it cannot establish the current agent's connection. Inspect the client's launch error if reconnection fails. |
| `MCP tool call requires approval, but approval policy is never` | Codex rejected the call before REA executed it. Do not install an engine or repeat setup to solve this client-policy decision. See the approval section below. |
| REA returns `provider_unavailable` or an unavailable tool entry | Read that operation's provider diagnostics or `binary_session.result.tool_availability`. Resolve only the prerequisite needed for the task; a complete tool catalog does not imply every operation is available. |
| `rea` is missing in a child shell, or Node is incompatible | Inspect the actual shell PATH and launcher/runtime. A prefix installation may be absent from a login shell's PATH; this does not establish a broken MCP registration. |
If the client cannot launch its shell/sandbox helper or read the installed skill,
inspect its launch diagnostic. For example, Codex can refuse helper aliases when
CODEX_HOME is under `/tmp`; use an authorized profile outside the OS temporary
directory. REA setup cannot repair the client's executor. Preserve the sandbox
instead of silently disabling it to make an analysis command work.
Static JavaScript and ordinary ZIP inventory need neither Hopper nor Ghidra.
HAR inspection is offline and does not require a live browser. Native
decompilation, Android analysis and firmware extraction have their own
bring-your-own prerequisites. Use supported inventory/static evidence when it
answers part of the question, and name the unavailable evidence separately.
Do not substitute another target. Label observations from ordinary shell tools
as such; only actual REA output supplies REA Evidence.
## Repair registration when needed
For Codex:
```bash
rea doctor --client codex --json
rea setup --client codex --dry-run --json
```
Use the current supported client's ID: `claude_code`, `claude_desktop`, `codex`,
`cursor`, `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`,
`copilot_cli`, `commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`,
`hermes`, or `grok_bot`. If unknown, inspect `doctor --json` before choosing a
scope. `grok_bot` uses a chat registration step rather than local `mcp.json`.
For unsupported clients, use version-pinned manual stdio registration or CLI.
Show the plan's exact paths, backups, changes and bundled-skill replacement.
Obtain approval for writes when that concrete scope is not already authorized,
then apply the same scope with `rea setup --client codex --yes`. Add
`--install-hopper` only when separately needed and explicitly approved. Do not
install or upgrade unrelated prerequisites. Restart/reconnect, verify tools
actually appear, and resume the original task. If they remain absent, inspect
the launch failure rather than looping through setup.
## Codex tool approval is separate from REA readiness
REA tools have no permission-grant handshake. Codex can still require approval
because an operation creates owned resources, launches a helper or has cleanup
effects. An advertised tool or successful doctor result does not override that
client policy. `workspace-write` alone does not resolve an MCP approval
rejection, and `approval_policy = "never"` does not mean every MCP tool is
preapproved.
When interactive approval is available, use the client's approval flow for the
requested operation. In an unattended run, continue through an authorized CLI
workflow if the execution environment permits it, or report the blocked
operation and the approval needed to proceed. Do not retry the same rejected
call, silently broaden approvals, or rewrite MCP effects as read-only.
For an operator who explicitly wants to preapprove named operations in a
disposable Codex profile, current Codex supports per-tool configuration. For
example, opening and closing an analysis session can be scoped independently:
```toml
[mcp_servers.rea.tools.open_binary]
approval_mode = "approve"
[mcp_servers.rea.tools.close_binary]
approval_mode = "approve"
```
Use the actual server ID and only the operations whose effects were approved;
this example does not approve inspection, extraction or runtime execution.
Show the proposed config edit and preserve unrelated entries and backups before
writing. Managed policy can constrain client settings. Consult the
[Codex configuration reference](https://learn.chatgpt.com/docs/config-file/config-reference)
for the installed client's supported settings. This is an optional recovery
choice, not a prerequisite for every REA task.
## Continue through an installed CLI
Check the command environment only if the launcher fails:
```bash
command -v rea
command -v node
node --version
rea --version
```
Use the public launcher at its known installation path with a supported existing
Node on PATH when the child shell loses a prefix installation. Avoid internal
`dist/cli.js` entrypoints or global toolchain changes. MCP may have its own valid
launcher even when a shell cannot resolve `rea`.
For a user-supplied JavaScript tree or ASAR, select output files in the authorized
workspace and save complete Evidence before reading a view:
```bash
rea analyze-javascript-application /absolute/path/to/app --json > app-evidence.json
jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json
rea inspect-analysis-view app-view.json
```
These commands do not register MCP or install an analysis engine. CLI JavaScript
JSON is complete and can be very large; redirect it rather than loading it into
context. Use an existing JSON processor if `jq` is absent, without installing
one just for this recipe. Inspect a module page or item using the same inline
source and the [JavaScript recipes](javascript-applications.md). Preserve graph
coverage, limitations and unknowns. Native CLI analysis still requires its
selected engine, and all CLI operations retain the host's execution permissions.
@@ -1,106 +1,20 @@
# JavaScript and Electron application artifacts # JavaScript and Electron application artifacts
Use `analyze_javascript_application` directly on the operator-supplied ASAR or Use `analyze_javascript_application` directly on the operator-supplied ASAR or
extracted tree. When size is unknown or large and the connected schema advertises extracted tree. The complete result, graph, and Evidence context are returned
`detail`, request `detail: "summary"`. This returns an `inspect_analysis_view` inline when they fit the MCP frame. Source-map contents are part of the static analysis. Start from its
Evidence record with a distinct ID and a `normalized_result.parent_evidence_id` findings, coverage, unknowns, and graph context, then make a focused follow-up
for the complete retained application analysis. Source-map contents are part of only when a specific question remains unanswered.
that static analysis. Read findings, coverage, limitations and unknowns, then
follow a specific unanswered question. Older servers without `detail` return
the complete application Evidence inline when it fits.
When the connected schema advertises `max_heap_mb` and `analysis_timeout_ms`,
these select the isolated source analyzer's old-space heap and per-source
deadline. Defaults are 1024 MiB and 300000 ms. They do not change the MCP
process heap. Resource-limited analyses preserve completed static facts and
report partial semantic coverage with specific unknown scopes; this does not
mean the source is malformed. Inspect those limits before retrying a source.
When expanded relationships exceed the parent's available memory, inspect the
file's modules item view for retained `static_analysis` and
`semantic_module_analysis`. Their facts remain available even when graph links
are unknown. Cumulative result exhaustion can stop analysis before every file
is examined; the inventory and remaining unknown scopes distinguish that case.
Timeout or cancellation after source analysis begins retains partial Evidence
in the same session. A returned `error.details.partial_observation` reference
can be supplied directly to `inspect_analysis_view`. If the SDK cancels locally, retrieve its recorded ID from
the Evidence bundle before inspecting the partial analysis.
## Read a module without repeating analysis
Use the actual returned parent ID, on the same connection, as `source.evidence_id`.
The following IDs are placeholders to replace with returned values. When the
module's node ID is not already known, page identities before selecting an item.
Pages return only node IDs, paths and roles; choose a useful page size and follow
`coverage.next_offset` when another page matters:
```json
{
"source": {
"kind": "retained-evidence",
"evidence_id": "ev_ACTUAL_PARENT_ID"
},
"view": { "kind": "page", "collection": "modules", "offset": 0, "limit": 16 }
}
```
Select the relevant page entry by its exact `node_id` and recorded role:
```json
{
"source": {
"kind": "retained-evidence",
"evidence_id": "ev_ACTUAL_PARENT_ID"
},
"view": {
"kind": "item",
"collection": "modules",
"selector": { "node_id": "jag_node_ACTUAL_NODE_ID" }
}
}
```
Module pages include asset, bundled and source module roles, so one path can
identify several entries. Use `selector.path` only when uniqueness is established;
the returned node ID avoids needing a complete inventory to prove uniqueness. On ambiguity,
select the matching node ID from the page or the error's candidates. Follow
`next_offset` when another page matters. An item exposes that module node's
recorded observations, identity, hashes, exports and source ranges; it does not
include every child syntax observation, related edge or original source file.
An absent item fact is not evidence of absence from the whole application.
## Trace only the remaining question
Use `trace_application_feature` on the parent application Evidence for one
literal node ID, route, string, API, IPC channel, module, or native export.
Select `incoming` for relationships leading to the seed, `outgoing` for those
leaving it, and `both` when the question needs both. Trace traversal includes
containment relationships, so even a narrow seed can expand many nodes; a
bidirectional trace can reach unrelated sibling modules. A module item or page
is preferable when its observations already answer the question. Do not treat
all reached nodes as participants in the feature.
Reuse retained parent Evidence when the server advertises it, or supply complete
Evidence inline. Include Hopper or Ghidra Evidence only when its artifact digest
matches exactly.
## Recover an oversized display
An oversized response reports `resource_constraint` and An oversized response reports `resource_constraint` and
`details.resource: "transport"`. Reuse `details.resource: "transport"`. Reuse
`details.reported_limits.evidence_reference` with `inspect_analysis_view` for a `details.reported_limits.evidence_reference` with `inspect_analysis_view` for a
summary, one module, or a stable page of module identities. summary, one module, or a stable page of module identities. Use
`trace_application_feature` once a module, route, or string seed is known.
`export_evidence_bundle` writes the complete canonical session to a `export_evidence_bundle` writes the complete canonical session to a
caller-selected file. These workflows keep the original Evidence and coverage; caller-selected file. These workflows keep the original Evidence and coverage;
a broad follow-up can also exceed framing. a broad follow-up can also exceed framing.
A client can truncate a successful result inside its text envelope without a
REA `resource_constraint`. Treat a truncation notice or incomplete JSON as a
display limitation; do not claim the full record was examined. Reuse the known
parent ID for smaller views, or export complete Evidence to a selected file
while the session remains open. If the parent ID was lost from the preview,
use the prior analysis summary or explicit ledger entries to recover it; do
not invent an ID or rerun analysis just to reproduce the same display.
BrowserWindow preferences, preload and contextBridge surfaces, IPC BrowserWindow preferences, preload and contextBridge surfaces, IPC
registrations, utility processes, and native binding requests are static syntax registrations, utility processes, and native binding requests are static syntax
observations. Only a unique exact literal IPC channel match supports an inferred observations. Only a unique exact literal IPC channel match supports an inferred
@@ -108,6 +22,13 @@ pairing. Dynamic or ambiguous channels remain unresolved. A requested `.node`
member is not a verified native export. Never claim runtime reachability, member is not a verified native export. Never claim runtime reachability,
registration, defaults, or policy enforcement from static analysis. registration, defaults, or policy enforcement from static analysis.
Use `trace_application_feature` on existing application Evidence for one
literal node ID, route, string, API, IPC channel, module, or native export.
Choose a direction and reuse the returned application Evidence ID when the
connected server advertises retained references, or supply complete Evidence
inline. Include
Hopper or Ghidra Evidence only when its artifact digest matches exactly.
For version comparison, analyze each version once, then call For version comparison, analyze each version once, then call
`compare_application_versions`. Accept only its digest, source-map, structural `compare_application_versions`. Accept only its digest, source-map, structural
fingerprint, or non-module semantic matches. Module ordinals and minified names fingerprint, or non-module semantic matches. Module ordinals and minified names
@@ -152,7 +152,7 @@ context. PSP/stack/device state remains unmodeled. See [DOS guide](https://githu
### Function annotations in Ghidra ### Function annotations in Ghidra
Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier. Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations.
## Offline PE resource inventory ## Offline PE resource inventory
+2 -2
View File
@@ -1,6 +1,6 @@
{ {
"_comment": "Test files are intentionally NOT excluded: the gate used to report zero clones because it skipped **/*.test.ts, hiding all test-tree duplication. Current actual is ~2.47% lines across src+ and tests/, down from 3.4% when tests were included. The 2.6% limit leaves modest headroom for ordinary changes; lower it as duplication is removed.", "_comment": "Test files are intentionally NOT excluded: the gate used to report zero clones because it skipped **/*.test.ts, hiding all test-tree duplication. Current actual is ~2.4% lines across src+ and tests/, down from 3.4% when tests were included. Lower this as duplication is removed.",
"threshold": 2.6, "threshold": 2.5,
"reporters": ["consoleFull"], "reporters": ["consoleFull"],
"ignore": [ "ignore": [
"node_modules/**", "node_modules/**",
+10 -81
View File
@@ -120,8 +120,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
); );
private static final String[] CAPABILITIES = { private static final String[] CAPABILITIES = {
"annotate_native_function", "annotate_native_function",
"set_address_name",
"set_addresses_names",
"inspect_native_load_image", "inspect_native_load_image",
"read_bytes", "read_bytes",
"address_to_file_offset", "address_to_file_offset",
@@ -376,12 +374,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
case "resolve_native_call_targets" -> resolveNativeCallTargets(request.params); case "resolve_native_call_targets" -> resolveNativeCallTargets(request.params);
case "xrefs" -> xrefs(request.params); case "xrefs" -> xrefs(request.params);
case "analyze_function" -> analyzeFunction(request.params); case "analyze_function" -> analyzeFunction(request.params);
case "set_address_name", "set_addresses_names" -> {
if (!descriptor.transport.equals("unix-socket"))
throw new RequestFailure("method_unavailable", "Windows P0 does not admit database mutation");
yield request.method.equals("set_address_name")
? setAddressName(request.params) : setAddressesNames(request.params);
}
case "annotate_native_function" -> { case "annotate_native_function" -> {
if (!descriptor.transport.equals("unix-socket")) if (!descriptor.transport.equals("unix-socket"))
throw new RequestFailure("method_unavailable", "Windows P0 does not admit database mutation"); throw new RequestFailure("method_unavailable", "Windows P0 does not admit database mutation");
@@ -425,8 +417,7 @@ public final class ReaGhidraBridge extends HeadlessScript {
result.addProperty("analysis_timed_out", timedOut); result.addProperty("analysis_timed_out", timedOut);
result.add("capabilities", GSON.toJsonTree( result.add("capabilities", GSON.toJsonTree(
java.util.Arrays.stream(CAPABILITIES) java.util.Arrays.stream(CAPABILITIES)
.filter(value -> !readOnly || !(value.equals("annotate_native_function") .filter(value -> !readOnly || !value.equals("annotate_native_function"))
|| value.equals("set_address_name") || value.equals("set_addresses_names")))
.toArray(String[]::new) .toArray(String[]::new)
)); ));
result.add("target", target); result.add("target", target);
@@ -1036,11 +1027,11 @@ public final class ReaGhidraBridge extends HeadlessScript {
for (String field : List.of("name", "comment", "inline_comment")) { for (String field : List.of("name", "comment", "inline_comment")) {
if (params.has(field)) validateAnnotationText(requireText(params, field), field); if (params.has(field)) validateAnnotationText(requireText(params, field), field);
} }
String leafName = params.has("name")
? annotationLeafName(function, requireString(params, "name")) : null;
int transaction = currentProgram.startTransaction("REA function annotations"); int transaction = currentProgram.startTransaction("REA function annotations");
boolean commit = false; boolean commit = false;
try { try {
String leafName = params.has("name")
? symbolLeafName(function.getSymbol(), requireString(params, "name")) : null;
// Validate edits through Ghidra's own setters, including name rules. // Validate edits through Ghidra's own setters, including name rules.
// A later invalid name must roll back earlier comment writes. // A later invalid name must roll back earlier comment writes.
for (String field : List.of("comment", "inline_comment")) { for (String field : List.of("comment", "inline_comment")) {
@@ -1089,18 +1080,18 @@ public final class ReaGhidraBridge extends HeadlessScript {
} }
} }
private String symbolLeafName(Symbol symbol, String requested) throws ghidra.util.exception.InvalidInputException { private String annotationLeafName(Function function, String requested) {
// Fully qualified readback must be usable as an idempotent rename input. // Fully qualified readback must be usable as an idempotent rename input.
if (requested.equals(symbol.getName(true))) return symbol.getName(); if (requested.equals(procedureName(function))) return function.getName();
String namespace = symbol.getParentNamespace().isGlobal() String namespace = function.getParentNamespace().isGlobal()
? "" : symbol.getParentNamespace().getName(true); ? "" : function.getParentNamespace().getName(true);
String prefix = namespace + "::"; String prefix = namespace + "::";
if (!namespace.isEmpty() && requested.startsWith(prefix)) { if (!namespace.isEmpty() && requested.startsWith(prefix)) {
String leaf = requested.substring(prefix.length()); String leaf = requested.substring(prefix.length());
if (!leaf.isEmpty()) return leaf; if (!leaf.isEmpty()) return leaf;
throw new ghidra.util.exception.InvalidInputException( throw new RequestFailure("invalid_function_name",
"Symbol name has an empty leaf within namespace " + namespace + " at " + "Function name has an empty leaf within namespace " + namespace + " at " +
canonicalAddress(symbol.getAddress()) + ": " + requested); canonicalAddress(function.getEntryPoint()) + ": " + requested);
} }
return requested; return requested;
} }
@@ -1425,68 +1416,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
return symbol == null ? JsonNull.INSTANCE : GSON.toJsonTree(symbol.getName(true)); return symbol == null ? JsonNull.INSTANCE : GSON.toJsonTree(symbol.getName(true));
} }
// Analyst naming: a function entry renames the function, other addresses get a primary label.
private boolean applyAddressName(Address address, String name) throws Exception {
if (!currentProgram.getMemory().contains(address))
throw new RequestFailure("invalid_request", "Address is not mapped: " + canonicalAddress(address));
validateAnnotationText(name, "name");
int transaction = currentProgram.startTransaction("REA address name");
boolean commit = false;
try {
Function function = currentProgram.getFunctionManager().getFunctionAt(address);
String expected = name;
if (function != null) {
expected = symbolLeafName(function.getSymbol(), name);
function.setName(expected, SourceType.USER_DEFINED);
} else {
var table = currentProgram.getSymbolTable();
Symbol primary = table.getPrimarySymbol(address);
if (primary != null && !primary.isDynamic()) {
expected = symbolLeafName(primary, name);
primary.setName(expected, SourceType.USER_DEFINED);
} else {
Symbol created = table.createLabel(address, name, SourceType.USER_DEFINED);
created.setPrimary();
}
}
invalidateAnalysisCaches();
Symbol readback = currentProgram.getSymbolTable().getPrimarySymbol(address);
commit = readback != null && expected.equals(readback.getName());
return commit;
}
catch (ghidra.util.exception.InvalidInputException | ghidra.util.exception.DuplicateNameException exception) {
return false;
}
finally {
currentProgram.endTransaction(transaction, commit);
}
}
private JsonElement setAddressName(JsonObject params) throws Exception {
requireKeys(params, Set.of("document", "address", "name"));
requireDocument(params);
return GSON.toJsonTree(applyAddressName(requireAddress(params, "address"), requireString(params, "name")));
}
private JsonElement setAddressesNames(JsonObject params) throws Exception {
requireKeys(params, Set.of("document", "names"));
requireDocument(params);
if (!params.has("names") || !params.get("names").isJsonObject())
throw new RequestFailure("invalid_request", "names must be an object of address -> name");
JsonObject result = new JsonObject();
for (Map.Entry<String, JsonElement> entry : params.getAsJsonObject("names").entrySet()) {
boolean ok;
try {
ok = applyAddressName(parseReaAddress(entry.getKey()), entry.getValue().getAsString());
}
catch (RequestFailure failure) {
ok = false;
}
result.addProperty(entry.getKey(), ok);
}
return result;
}
private JsonArray listDocuments(JsonObject params) { private JsonArray listDocuments(JsonObject params) {
requireKeys(params, Set.of()); requireKeys(params, Set.of());
JsonArray result = new JsonArray(); JsonArray result = new JsonArray();
-6
View File
@@ -136,12 +136,6 @@ existing analysis families directly, for example
- Pull and capture outputs are additive: the destination must not already - Pull and capture outputs are additive: the destination must not already
exist. Re-running a completed pull refuses the existing directory instead exist. Re-running a completed pull refuses the existing directory instead
of overwriting evidence. of overwriting evidence.
- Cancellation during a package pull preserves caller-selected output. The
error includes partial observations identifying completed artifacts and the
interrupted local path; remaining APKs were not attempted.
- Failed package acquisition rollback retains its directory identity and retries
cleanup before later work and during shutdown. Cleanup refuses a replacement
directory. Terminal shutdown stops admission and drains active operations.
- The provider holds no long-lived processes; every invocation is one - The provider holds no long-lived processes; every invocation is one
bounded adb run. The adb server belongs to the caller. bounded adb run. The adb server belongs to the caller.
+3 -10
View File
@@ -166,16 +166,9 @@ rea import-reference-source /absolute/path/to/source
``` ```
The import records hashes and metadata for the supplied files, separately The import records hashes and metadata for the supplied files, separately
from observations of the current app. It honors the source tree's `.gitignore` from observations of the current app. File names do not automatically exclude
and applies default exclusions for common generated and dependency paths such files. Set `REA_REFERENCE_SECRET_PATTERNS_JSON` to a JSON array of ignore
as `node_modules/`, `dist/`, and `*.log`. Pattern-based exclusions record the patterns when you want to exclude selected paths.
matched pattern and whether it came from project, default, caller, or sensitive
path policy. Set `REA_REFERENCE_SECRET_PATTERNS_JSON` to a JSON array of ignore
patterns when you want to mark selected paths as sensitive; those patterns are
reported separately as configured-secret exclusions.
Re-import saved graphs whose pattern-based exclusions lack `pattern` before
using them in source-to-bundle comparisons.
JavaScript and TypeScript import parsing requires valid UTF-8. Malformed source JavaScript and TypeScript import parsing requires valid UTF-8. Malformed source
bytes retain their original hashes and sizes with a decoding diagnostic; REA bytes retain their original hashes and sizes with a decoding diagnostic; REA
-12
View File
@@ -29,13 +29,6 @@ section names or the record being inside the initial 4 KiB header probe.
Program and section declarations must agree; duplicate, conflicting, truncated Program and section declarations must agree; duplicate, conflicting, truncated
or out-of-file records are rejected. Stripped section tables and section-only or out-of-file records are rejected. Stripped section tables and section-only
records are supported. Extended table numbering is explicitly unsupported. records are supported. Extended table numbering is explicitly unsupported.
For carved ELFs whose section table extends beyond the captured file, REA
examines every complete captured section header and records the unavailable
headers as a limitation. A valid program-header ABI record can still establish
the supported declaration. Available section declarations must agree with it;
their duplicate, conflicting or unreadable records remain errors. A missing
program-header record does not make ABI declarations in uncaptured sections
absent or supply a default ABI.
The field layout follows The field layout follows
[`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h). [`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h).
@@ -99,11 +92,6 @@ observations and does not relabel the upstream result as verified.
CLI/MCP discovery and decompilation, direct calls, target/profile/Evidence CLI/MCP discovery and decompilation, direct calls, target/profile/Evidence
identity, snapshot round-trip and final process cleanup remain covered. identity, snapshot round-trip and final process cleanup remain covered.
For both byte orders, the lane also carves away the section table while retaining
all program-header payloads. It checks the actual Ghidra language, entry-address
decompilation, distinct artifact digest and missing-section limitations through
CLI Evidence and an MCP-produced snapshot. Caller-selected Ghidra heap limits
are passed to these child workflows.
Pseudocode presence is a liveness check, not proof of semantic equivalence. Pseudocode presence is a liveness check, not proof of semantic equivalence.
Instruction decoding and reader agreement likewise do not establish runtime, Instruction decoding and reader agreement likewise do not establish runtime,
delay-slot execution or complete floating-point semantics. Assertion regressions delay-slot execution or complete floating-point semantics. Assertion regressions
+4 -45
View File
@@ -68,15 +68,7 @@ selects the npm release, and persistent MCP registrations are pinned to the
version that performed setup. Installing newer instructions does not update a version that performed setup. Installing newer instructions does not update a
running server or its registration. running server or its registration.
The npm release checked on **2026-10-10** was **6.3.0**, published from The release checked on **2026-10-07** was **5.0.0** (133 MCP tools), published
[`74cf9e14`](https://github.com/morluto/rea/commit/74cf9e1401fcd89482fe39c27cc898f5858e9b72),
with bundled skill version 34. It does not include automatic Pi or Hermes
registration: `setup --client pi` and `setup --client hermes` reject those IDs
before writing configuration. Their setup workflows below require a source build
or a later release containing those integrations; `@latest` alone cannot select
unpublished changes.
The earlier release checked on **2026-10-07** was **5.0.0** (133 MCP tools), published
from the fixed checkpoint from the fixed checkpoint
[`b33236ec`](https://github.com/morluto/rea/releases/tag/rea-agents-5.0.0). [`b33236ec`](https://github.com/morluto/rea/releases/tag/rea-agents-5.0.0).
The public CLI, MCP catalog and target-free session, and isolated update from The public CLI, MCP catalog and target-free session, and isolated update from
@@ -150,29 +142,7 @@ listed after the table because its connector is not one of these files:
| Pi | `pi` | | Pi | `pi` |
| Hermes | `hermes` | | Hermes | `hermes` |
For GitHub Copilot CLI 1.0.95, `gpt-4.1` model metadata can block ordinary
chat with REA's full input-schema profile before sending a model request
(`compaction_static_context_blocked`; [#1554](https://github.com/morluto/rea/issues/1554)).
For the native offline OpenAI-compatible BYOK adapter, full-profile chat and
analysis also passed with a distinct `COPILOT_PROVIDER_MODEL_ID` such as
`rea-gpt-4.1`, `COPILOT_PROVIDER_WIRE_MODEL=gpt-4.1`, and that distinct ID selected
through `copilot --model rea-gpt-4.1`. The local fixture requested
`COPILOT_PROVIDER_MAX_PROMPT_TOKENS=1000000`; configure the limit supported by
your backend. This keeps every full schema and the actual wire model intact.
It does not establish the effective client budget or fix the built-in-ID path.
The existing compact profile passed native chat and analysis with the complete
tool inventory on Linux using a loopback model fixture. After setup, add
`"REA_MCP_INPUT_SCHEMA_PROFILE": "compact"` to the `env` object of the
`mcpServers.rea` entry in Copilot's `mcp-config.json`, preserving its other
settings, then restart the client. Compact advertisements can reduce nested
schema constraints; REA still validates complete canonical inputs. Full-profile
`gpt-5.4` metadata is also verified locally. Live model API acceptance and
Windows remain unverified; see [native client verification](testing.md).
For [Pi](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/mcp.md), For [Pi](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/mcp.md),
automatic setup requires a build containing the integration; npm 6.3.0 does not
include it. See [released package and main](#released-package-and-main).
In a supported build,
setup writes a `type: "stdio"` entry under `mcpServers.rea` in the user-level setup writes a `type: "stdio"` entry under `mcpServers.rea` in the user-level
`~/.pi/agent/mcp.json` and installs the bundled skill under `~/.agents/skills`. `~/.pi/agent/mcp.json` and installs the bundled skill under `~/.agents/skills`.
Use `rea setup --client pi --dry-run --json` to inspect the plan, then Use `rea setup --client pi --dry-run --json` to inspect the plan, then
@@ -231,10 +201,7 @@ registration. Doctor treats an `enabled: false` entry as active when
`enabledServers` lists `rea`, as OMP does, unless `disabledServers` also lists `enabledServers` lists `rea`, as OMP does, unless `disabledServers` also lists
it. Run setup under each profile that should load REA. it. Run setup under each profile that should load REA.
For Hermes, automatic setup also requires a build containing the integration; For Hermes, setup writes a `mcp_servers.rea` entry to `config.yaml`, preserving
npm 6.3.0 does not include it. See
[released package and main](#released-package-and-main).
In a supported build, setup writes a `mcp_servers.rea` entry to `config.yaml`, preserving
comments and unrelated settings. It resolves that file from `HERMES_HOME`, else comments and unrelated settings. It resolves that file from `HERMES_HOME`, else
from the platform default Hermes itself uses — `%LOCALAPPDATA%\hermes` on from the platform default Hermes itself uses — `%LOCALAPPDATA%\hermes` on
Windows, `~/.hermes` elsewhere — including any `HERMES_DATA_DIR_SUFFIX`. Hermes Windows, `~/.hermes` elsewhere — including any `HERMES_DATA_DIR_SUFFIX`. Hermes
@@ -367,12 +334,6 @@ Hopper installation or any file write. Declining or pressing Ctrl-C makes no
changes. Agent configuration writes preserve unrelated entries and comments, changes. Agent configuration writes preserve unrelated entries and comments,
create backups, use atomic replacement, and verify their result. Setup and create backups, use atomic replacement, and verify their result. Setup and
uninstall retain an existing `.rea.backup` rather than replacing the first snapshot. uninstall retain an existing `.rea.backup` rather than replacing the first snapshot.
Setup also retains valid environment overrides in REA's existing server entry,
such as `REA_MCP_INPUT_SCHEMA_PROFILE` and `REA_MCP_MAX_RESPONSE_BYTES`.
Detected provider settings replace only matching environment keys.
Setup and doctor report a malformed REA environment map or invalid runtime
setting before setup writes files. Repair the named setting in the client
configuration; setup does not silently discard the rest of the map.
After setup, REA reports which agents, analysis tools, and workflow files passed After setup, REA reports which agents, analysis tools, and workflow files passed
its final checks. Restart any agent named in the completion message, then begin its final checks. Restart any agent named in the completion message, then begin
@@ -637,10 +598,8 @@ dossiers, instructions, recovered data types, measured load mappings, loaded
memory bytes, and observed file offsets. Independent load-image attestation memory bytes, and observed file offsets. Independent load-image attestation
supports DOS MZ and explicitly selected COM; PE returns its measurements with that limitation. supports DOS MZ and explicitly selected COM; PE returns its measurements with that limitation.
On Linux and macOS, `annotate_native_function` also edits a function name and/or On Linux and macOS, `annotate_native_function` also edits a function name and/or
entry comments atomically and returns refreshed analysis, and `set_address_name` entry comments atomically and returns refreshed analysis. These session metadata
and `set_addresses_names` name a function entry or any other mapped address edits leave executable bytes unchanged and are discarded on close. GUI controls
(creating or renaming its primary label) with per-address success results. These
session metadata edits leave executable bytes unchanged and are discarded on close. GUI controls
require Hopper; Windows P0 remains read-only. require Hopper; Windows P0 remains read-only.
Windows P0 admits native x86 and x86-64 PE applications and DLLs on fixed local NTFS volumes. Windows P0 admits native x86 and x86-64 PE applications and DLLs on fixed local NTFS volumes.
+3 -43
View File
@@ -89,9 +89,9 @@ queries use per-file range indexes, while call sites use exact-range lookups.
Existing node budgets and coverage reporting remain in effect. Existing node budgets and coverage reporting remain in effect.
Progress identifies the source file being parsed and projected. Cancellation is Progress identifies the source file being parsed and projected. Cancellation is
checked between files and throughout result transfer. The owned analyzer can checked between files, with event-loop yields that allow notifications and
be stopped while its synchronous parser or semantic extractor is active; completed file-local allocations to be released. A single file's synchronous
the CLI/MCP process remains available for control requests. parser and semantic extraction still need to finish before that boundary.
Graph and Evidence identifiers hash canonical JSON incrementally, without Graph and Evidence identifiers hash canonical JSON incrementally, without
assembling a single string for the whole graph. The canonical bytes and existing assembling a single string for the whole graph. The canonical bytes and existing
@@ -113,46 +113,6 @@ smaller view, for example `--format json --filter-output
evidence_id,normalized_result.statistics`. Streaming output does not bound the evidence_id,normalized_result.statistics`. Streaming output does not bound the
memory needed to construct the analysis graph itself. memory needed to construct the analysis graph itself.
JavaScript source parsing, static analysis and semantic projection run in one
owned analysis process, reused sequentially across files. Target code is never
executed. The analyzer defaults to 1024 MiB of V8 old-space and a 300000 ms
deadline per source, including startup and fact transfer. Select
`max_heap_mb` and `analysis_timeout_ms` through MCP, or `--max-heap-mb` and
`--analysis-timeout-ms` on the dedicated CLI command. These choices control the
analyzer; the caller's `NODE_OPTIONS` still controls the CLI/MCP process heap.
V8 old-space limits do not impose a process RSS limit.
A source that exhausts the analyzer heap does not terminate the CLI/MCP
process. Completed static facts are retained, other files can still be analyzed,
and unexamined semantic relationships remain explicit unknowns. The parent
also checks transfer expansion and accumulated results against its remaining
heap before accepting semantic facts. A resource-limited result has partial
coverage, the observed limits and an `analyze-javascript-source` unknown scope
with the specific failure, including captured diagnostics when available.
Resource failures do not increment syntax parse failures. There is no fixed
source-length cutoff.
Static observations also need memory when expanded into nodes, relationships
and Evidence. If that expansion does not fit, the file node retains
`static_analysis` and any admitted `semantic_module_analysis` observations,
including their exact values and source locations. Use the public modules item
view to read them. The graph marks those unexpanded relationships as unknown.
If cumulative result capacity is exhausted, analysis stops; inventoried but
unexamined files remain present with explicit unknown scopes.
A timeout stops the remaining analysis and returns typed failure details with
completed partial Evidence. MCP failure details return
`partial_observation: { kind: "retained-evidence", evidence_id: "..." }`
after recording succeeds, so `inspect_analysis_view` can open that analysis
directly without serializing the entire partial graph into the error response.
CLI failure output retains the complete partial Evidence.
Cancellation after source analysis begins likewise
retains completed facts; an SDK client may reject its cancelled request before
receiving that reply. On the same MCP connection, inspect the Evidence bundle
and use the retained analysis ID with `inspect_analysis_view`. Owned analyzer
cleanup is verified before its private files are released. Cleanup uncertainty
is reported separately and retains ownership for retry.
CLI workflows parse larger JSON input files incrementally from one verified CLI workflows parse larger JSON input files incrementally from one verified
regular-file handle, without constructing a document-sized string. Smaller regular-file handle, without constructing a document-sized string. Smaller
files retain native JSON parsing. Both paths require strict JSON and valid files retain native JSON parsing. Both paths require strict JSON and valid
-6
View File
@@ -397,12 +397,6 @@ unpaired Unicode surrogate; a rejection identifies the field and UTF-16 index
and leaves every annotation unchanged. CRLF, supplementary Unicode characters, and leaves every annotation unchanged. CRLF, supplementary Unicode characters,
and combining characters are preserved. Windows P0 remains read-only. Ghidra has no and combining characters are preserved. Windows P0 remains read-only. Ghidra has no
GUI authority, and REA never falls back automatically to Hopper. GUI authority, and REA never falls back automatically to Hopper.
`set_address_name` and `set_addresses_names` give Ghidra sessions the same
compact naming results as Hopper: a local function entry renames that function,
and any other mapped address, such as a global, receives a new or renamed primary
label. A batch applies each address separately and reports one success boolean
per address. Use them to name many globals and helpers cheaply; use
`annotate_native_function` when you also want comments and a refreshed dossier.
Ghidra name edits preserve the existing namespace. Supply either a leaf name Ghidra name edits preserve the existing namespace. Supply either a leaf name
such as `renamed` or a fully qualified name in that namespace, such as such as `renamed` or a fully qualified name in that namespace, such as
`alpha::renamed`. The returned qualified name can be reused as an idempotent `alpha::renamed`. The returned qualified name can be reused as an idempotent
-6
View File
@@ -114,12 +114,6 @@ from launch, must be ordered, and must fall within `timeout_ms`. Inputs marked
Environment overrides are recorded in the scenario commitment; inherited Environment overrides are recorded in the scenario commitment; inherited
values are not copied into Evidence. values are not copied into Evidence.
Terminal dimensions must satisfy `columns × (rows + scrollback) ≤ 1,000,000`.
Every scheduled resize uses the same selected scrollback and must fit this
working-buffer budget. REA rejects oversized scenarios before launching the
command; it never clamps the requested dimensions. This limit is independent
of `limits.output_bytes`, which bounds retained terminal observations.
The capture contains raw PTY output chunks and rendered terminal states, The capture contains raw PTY output chunks and rendered terminal states,
interaction dispatch outcomes, exit reason, sampled process-tree observations, interaction dispatch outcomes, exit reason, sampled process-tree observations,
and settlement status. When filesystem paths are selected, REA records and settlement status. When filesystem paths are selected, REA records
+10 -89
View File
@@ -99,10 +99,9 @@ result that its client offloads, and Pi exercises default codemode execution.
Use `REA_VERIFY_RUNTIME_ROOT` to select a production-only installed REA package. Use `REA_VERIFY_RUNTIME_ROOT` to select a production-only installed REA package.
Receipts retain client versions, result digests, host coverage and owned-process Receipts retain client versions, result digests, host coverage and owned-process
cleanup. These POSIX lanes use a deterministic loopback model, so they do not cleanup. These POSIX lanes use a deterministic loopback model, so they do not
prove live model-provider or native Windows compatibility. Qwen, Pi and Hermes prove live model-provider or native Windows compatibility. Qwen and Pi use
bind both `HOME` and `USERPROFILE` to the disposable account. Qwen and Pi discover caller-configured skill directories to isolate the fixture account from the OS
the shared personal skill through their default search paths without an explicit home; this does not prove their default home-directory discovery.
skill setting; Hermes discovers it through the selected `HERMES_HOME`.
Set `REA_VERIFY_HERMES_STICKY_PROFILE=1` to exercise native Hermes selection of Set `REA_VERIFY_HERMES_STICKY_PROFILE=1` to exercise native Hermes selection of
a named sticky profile with `HERMES_HOME` still pointing to its root. a named sticky profile with `HERMES_HOME` still pointing to its root.
@@ -160,39 +159,11 @@ Use `-- chat` for ordinary
chat or `REA_VERIFY_RUNTIME_ROOT` for a production-only installed package. chat or `REA_VERIFY_RUNTIME_ROOT` for a production-only installed package.
The native offline BYOK adapter uses a loopback OpenAI completions/SSE fixture The native offline BYOK adapter uses a loopback OpenAI completions/SSE fixture
and `gpt-5.4` model metadata; model inference and token usage are synthetic. and `gpt-5.4` model metadata; model inference and token usage are synthetic.
Set `REA_VERIFY_COPILOT_MODEL` to examine another client model ID and Set `REA_VERIFY_COPILOT_MODEL` to examine another model configuration. In the
`REA_VERIFY_COPILOT_WIRE_MODEL` for its backend model name (defaults to the client verified client, `gpt-4.1` blocks the complete catalog before HTTP with
ID). Every actual HTTP request's model is checked and recorded separately.
Both `HOME` and `USERPROFILE` point to the disposable account; native `skill add`
still selects the installed skill directory explicitly.
The same native client passed full-profile chat and analysis with the actual
wire model `gpt-4.1` under a distinct BYOK model ID:
```bash
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- chat
REA_VERIFY_COPILOT_MODEL=rea-gpt-4.1 REA_VERIFY_COPILOT_WIRE_MODEL=gpt-4.1 npm run verify:copilot-client -- call
```
These runs retain full canonical input/output schemas, all tools, installed-skill
loading and complete Evidence readback. They use the fixture's requested
one-million-token BYOK prompt setting; effective capacity and live API acceptance
remain unknown. With the built-in `gpt-4.1` model ID, the verified client still
blocks the full input-schema profile before HTTP with
`compaction_static_context_blocked`, even when requesting a larger BYOK prompt `compaction_static_context_blocked`, even when requesting a larger BYOK prompt
capacity. The effective capacity is unknown. The default verifier preserves capacity. The effective capacity is unknown; this lane does not establish a
the complete catalog and full schemas. To verify the existing compact profile: fix for that client/model limit. It preserves the complete catalog and schemas.
```bash
REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- chat
REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- call
```
Both native workflows passed with the complete tool inventory. The verifier selects the
profile only in its disposable Copilot registration and records it in the
receipt. Compact advertisements can reduce nested validation structure; the
server still validates complete canonical inputs. This is a tested alternate
workflow, not a fix for the full-profile client/model limit tracked in #1554.
Live model APIs, native Windows and default OS-home skill discovery remain Live model APIs, native Windows and default OS-home skill discovery remain
unverified. unverified.
@@ -261,9 +232,8 @@ unverified; skill discovery uses an explicit isolated custom directory.
verified with `@deepseek-ai/dsh@0.2.0-rc.2`) and Git. Set verified with `@deepseek-ai/dsh@0.2.0-rc.2`) and Git. Set
`REA_VERIFY_DEEPSEEK_COMMAND` to select its executable. This optional POSIX lane `REA_VERIFY_DEEPSEEK_COMMAND` to select its executable. This optional POSIX lane
installs the skill through `rea setup --skill`, writes the native Cordis MCP installs the skill through `rea setup --skill`, writes the native Cordis MCP
patch into an isolated `DSH_HOME`, and binds `HOME` and `USERPROFILE` to the patch into an isolated `DSH_HOME`, and uses `DSH_AGENTS_HOME` for native personal
disposable account. The native default `~/.agents/skills` search path discovers skill discovery. A separate Git root prevents inherited project skills from
the installed skill without a `DSH_AGENTS_HOME` override. A separate Git root prevents inherited project skills from
masking a missing installation. The default `call` mode checks skill loading, masking a missing installation. The default `call` mode checks skill loading,
complete catalog discovery, the actual forwarded input schemas and regexes, complete catalog discovery, the actual forwarded input schemas and regexes,
and full JavaScript Evidence for a Unicode path. Use `-- chat` for ordinary and full JavaScript Evidence for a Unicode path. Use `-- chat` for ordinary
@@ -1083,39 +1053,6 @@ real local Codex CLI with:
npm run verify:agent npm run verify:agent
``` ```
The lane runs scenarios sequentially in standalone Codex (`--no-daemon`) with a
disposable HOME and CODEX_HOME under `~/.cache/rea-agent-evaluations/`. Set
`REA_AGENT_EVAL_ROOT` to choose another parent directory outside the OS temporary
directory: current Codex refuses helper aliases under `/tmp`, preventing shell
commands and skill reads even if MCP works. Each owned run directory is removed
after verification unless fixture retention is requested. The lane
installs the packaged skill and MCP registration through public `rea setup`,
rather than copying a skill into a project that also inherits the caller's
skills and configuration. Only authentication is copied, when present, and it
is removed even with `REA_AGENT_EVAL_KEEP_FIXTURES=true`. No user configuration,
rules, plugins or history is copied; shell snapshots and additional agents are
disabled. Existing API-key environment authentication remains available.
The native missing-provider scenario preapproves only `open_binary` and
`close_binary` in the disposable client so it reaches REA's actual provider
boundary instead of stopping at Codex approval. This does not validate a real
native engine or authorize arbitrary runtime execution.
Select a smaller run with `REA_AGENT_EVAL_SCENARIOS`, a comma-separated list of
scenario IDs. `asar-zh` exercises the packaged desktop rubric from a Chinese
request. `javascript-module-view` exercises summary analysis, module paging and
item inspection in a tree with unrelated modules; its routing and text checks
remain heuristic, so review the retained transcript for selector choice and
coverage claims. Set `REA_CODEX_CLI` to an installed Codex executable and
`REA_AGENT_EVAL_MODEL` to an account-supported model. Use an external resource
limit for the whole process tree when testing on a constrained host; a Node
heap limit alone does not bound client and child-process memory.
`missing-target` leaves desktop fixtures visible without selecting one in the
request. It requires zero REA calls and a structured request for an app name or
artifact path. Its `expectedFirstTool` is `null`, and
`targetClarificationPassed` grades that limited routing outcome independently
of analysis prose heuristics. Analyzing a nearby example cannot pass this case.
The packaged desktop and parser-comparison scenarios now assess a closed set of The packaged desktop and parser-comparison scenarios now assess a closed set of
known fixture claims. Each final answer must be a strict JSON manifest containing known fixture claims. Each final answer must be a strict JSON manifest containing
exactly the requested claim IDs, values, Evidence IDs, and Evidence authority and exactly the requested claim IDs, values, Evidence IDs, and Evidence authority and
@@ -1140,21 +1077,9 @@ navigation-context, and address-context scenarios have no factual rubric and
remain `not_assessed`; the managed workflow requires both artifact and member remain `not_assessed`; the managed workflow requires both artifact and member
inspection to answer its type and entry-point question. inspection to answer its type and entry-point question.
The closed factual scenarios request complete producer Evidence on the initial
analysis so their source selectors can authenticate all configured facts.
Summary/page/item workflows are exercised separately by `javascript-module-view`;
its heuristic gate does not establish full-producer factual correctness.
All scenarios retain routing, workflow, validation, repetition, process-exit, All scenarios retain routing, workflow, validation, repetition, process-exit,
and token-use gates. Configured factual scenarios additionally require a factual and token-use gates. Configured factual scenarios additionally require a factual
pass. Their answer-text heuristics are diagnostic and do not affect the gate. pass. Their answer-text heuristics are diagnostic and do not affect the gate.
Validation counts include complete text-only REA `invalid_request` envelopes
and failed SDK calls whose arguments violate the named current REA contract.
Client approval rejections and provider unavailability remain distinct failures;
truncated JSON previews are not parsed into invented error codes.
Repeated `binary_session` observations separated by a successful `open_binary`
or `close_binary` are treated as lifecycle verification. An unchanged retry,
including one after a failed lifecycle call, still counts as repetition.
Scenarios without a rubric retain the legacy text-heuristic gate. Scenarios without a rubric retain the legacy text-heuristic gate.
`answerTermCoverageMet` checks case-insensitive substrings, `answerTermCoverageMet` checks case-insensitive substrings,
`epistemicCuePresent` checks keywords, and `finalCitesEvidence` checks only an ID's `epistemicCuePresent` checks keywords, and `finalCitesEvidence` checks only an ID's
@@ -1162,11 +1087,7 @@ presence. These metrics can still accept fabricated prose and must not be read
as factual assessments. Set `REA_AGENT_EVAL_TRANSCRIPT_DIR` to retain complete as factual assessments. Set `REA_AGENT_EVAL_TRANSCRIPT_DIR` to retain complete
tool results and final answers for review. tool results and final answers for review.
Report schema version 4 adds negative routing scenarios with nullable Report schema version 3 changes the top-level `factualCorrectness` from a constant
`expectedFirstTool` and a `targetClarificationPassed` outcome. Consumers must
allow an expected absence of REA calls; this does not make an analysis scenario
pass without its required tools. Version 3 changed the top-level
`factualCorrectness` from a constant
string to an assessment summary with status, scope, and assessed/passed/failed/ string to an assessment summary with status, scope, and assessed/passed/failed/
not-assessed scenario counts. Scenario records include the factual assessment and not-assessed scenario counts. Scenario records include the factual assessment and
configured claim IDs. The evaluation scope is configured claim IDs. The evaluation scope is
+4 -7
View File
@@ -85,22 +85,19 @@ if [[ "$version" =~ \+ ]]; then
fail "npm cannot install an exact version with build metadata. Pass a version without build metadata, then retry." fail "npm cannot install an exact version with build metadata. Pass a version without build metadata, then retry."
fi fi
install_args=(install --global) prefix_args=()
if [[ "$platform" == "Linux" ]]; then if [[ "$platform" == "Linux" ]]; then
install_args+=(--prefix "$HOME/.local") prefix_args=(--prefix "$HOME/.local")
install_bin="$HOME/.local/bin/rea" install_bin="$HOME/.local/bin/rea"
else else
npm_prefix="$(npm prefix --global)" || fail "the npm global prefix could not be read. Repair the npm configuration, then retry." npm_prefix="$(npm prefix --global)" || fail "the npm global prefix could not be read. Repair the npm configuration, then retry."
install_bin="$npm_prefix/bin/rea" install_bin="$npm_prefix/bin/rea"
fi fi
install_args+=("$PACKAGE@$version")
printf 'REA install plan\n' printf 'REA install plan\n'
printf ' Version: %s\n' "$version" printf ' Version: %s\n' "$version"
printf ' Runtime: Node.js %s\n' "$node_version" printf ' Runtime: Node.js %s\n' "$node_version"
printf ' Command:' printf ' Command: npm install --global %s@%s\n' "$PACKAGE" "$version"
printf ' %q' npm "${install_args[@]}"
printf '\n'
printf ' Binary: %s\n' "$install_bin" printf ' Binary: %s\n' "$install_bin"
printf ' Setup: %s\n' "$([[ "$start_setup" == true ]] && printf 'start when a terminal is available' || printf 'skipped')" printf ' Setup: %s\n' "$([[ "$start_setup" == true ]] && printf 'start when a terminal is available' || printf 'skipped')"
@@ -110,7 +107,7 @@ if [[ "$dry_run" == true ]]; then
fi fi
printf 'Installing %s@%s...\n' "$PACKAGE" "$version" printf 'Installing %s@%s...\n' "$PACKAGE" "$version"
npm "${install_args[@]}" || fail "npm could not install REA. Check registry access and npm permissions, then retry." npm install --global ${prefix_args[@]+"${prefix_args[@]}"} "$PACKAGE@$version" || fail "npm could not install REA. Check registry access and npm permissions, then retry."
[[ -x "$install_bin" ]] || fail "npm completed without installing the rea command. Check the npm global bin directory and PATH, then retry." [[ -x "$install_bin" ]] || fail "npm completed without installing the rea command. Check the npm global bin directory and PATH, then retry."
installed_version="$("$install_bin" --version 2>/dev/null | tr -d '[:space:]')" || fail "the installed REA version could not be read. Reinstall the requested version, then retry." installed_version="$("$install_bin" --version 2>/dev/null | tr -d '[:space:]')" || fail "the installed REA version could not be read. Reinstall the requested version, then retry."
[[ "$installed_version" == "$version" ]] || fail "installed version $installed_version does not match $version. Reinstall the requested version, then retry." [[ "$installed_version" == "$version" ]] || fail "installed version $installed_version does not match $version. Reinstall the requested version, then retry."
-1
View File
@@ -3,7 +3,6 @@
"entry": [ "entry": [
"src/main.ts", "src/main.ts",
"src/cli.ts", "src/cli.ts",
"src/composition/javascriptAnalysisWorkerProcess.ts",
"src/contracts/errorSchemas.ts", "src/contracts/errorSchemas.ts",
"src/evaluation/AgentEvaluationFixtures.ts", "src/evaluation/AgentEvaluationFixtures.ts",
"src/application/RealHopperAssertions.ts", "src/application/RealHopperAssertions.ts",
-3
View File
@@ -30,9 +30,6 @@
"access": "public" "access": "public"
}, },
"type": "module", "type": "module",
"imports": {
"#javascript-analysis-worker": "./dist/composition/javascriptAnalysisWorkerProcess.js"
},
"bin": { "bin": {
"rea": "scripts/rea.mjs", "rea": "scripts/rea.mjs",
"rea-agents": "scripts/rea.mjs" "rea-agents": "scripts/rea.mjs"
@@ -1,28 +0,0 @@
import { analyzeJavaScriptApplication } from "../../dist/application/javascript/JavaScriptApplicationService.js";
import { ArtifactResourceScope } from "../../dist/artifacts/ArtifactResourceScope.js";
/** Analyze one JavaScript application and close its caller-owned resources. */
export const analyzeJavaScriptApplicationWithOwnedScope = async (input) => {
const scope = new ArtifactResourceScope();
let outcome;
try {
const result = await analyzeJavaScriptApplication(input, scope);
outcome = result.ok
? { kind: "completed", value: result.value }
: { kind: "failed", cause: result.error };
} catch (cause) {
outcome = { kind: "failed", cause };
}
try {
await scope.close();
} catch (cleanupCause) {
if (outcome.kind === "failed")
throw new AggregateError(
[outcome.cause, cleanupCause],
"JavaScript analysis and artifact cleanup both failed",
);
throw cleanupCause;
}
if (outcome.kind === "failed") throw outcome.cause;
return outcome.value;
};
+4 -22
View File
@@ -21,6 +21,7 @@ export async function verifyGhidraEntryAliases({
new URL("../../tests/conformance/ghidra/entry-aliases.c", import.meta.url), new URL("../../tests/conformance/ghidra/entry-aliases.c", import.meta.url),
); );
const path = join(workspace, "entry-aliases"); const path = join(workspace, "entry-aliases");
const prefix = process.platform === "darwin" ? "_" : "";
try { try {
await exec(compiler, ["-O0", "-g", "-fno-inline", source, "-o", path], { await exec(compiler, ["-O0", "-g", "-fno-inline", source, "-o", path], {
env, env,
@@ -28,8 +29,7 @@ export async function verifyGhidraEntryAliases({
}); });
const bytes = await readFile(path); const bytes = await readFile(path);
await call("close_binary"); await call("close_binary");
const imported = await call("open_binary", { path, provider_id: "ghidra" }); await call("open_binary", { path, provider_id: "ghidra" });
const prefix = imported.format === "mach-o" ? "_" : "";
const names = await call("list_names"); const names = await call("list_names");
const alias = names.find( const alias = names.find(
({ value }) => value === `${prefix}rea_entry_alias`, ({ value }) => value === `${prefix}rea_entry_alias`,
@@ -46,10 +46,8 @@ export async function verifyGhidraEntryAliases({
); );
assert.ok(canonical, "Fixture alias is not at a function entry"); assert.ok(canonical, "Fixture alias is not at a function entry");
const xrefsSelectors = []; const xrefsSelectors = [];
// CLI xrefs keeps bare hexadecimal selectors as addresses. A Mach-O
// primary label such as "dead" is tested below as a procedure name instead.
for (const value of [ for (const value of [
`${prefix}rea_alias_target`, canonical.value,
alias.value, alias.value,
`${prefix}rea_xrefs_data`, `${prefix}rea_xrefs_data`,
`${prefix}rea_xrefs_unreferenced`, `${prefix}rea_xrefs_unreferenced`,
@@ -91,18 +89,6 @@ export async function verifyGhidraEntryAliases({
); );
assert.ok(interior); assert.ok(interior);
assert.notEqual(interior.address, alias.address); assert.notEqual(interior.address, alias.address);
assert.equal(
interior.symbol.type,
"label",
"Fixture interior control must remain an imported label",
);
assert.equal(
(await call("list_procedures")).some(
({ address }) => address === interior.address,
),
false,
"Fixture interior control must not be a Ghidra procedure entry",
);
const failure = await reject("procedure_address", { const failure = await reject("procedure_address", {
procedure: interior.value, procedure: interior.value,
}); });
@@ -167,11 +153,7 @@ export async function verifyGhidraEntryAliases({
{ env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 }, { env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 },
); );
const evidence = JSON.parse(stdout); const evidence = JSON.parse(stdout);
assert.deepEqual( assert.deepEqual(evidence.normalized_result, expected);
evidence.normalized_result,
expected,
`CLI xrefs mismatch for ${JSON.stringify(selector)}`,
);
assert.equal(evidence.parameters.address, selector.at(-1)); assert.equal(evidence.parameters.address, selector.at(-1));
} }
} }
@@ -54,14 +54,6 @@ export async function verifyGhidraNamespaceAnnotations({
); );
return { namespace, address: procedure.address }; return { namespace, address: procedure.address };
}); });
const names = await call("list_names");
const globals = ["alpha", "outer::inner"].map((namespace) => {
const symbol = names.find(
(item) => item.value === `${namespace}::value` && item.symbol.primary,
);
assert.ok(symbol, `Missing demangled fixture global ${namespace}::value`);
return { namespace, address: symbol.address };
});
const ambiguous = await reject("procedure_address", { procedure: "same" }); const ambiguous = await reject("procedure_address", { procedure: "same" });
assert.equal(ambiguous.code, "invalid_request"); assert.equal(ambiguous.code, "invalid_request");
for (const { address } of entries) for (const { address } of entries)
@@ -133,16 +125,6 @@ export async function verifyGhidraNamespaceAnnotations({
}); });
assert.deepEqual(unchanged.annotations, qualified.annotations); assert.deepEqual(unchanged.annotations, qualified.annotations);
} }
await verifyAddressNames(call, [entries[0], entries[2], ...globals]);
for (const { namespace, address } of entries) {
assert.equal(
await call("set_address_name", {
address,
name: `${namespace}::qualified`,
}),
true,
);
}
const { stdout } = await exec( const { stdout } = await exec(
process.execPath, process.execPath,
[ [
@@ -177,96 +159,6 @@ export async function verifyGhidraNamespaceAnnotations({
} }
} }
async function verifyAddressNames(call, symbols) {
for (const { namespace, address } of symbols) {
assert.equal(
await call("set_address_name", { address, name: "address_renamed" }),
true,
);
const qualified = `${namespace}::address_qualified`;
assert.equal(
await call("set_address_name", { address, name: qualified }),
true,
);
assert.equal(await call("address_name", { address }), qualified);
assert.equal(
await call("set_address_name", { address, name: "other::literal" }),
true,
);
const literal = await call("address_name", { address });
assert.equal(literal, `${namespace}::other::literal`);
assert.equal(
await call("set_address_name", { address, name: literal }),
true,
);
assert.equal(await call("address_name", { address }), literal);
assert.equal(
await call("set_address_name", { address, name: qualified }),
true,
);
for (let repeat = 0; repeat < 2; repeat++) {
const readback = await call("address_name", { address });
assert.deepEqual(
await call("set_addresses_names", { names: { [address]: readback } }),
{ [address]: true },
);
assert.equal(await call("address_name", { address }), qualified);
}
assert.equal(
await call("set_address_name", { address, name: `${namespace}::` }),
false,
);
const session = await call("binary_session");
for (const operation of [
"set_address_name",
"set_addresses_names",
"address_name",
"list_names",
])
assert.equal(
session.capabilities.find((item) => item.operation === operation)
?.available,
true,
`An invalid name must not mark ${operation} unavailable`,
);
assert.equal(await call("address_name", { address }), qualified);
}
const [valid, invalid] = symbols;
assert.deepEqual(
await call("set_addresses_names", {
names: {
[valid.address]: `${valid.namespace}::batch_renamed`,
[invalid.address]: `${invalid.namespace}::`,
},
}),
{ [valid.address]: true, [invalid.address]: false },
);
assert.equal(
await call("address_name", { address: valid.address }),
`${valid.namespace}::batch_renamed`,
);
assert.equal(
await call("address_name", { address: invalid.address }),
`${invalid.namespace}::address_qualified`,
);
const names = await call("list_names");
for (const { namespace, address } of symbols) {
const expected =
address === valid.address
? `${namespace}::batch_renamed`
: `${namespace}::address_qualified`;
assert.ok(
names.some(
(item) =>
item.address === address &&
item.value === expected &&
item.symbol.primary,
),
`Missing refreshed primary symbol ${expected}`,
);
}
}
async function requireNamespaceDemangler(env) { async function requireNamespaceDemangler(env) {
const inspection = inspectGhidraInstallation({ const inspection = inspectGhidraInstallation({
environment: process.env, environment: process.env,
+32 -197
View File
@@ -1,18 +1,9 @@
import { spawn } from "node:child_process"; import { spawn } from "node:child_process";
import { import { cp, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
chmod, import { tmpdir } from "node:os";
copyFile, import { dirname, join, resolve } from "node:path";
mkdir,
mkdtemp,
readFile,
rm,
writeFile,
} from "node:fs/promises";
import { homedir } from "node:os";
import { delimiter, dirname, join, resolve } from "node:path";
import { createInterface } from "node:readline"; import { createInterface } from "node:readline";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import { z } from "zod";
import { evaluateCodexEvents } from "../dist/evaluation/CodexAgentEval.js"; import { evaluateCodexEvents } from "../dist/evaluation/CodexAgentEval.js";
import { import {
@@ -24,95 +15,32 @@ import {
agentEvaluationPassed, agentEvaluationPassed,
summarizeFactualCorrectness, summarizeFactualCorrectness,
} from "../dist/evaluation/AgentEvaluationReport.js"; } from "../dist/evaluation/AgentEvaluationReport.js";
import { PRODUCT_IDENTITY } from "../dist/identity.js"; import { MCP_STARTUP_POLICY } from "../dist/mcpStartupPolicy.js";
import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs"; import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs";
const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..");
const verifierRun = createVerifierRun(); const verifierRun = createVerifierRun();
const evaluationRoot = await mkdtemp(join(tmpdir(), "rea-agent-eval-"));
const fixtureRoot = join(evaluationRoot, "targets");
const skillDestination = join(
evaluationRoot,
".agents/skills/reverse-engineer-anything",
);
const timeoutMs = Number(process.env.REA_AGENT_EVAL_TIMEOUT_MS ?? 480_000); const timeoutMs = Number(process.env.REA_AGENT_EVAL_TIMEOUT_MS ?? 480_000);
const codex = process.env.REA_CODEX_CLI ?? "codex"; const codex = process.env.REA_CODEX_CLI ?? "codex";
const optionalModel = process.env.REA_AGENT_EVAL_MODEL; const optionalModel = process.env.REA_AGENT_EVAL_MODEL;
if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 10_000) if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 10_000)
throw new Error("REA_AGENT_EVAL_TIMEOUT_MS must be an integer >= 10000"); throw new Error("REA_AGENT_EVAL_TIMEOUT_MS must be an integer >= 10000");
// Codex refuses helper aliases when CODEX_HOME is under the OS temp directory. const codexVersion = await readCommandVersion(codex);
// Keep the disposable account in a private cache, then remove the owned run.
const evaluationParent = resolve(
process.env.REA_AGENT_EVAL_ROOT ??
join(homedir(), ".cache", "rea-agent-evaluations"),
);
await mkdir(evaluationParent, { recursive: true, mode: 0o700 });
const evaluationRoot = await mkdtemp(join(evaluationParent, "run-"));
const fixtureRoot = join(evaluationRoot, "targets");
const account = join(evaluationRoot, "account");
const codexHome = join(evaluationRoot, "codex");
const authPath = join(codexHome, "auth.json");
const skillDestination = join(
account,
".agents/skills/reverse-engineer-anything",
);
const evaluationEnvironment = {
...Object.fromEntries(
Object.entries(process.env).filter(
([name]) => !/^(?:REA_|CODEX_|GHIDRA_|HOPPER_|IDA_)/u.test(name),
),
),
HOME: account,
USERPROFILE: account,
CODEX_HOME: codexHome,
XDG_CONFIG_HOME: join(account, ".config"),
XDG_DATA_HOME: join(account, ".local/share"),
XDG_CACHE_HOME: join(account, ".cache"),
REA_PROCESS_RUN_ID: verifierRun.run_id,
PATH: `${dirname(process.execPath)}${delimiter}${process.env.PATH ?? ""}`,
};
try { try {
const codexVersion = await readCommandVersion(codex); await mkdir(fixtureRoot, { recursive: true });
await Promise.all([ await cp(
mkdir(fixtureRoot, { recursive: true }), join(repositoryRoot, "skills/reverse-engineer-anything"),
mkdir(account), skillDestination,
mkdir(codexHome, { mode: 0o700 }), { recursive: true },
]);
// Reuse authentication only. User config, plugins, rules, history and skills
// must not make a fresh installed-skill trial pass accidentally.
try {
await copyFile(
join(process.env.CODEX_HOME ?? join(homedir(), ".codex"), "auth.json"),
authPath,
);
await chmod(authPath, 0o600);
} catch (cause) {
if (cause?.code !== "ENOENT") throw cause;
// Environment/API-key authentication can work without a local auth file.
}
await writeFile(
join(codexHome, "config.toml"),
'approval_policy = "never"\nsandbox_mode = "read-only"\nweb_search = "disabled"\n[features]\nmulti_agent = false\nshell_snapshot = false\n[history]\npersistence = "none"\n',
{ mode: 0o600 },
); );
for (const approval of ["--dry-run", "--yes"]) {
await runProcess(
process.execPath,
[
join(repositoryRoot, "scripts/rea.mjs"),
"setup",
"--client",
"codex",
approval,
"--json",
],
evaluationRoot,
120_000,
evaluationEnvironment,
);
}
const installedSkill = await readFile(
join(skillDestination, "SKILL.md"),
"utf8",
);
if (!installedSkill.includes(` version: "${PRODUCT_IDENTITY.skillVersion}"`))
throw new Error("Setup did not install the current packaged REA skill");
const requestedScenarioIds = new Set( const requestedScenarioIds = new Set(
(process.env.REA_AGENT_EVAL_SCENARIOS ?? "") (process.env.REA_AGENT_EVAL_SCENARIOS ?? "")
.split(",") .split(",")
@@ -129,11 +57,7 @@ try {
expectedFirstTool: "open_binary", expectedFirstTool: "open_binary",
requiresEvidence: false, requiresEvidence: false,
requiredAnswerTermGroups: [["unavailable", "could not", "provider"]], requiredAnswerTermGroups: [["unavailable", "could not", "provider"]],
// The verifier authorizes these named session operations, not arbitrary prompt: `Explain what the shipped native program at ${targets.native} does. Base the answer on artifact analysis, and clearly state any facts that remain unavailable.`,
// writes or runtime execution. Missing-engine behavior stays observable.
approvedTools: ["open_binary", "close_binary"],
requiredToolSubsequence: ["open_binary", "close_binary"],
prompt: `Explain what static artifact evidence establishes about the shipped native program at ${targets.native}. Do not execute the target or install tools. Clearly state unavailable provider evidence, and close any analysis session you opened before finishing.`,
}, },
{ {
id: "asar", id: "asar",
@@ -146,38 +70,6 @@ try {
], ],
prompt: `Explain how the desktop application at ${targets.javascript} exposes APIs to its renderer. Base the answer on the shipped application artifact, and state what remains unknown.`, prompt: `Explain how the desktop application at ${targets.javascript} exposes APIs to its renderer. Base the answer on the shipped application artifact, and state what remains unknown.`,
}, },
{
id: "asar-zh",
fixtureScenarioId: "asar",
expectedFirstTool: "analyze_javascript_application",
requiresEvidence: true,
requiredAnswerTermGroups: [["profileapi"], ["profile:read"]],
prompt: `請靜態分析已交付的 Electron 程式 ${targets.javascript},說明 renderer 如何透過 preload 讀取 profile,以及 main handler 的關係。不要執行目標或安裝分析引擎;引用完整的產出 Evidence,保留 runtime 未知事項。`,
},
{
id: "javascript-module-view",
expectedFirstTool: "analyze_javascript_application",
requiresEvidence: true,
requiredToolSubsequence: [
"analyze_javascript_application",
"inspect_analysis_view",
"inspect_analysis_view",
],
requiredAnswerTermGroups: [
["main.js"],
["node", "identity"],
["observations", "recorded"],
],
prompt: `Inspect the recorded module identity, exports and observations for main.js in the shipped JavaScript tree ${targets.largeJavascript}. Use REA without directly reading target files or executing the target. Keep results focused on that module, cite Evidence, and state the limits of what its module view establishes.`,
},
{
id: "missing-target",
expectedFirstTool: null,
requiresEvidence: false,
requiredAnswerTermGroups: [],
prompt:
'Reverse engineer my desktop app and explain how it saves user settings. Use static analysis; do not install software. End with one JSON object containing status ("needs_target" or "analyzed"), requested_input ("app_name_or_artifact_path" or null), and question (a string or null).',
},
{ {
id: "javascript-export-shape", id: "javascript-export-shape",
expectedFirstTool: "analyze_javascript_application", expectedFirstTool: "analyze_javascript_application",
@@ -253,14 +145,12 @@ try {
const results = []; const results = [];
for (const scenario of scenarios) { for (const scenario of scenarios) {
process.stderr.write(`Running Codex agent evaluation: ${scenario.id}\n`); process.stderr.write(`Running Codex agent evaluation: ${scenario.id}\n`);
const fixtureId = scenario.fixtureScenarioId ?? scenario.id; const fixtureClaims = agentFixtureClaims(scenario.id, targets);
const fixtureClaims = agentFixtureClaims(fixtureId, targets); const answerInstructions = agentFixtureAnswerInstructions(scenario.id);
const answerInstructions = agentFixtureAnswerInstructions(fixtureId);
const execution = await runCodex( const execution = await runCodex(
answerInstructions.length === 0 answerInstructions.length === 0
? scenario.prompt ? scenario.prompt
: `${scenario.prompt}\n\nThis closed factual rubric needs complete producer Evidence in the transcript. Request complete detail on each initial application analysis when supported; summary or view records alone are not accepted by this rubric.\n\n${answerInstructions}`, : `${scenario.prompt}\n\n${answerInstructions}`,
scenario.approvedTools ?? [],
); );
const transcriptDirectory = process.env.REA_AGENT_EVAL_TRANSCRIPT_DIR; const transcriptDirectory = process.env.REA_AGENT_EVAL_TRANSCRIPT_DIR;
if (transcriptDirectory !== undefined) { if (transcriptDirectory !== undefined) {
@@ -285,12 +175,6 @@ try {
...(fixtureClaims === undefined ? {} : { fixtureClaims }), ...(fixtureClaims === undefined ? {} : { fixtureClaims }),
}, },
); );
// Fixtures are deliberately visible beside prior investigation artifacts.
// Their existence does not identify the unnamed app in this request.
const targetClarificationPassed =
scenario.expectedFirstTool === null
? validTargetClarification(metrics.finalMessage)
: undefined;
const result = { const result = {
id: scenario.id, id: scenario.id,
expectedFirstTool: scenario.expectedFirstTool, expectedFirstTool: scenario.expectedFirstTool,
@@ -301,9 +185,6 @@ try {
exitCode: execution.exitCode, exitCode: execution.exitCode,
stderr: execution.stderr, stderr: execution.stderr,
...metrics, ...metrics,
...(targetClarificationPassed === undefined
? {}
: { targetClarificationPassed }),
}; };
results.push(result); results.push(result);
process.stderr.write( process.stderr.write(
@@ -313,20 +194,13 @@ try {
const factualCorrectness = summarizeFactualCorrectness(results); const factualCorrectness = summarizeFactualCorrectness(results);
const summary = { const summary = {
schemaVersion: 4, schemaVersion: 3,
evaluationScope: "routing_workflow_and_configured_fixture_claims", evaluationScope: "routing_workflow_and_configured_fixture_claims",
factualCorrectness, factualCorrectness,
verifier_run: await completeVerifierRun(verifierRun), verifier_run: await completeVerifierRun(verifierRun),
codex, codex,
codexVersion, codexVersion,
model: optionalModel ?? null, model: optionalModel ?? null,
installation: {
method: "rea_setup_codex",
isolatedHome: true,
isolatedCodexHome: true,
skillVersion: PRODUCT_IDENTITY.skillVersion,
authentication: "caller_auth_only_removed_after_run",
},
scenarios: results, scenarios: results,
totals: { totals: {
scenarios: results.length, scenarios: results.length,
@@ -379,32 +253,12 @@ try {
`Agent routing/workflow/answer checks failed: ${failed.map(({ id }) => id).join(", ")}`, `Agent routing/workflow/answer checks failed: ${failed.map(({ id }) => id).join(", ")}`,
); );
} finally { } finally {
// Even a deliberately retained fixture directory must never retain auth.
await rm(authPath, { force: true });
if (process.env.REA_AGENT_EVAL_KEEP_FIXTURES !== "true") if (process.env.REA_AGENT_EVAL_KEEP_FIXTURES !== "true")
await rm(evaluationRoot, { recursive: true, force: true }); await rm(evaluationRoot, { recursive: true, force: true });
} }
async function createTargets(root, includeManaged) { async function createTargets(root, includeManaged) {
const fixtures = await createAgentEvaluationFixtures(root, repositoryRoot); const fixtures = await createAgentEvaluationFixtures(root, repositoryRoot);
const largeJavascript = join(root, "large-app");
await mkdir(join(largeJavascript, "lib"), { recursive: true });
await Promise.all([
writeFile(
join(largeJavascript, "package.json"),
'{"name":"large-app","main":"main.js"}\n',
),
writeFile(
join(largeJavascript, "main.js"),
'const { ipcMain } = require("electron");\nipcMain.handle("profile:read", (_event, id) => ({ id }));\n',
),
...Array.from({ length: 180 }, (_, index) =>
writeFile(
join(largeJavascript, "lib", `formatter-${String(index)}.js`),
`export const format = (value) => String(value).trim();\n`,
),
),
]);
const managedOutput = join(root, "managed-output"); const managedOutput = join(root, "managed-output");
if (includeManaged) { if (includeManaged) {
@@ -438,29 +292,15 @@ async function createTargets(root, includeManaged) {
return { return {
native: "/bin/true", native: "/bin/true",
...fixtures, ...fixtures,
largeJavascript,
managed: join(managedOutput, "AgentEval.dll"), managed: join(managedOutput, "AgentEval.dll"),
}; };
} }
function validTargetClarification(message) { async function runCodex(prompt) {
const schema = z.strictObject({
status: z.literal("needs_target"),
requested_input: z.literal("app_name_or_artifact_path"),
question: z.string().refine((value) => value.trim().length > 0),
});
try {
return schema.safeParse(JSON.parse(message)).success;
} catch (cause) {
if (cause instanceof SyntaxError) return false;
throw cause;
}
}
async function runCodex(prompt, approvedTools) {
const arguments_ = [ const arguments_ = [
"--no-daemon",
"exec", "exec",
"--ignore-user-config",
"--ignore-rules",
"--ephemeral", "--ephemeral",
"--skip-git-repo-check", "--skip-git-repo-check",
"--sandbox", "--sandbox",
@@ -472,16 +312,18 @@ async function runCodex(prompt, approvedTools) {
evaluationRoot, evaluationRoot,
"-c", "-c",
'approval_policy="never"', 'approval_policy="never"',
...approvedTools.flatMap((name) => [ "-c",
"-c", `mcp_servers.rea.command=${JSON.stringify(process.execPath)}`,
`mcp_servers.rea.tools.${name}.approval_mode="approve"`, "-c",
]), `mcp_servers.rea.args=${JSON.stringify([join(repositoryRoot, "scripts/rea.mjs"), "mcp"])}`,
"-c",
`mcp_servers.rea.startup_timeout_sec=${String(MCP_STARTUP_POLICY.codexStartupTimeoutSeconds)}`,
...(optionalModel === undefined ? [] : ["--model", optionalModel]), ...(optionalModel === undefined ? [] : ["--model", optionalModel]),
prompt, prompt,
]; ];
const child = spawn(codex, arguments_, { const child = spawn(codex, arguments_, {
cwd: evaluationRoot, cwd: evaluationRoot,
env: evaluationEnvironment, env: process.env,
stdio: ["ignore", "pipe", "pipe"], stdio: ["ignore", "pipe", "pipe"],
}); });
const events = []; const events = [];
@@ -520,16 +362,9 @@ async function runCodex(prompt, approvedTools) {
return { events, exitCode, stderr: stderr.join("").slice(0, 32_768) }; return { events, exitCode, stderr: stderr.join("").slice(0, 32_768) };
} }
async function runProcess( async function runProcess(command, arguments_, cwd, timeout) {
command,
arguments_,
cwd,
timeout,
env = process.env,
) {
const child = spawn(command, arguments_, { const child = spawn(command, arguments_, {
cwd, cwd,
env,
stdio: ["ignore", "ignore", "pipe"], stdio: ["ignore", "ignore", "pipe"],
}); });
let stderr = ""; let stderr = "";
+2 -32
View File
@@ -22,13 +22,6 @@ const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
const command = process.env.REA_VERIFY_COPILOT_COMMAND ?? "copilot"; const command = process.env.REA_VERIFY_COPILOT_COMMAND ?? "copilot";
const mode = process.argv[2] ?? "call"; const mode = process.argv[2] ?? "call";
const model = process.env.REA_VERIFY_COPILOT_MODEL ?? "gpt-5.4"; const model = process.env.REA_VERIFY_COPILOT_MODEL ?? "gpt-5.4";
const wireModel = process.env.REA_VERIFY_COPILOT_WIRE_MODEL ?? model;
const inputSchemaProfile =
process.env.REA_VERIFY_COPILOT_SCHEMA_PROFILE ?? "full";
assert(
["full", "compact"].includes(inputSchemaProfile),
"REA_VERIFY_COPILOT_SCHEMA_PROFILE must be full or compact",
);
assert( assert(
["chat", "call"].includes(mode), ["chat", "call"].includes(mode),
"Usage: verify-copilot-client.mjs [chat|call]", "Usage: verify-copilot-client.mjs [chat|call]",
@@ -51,7 +44,6 @@ await exec("git", ["init", "--quiet", workspace], { timeout: 30_000 });
const environment = { const environment = {
PATH: process.env.PATH, PATH: process.env.PATH,
NODE_OPTIONS: process.env.NODE_OPTIONS, NODE_OPTIONS: process.env.NODE_OPTIONS,
HOME: account,
USERPROFILE: account, USERPROFILE: account,
COPILOT_HOME: profile, COPILOT_HOME: profile,
COPILOT_OFFLINE: "true", COPILOT_OFFLINE: "true",
@@ -62,7 +54,7 @@ const environment = {
COPILOT_PROVIDER_BEARER_TOKEN: "", COPILOT_PROVIDER_BEARER_TOKEN: "",
COPILOT_PROVIDER_HEADERS: "", COPILOT_PROVIDER_HEADERS: "",
COPILOT_PROVIDER_MODEL_ID: model, COPILOT_PROVIDER_MODEL_ID: model,
COPILOT_PROVIDER_WIRE_MODEL: wireModel, COPILOT_PROVIDER_WIRE_MODEL: model,
COPILOT_PROVIDER_MAX_PROMPT_TOKENS: "1000000", COPILOT_PROVIDER_MAX_PROMPT_TOKENS: "1000000",
COPILOT_PROVIDER_MAX_OUTPUT_TOKENS: "4096", COPILOT_PROVIDER_MAX_OUTPUT_TOKENS: "4096",
REA_PROCESS_RUN_ID: verifier.run_id, REA_PROCESS_RUN_ID: verifier.run_id,
@@ -120,17 +112,10 @@ const extractEvidence = (value) => {
} }
return undefined; return undefined;
}; };
const wireModels = [];
const fixture = await createOpenAiModelFixture({ const fixture = await createOpenAiModelFixture({
directory: lab, directory: lab,
prefix: "copilot", prefix: "copilot",
onRequest: async ({ body, tools, requestIndex }) => { onRequest: async ({ body, tools, requestIndex }) => {
assert.equal(
body.model,
wireModel,
"Copilot must send the selected wire model",
);
wireModels.push(body.model);
// Auxiliary requests without tools cannot execute the analyst's workflow. // Auxiliary requests without tools cannot execute the analyst's workflow.
if (tools.length === 0) if (tools.length === 0)
return { role: "assistant", content: "REA fixture title" }; return { role: "assistant", content: "REA fixture title" };
@@ -326,16 +311,6 @@ try {
assert.deepEqual(JSON.parse(repeated.stdout).appliedActions, []); assert.deepEqual(JSON.parse(repeated.stdout).appliedActions, []);
assert.equal(await readFile(configPath, "utf8"), configured); assert.equal(await readFile(configPath, "utf8"), configured);
assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original); assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original);
if (inputSchemaProfile === "compact") {
// Select the existing presentation in this disposable native-client profile.
const customized = JSON.parse(configured);
assert(customized.mcpServers.rea);
customized.mcpServers.rea.env = {
...customized.mcpServers.rea.env,
REA_MCP_INPUT_SCHEMA_PROFILE: inputSchemaProfile,
};
await writeFile(configPath, JSON.stringify(customized, null, 2));
}
const installedSkill = await readFile( const installedSkill = await readFile(
join(skillDirectory, "SKILL.md"), join(skillDirectory, "SKILL.md"),
"utf8", "utf8",
@@ -433,8 +408,6 @@ try {
clientVersion, clientVersion,
mode, mode,
model, model,
wireModel,
inputSchemaProfile,
passed: failure === undefined, passed: failure === undefined,
artifacts: lab, artifacts: lab,
runtimeRoot, runtimeRoot,
@@ -448,9 +421,7 @@ try {
skillDiscovery: skillDiscovery:
"native COPILOT_HOME MCP configuration and caller-added isolated skill directory; default OS-home discovery unverified", "native COPILOT_HOME MCP configuration and caller-added isolated skill directory; default OS-home discovery unverified",
capacityBoundary: capacityBoundary:
inputSchemaProfile === "compact" "gpt-4.1 model configuration blocks the full catalog before HTTP even with a ten-million-token requested prompt override; gpt-5.4 is the verified local fixture configuration",
? "Explicit compact profile: complete inventory and canonical server validation; advertised schemas may be reduced. This does not fix the full-profile gpt-4.1 context boundary."
: "Complete canonical input and output schemas. Client model ID and wire model are recorded separately; effective token capacity and live provider compatibility remain unknown.",
catalogSize, catalogSize,
schemasChecked, schemasChecked,
spillReadRequested, spillReadRequested,
@@ -458,7 +429,6 @@ try {
skillLoaded, skillLoaded,
artifactDigest, artifactDigest,
finalSeen, finalSeen,
wireModels,
requests: fixture.requests, requests: fixture.requests,
probes: fixture.probes, probes: fixture.probes,
fixtureFailure: fixture.failure, fixtureFailure: fixture.failure,
+2 -2
View File
@@ -40,9 +40,9 @@ await Promise.all([
]); ]);
const environment = { const environment = {
PATH: process.env.PATH ?? "", PATH: process.env.PATH ?? "",
HOME: account,
USERPROFILE: account, USERPROFILE: account,
DSH_HOME: profile, DSH_HOME: profile,
DSH_AGENTS_HOME: join(account, ".agents"),
OPENAI_API_KEY: "local-fixture-key", OPENAI_API_KEY: "local-fixture-key",
REA_PROCESS_RUN_ID: verifier.run_id, REA_PROCESS_RUN_ID: verifier.run_id,
}; };
@@ -328,7 +328,7 @@ try {
"loopback custom OpenAI-compatible adapter; no live DeepSeek provider claim", "loopback custom OpenAI-compatible adapter; no live DeepSeek provider claim",
modelContextWindow: 1000000, modelContextWindow: 1000000,
skillDiscovery: skillDiscovery:
"native HOME shared personal directory; isolated project root", "native DSH_AGENTS_HOME personal directory; isolated project root",
catalogSize: catalog?.length, catalogSize: catalog?.length,
patternCount, patternCount,
uniquePatterns: patterns.size, uniquePatterns: patterns.size,
+8 -2
View File
@@ -54,7 +54,6 @@ await writeFile(
); );
const environment = { const environment = {
PATH: process.env.PATH ?? "", PATH: process.env.PATH ?? "",
HOME: account,
USERPROFILE: account, USERPROFILE: account,
[client === "pi" ? "PI_CODING_AGENT_DIR" : "HERMES_HOME"]: profile, [client === "pi" ? "PI_CODING_AGENT_DIR" : "HERMES_HOME"]: profile,
PI_OFFLINE: "1", PI_OFFLINE: "1",
@@ -72,6 +71,13 @@ const original =
}) })
: "# Keep caller settings.\nmcp_servers:\n other:\n command: unrelated-server\n enabled: false\n"; : "# Keep caller settings.\nmcp_servers:\n other:\n command: unrelated-server\n enabled: false\n";
await writeFile(configPath, original); await writeFile(configPath, original);
if (client === "pi") {
// The CLI fixture account and native os.homedir() differ; use Pi's caller-owned skill setting.
await writeFile(
join(profile, "settings.json"),
JSON.stringify({ skills: [join(account, ".agents", "skills")] }),
);
}
const clientVersion = ( const clientVersion = (
await exec(command, ["--version"], { env: environment, timeout: 30_000 }) await exec(command, ["--version"], { env: environment, timeout: 30_000 })
).stdout.trim(); ).stdout.trim();
@@ -442,7 +448,7 @@ try {
"loopback deterministic OpenAI-compatible endpoint; no live provider claim", "loopback deterministic OpenAI-compatible endpoint; no live provider claim",
skillDiscovery: skillDiscovery:
client === "pi" client === "pi"
? "native HOME shared personal directory; no explicit skill setting" ? "caller-configured isolated shared directory; default OS home discovery unverified"
: "native HERMES_HOME personal directory", : "native HERMES_HOME personal directory",
stickyHermes, stickyHermes,
catalogSize: catalog?.length, catalogSize: catalog?.length,
+5 -3
View File
@@ -45,7 +45,6 @@ await Promise.all([
]); ]);
const baseEnvironment = { const baseEnvironment = {
PATH: process.env.PATH ?? "", PATH: process.env.PATH ?? "",
HOME: account,
USERPROFILE: account, USERPROFILE: account,
QWEN_HOME: profile, QWEN_HOME: profile,
QWEN_RUNTIME_DIR: runtime, QWEN_RUNTIME_DIR: runtime,
@@ -72,6 +71,9 @@ const original = JSON.stringify(
}, },
telemetry: { enabled: false }, telemetry: { enabled: false },
model: { name: "rea-client-fixture" }, model: { name: "rea-client-fixture" },
// Qwen uses os.homedir(), while REA's isolated CLI account uses USERPROFILE.
// This caller-owned setting lets the real client load the REA-installed bundle.
skills: { directories: [sharedSkills] },
mcpServers: { other: { command: "unrelated-server", disabled: true } }, mcpServers: { other: { command: "unrelated-server", disabled: true } },
}, },
null, null,
@@ -110,7 +112,7 @@ const configured = JSON.parse(
await readFile(join(profile, "settings.json"), "utf8"), await readFile(join(profile, "settings.json"), "utf8"),
); );
assert.deepEqual(configured.model, { name: "rea-client-fixture" }); assert.deepEqual(configured.model, { name: "rea-client-fixture" });
assert.equal(configured.skills, undefined); assert.deepEqual(configured.skills, { directories: [sharedSkills] });
assert.deepEqual(configured.mcpServers.other, { assert.deepEqual(configured.mcpServers.other, {
command: "unrelated-server", command: "unrelated-server",
disabled: true, disabled: true,
@@ -382,7 +384,7 @@ try {
modelFixture: modelFixture:
"loopback deterministic OpenAI-compatible endpoint; no live model-provider claim", "loopback deterministic OpenAI-compatible endpoint; no live model-provider claim",
skillDiscovery: skillDiscovery:
"native HOME shared personal directory; no explicit skill setting", "caller-configured isolated shared directory; default OS home discovery unverified",
catalogSize: TOOL_CONTRACTS.length, catalogSize: TOOL_CONTRACTS.length,
requests: fixture.requests, requests: fixture.requests,
probes: fixture.probes, probes: fixture.probes,
+1 -109
View File
@@ -2,7 +2,7 @@
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { execFile } from "node:child_process"; import { execFile } from "node:child_process";
import { createHash } from "node:crypto"; import { createHash } from "node:crypto";
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; import { mkdtemp, mkdir, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { join } from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
@@ -60,9 +60,6 @@ const env = {
JAVA_HOME: installation.javaHome, JAVA_HOME: installation.javaHome,
HOPPER_LAUNCHER_PATH: "/rea-unconfigured-deep-provider/hopper", HOPPER_LAUNCHER_PATH: "/rea-unconfigured-deep-provider/hopper",
}; };
for (const setting of ["GHIDRA_HEADLESS_MAXMEM", "GHIDRA_MAXMEM"]) {
if (process.env[setting] !== undefined) env[setting] = process.env[setting];
}
const reports = []; const reports = [];
let primaryFailure; let primaryFailure;
try { try {
@@ -230,103 +227,6 @@ try {
fixture.sha256, fixture.sha256,
"Analysis modified the source executable", "Analysis modified the source executable",
); );
// Carved firmware keeps all program headers/load bytes but loses sections.
const read16 = (offset) =>
byteOrder === "little"
? sourceBytes.readUInt16LE(offset)
: sourceBytes.readUInt16BE(offset);
const read32 = (offset) =>
byteOrder === "little"
? sourceBytes.readUInt32LE(offset)
: sourceBytes.readUInt32BE(offset);
const sectionOffset = read32(32);
const cut = sectionOffset - 1;
const programOffset = read32(28);
const programCount = read16(44);
assert.equal(read16(42), 32);
assert.ok(cut >= programOffset + programCount * 32);
for (let i = 0; i < programCount; i += 1) {
const offset = programOffset + i * 32;
assert.ok(read32(offset + 4) + read32(offset + 16) <= cut);
}
const carvedBytes = sourceBytes.subarray(0, cut);
const carvedPath = join(workspace, `${byteOrder}-carved.elf`);
const carvedSha256 = createHash("sha256")
.update(carvedBytes)
.digest("hex");
await writeFile(carvedPath, carvedBytes);
const carvedResolved = await parseBinaryTarget(carvedPath);
if (!carvedResolved.ok) throw carvedResolved.error;
assert.deepEqual(
carvedResolved.value.mips.abiFlags,
independent.mips.abiFlags,
);
assert.match(
carvedResolved.value.mips.limitations.join(" "),
/section header table/u,
);
const carvedTarget = await call("open_binary", {
path: carvedPath,
provider_id: "ghidra",
});
assert.equal(carvedTarget.sha256, carvedSha256);
const carvedImage = await call("inspect_native_load_image");
assertMipsLoadedImage(carvedImage.observations, byteOrder, carvedSha256);
const carvedProcedures = await call("list_procedures");
assert.ok(
carvedProcedures.some(
(p) =>
BigInt(p.address) === BigInt(independent.symbols.rea_mips_entry),
),
);
const carvedDossier = functionDossierSchema.parse(
await call("analyze_function", {
procedure: independent.symbols.rea_mips_entry,
}),
);
assert.ok(carvedDossier.pseudocode.trim().length > 0);
const carvedSnapshotPath = join(
workspace,
`${byteOrder}-carved.snapshot.json`,
);
await call("close_binary", { snapshot_path: carvedSnapshotPath });
const carvedSnapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(carvedSnapshotPath, "utf8")),
);
assert.equal(carvedSnapshot.target.sha256, carvedSha256);
assert.ok(
carvedSnapshot.evidence_bundle.records.some((record) =>
parseEvidence(record).limitations.some((item) =>
item.includes("section header table"),
),
),
);
const carvedCli = await execute(
process.execPath,
[
entrypoint,
"function",
carvedPath,
independent.symbols.rea_mips_entry,
"--provider",
"ghidra",
"--json",
],
{ env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 },
);
const carvedEvidence = parseEvidence(JSON.parse(carvedCli.stdout));
assert.equal(carvedEvidence.subject?.digest.sha256, carvedSha256);
assert.ok(
carvedEvidence.limitations.some((item) =>
item.includes("section header table"),
),
);
assert.equal(
functionDossierSchema.parse(carvedEvidence.normalized_result).procedure
.address,
carvedDossier.procedure.address,
);
assert.deepEqual(await readFile(carvedPath), carvedBytes);
reports.push({ reports.push({
...fixture, ...fixture,
independent_reader: independent, independent_reader: independent,
@@ -341,14 +241,6 @@ try {
"CLI/MCP and snapshot identity", "CLI/MCP and snapshot identity",
], ],
procedures: procedures.length, procedures: procedures.length,
carved: {
sha256: carvedSha256,
source_bytes: carvedBytes.length,
section_header_offset: sectionOffset,
effective_language: carvedImage.observations.language_id,
limitations: carvedResolved.value.mips.limitations,
procedures: carvedProcedures.length,
},
status: "passed", status: "passed",
}); });
} catch (cause) { } catch (cause) {
@@ -3,7 +3,7 @@
import { realpath } from "node:fs/promises"; import { realpath } from "node:fs/promises";
import { parseArgs } from "node:util"; import { parseArgs } from "node:util";
import { analyzeJavaScriptApplicationWithOwnedScope } from "../../lib/javascript-application-analysis.mjs"; import { analyzeJavaScriptApplication } from "../../../dist/application/javascript/JavaScriptApplicationService.js";
import { import {
compareApplicationVersionsEvidence, compareApplicationVersionsEvidence,
compareJavaScriptExportShapesEvidence, compareJavaScriptExportShapesEvidence,
@@ -199,5 +199,9 @@ function selectorKey(selector) {
} }
async function analyze(path) { async function analyze(path) {
return analyzeJavaScriptApplicationWithOwnedScope({ input_path: path }); const result = await analyzeJavaScriptApplication({
input_path: path,
});
if (!result.ok) throw result.error;
return result.value;
} }
@@ -3,8 +3,8 @@
import { readFile, realpath } from "node:fs/promises"; import { readFile, realpath } from "node:fs/promises";
import { parseArgs } from "node:util"; import { parseArgs } from "node:util";
import { analyzeJavaScriptApplication } from "../../../dist/application/javascript/JavaScriptApplicationService.js";
import { reconcileJavaScriptRuntimeEvidence } from "../../../dist/application/javascript/JavaScriptRuntimeReconciliationService.js"; import { reconcileJavaScriptRuntimeEvidence } from "../../../dist/application/javascript/JavaScriptRuntimeReconciliationService.js";
import { analyzeJavaScriptApplicationWithOwnedScope } from "../../lib/javascript-application-analysis.mjs";
import { import {
completeVerifierRun, completeVerifierRun,
createVerifierRun, createVerifierRun,
@@ -44,13 +44,14 @@ const canonicalLayers = await Promise.all(
); );
const staticLayers = []; const staticLayers = [];
for (const layer of canonicalLayers) { for (const layer of canonicalLayers) {
const analyzed = await analyzeJavaScriptApplicationWithOwnedScope({ const analyzed = await analyzeJavaScriptApplication({
input_path: layer.path, input_path: layer.path,
format: "auto", format: "auto",
}); });
if (!analyzed.ok) throw analyzed.error;
staticLayers.push({ staticLayers.push({
role: layer.role, role: layer.role,
analysis: analyzed, analysis: analyzed.value,
runtime_mappings: [], runtime_mappings: [],
}); });
} }
+46 -348
View File
@@ -1,6 +1,6 @@
import { execFile } from "node:child_process"; import { execFile } from "node:child_process";
import { createHash } from "node:crypto"; import { createHash } from "node:crypto";
import { constants as fsConstants } from "node:fs"; import { createReadStream, constants as fsConstants } from "node:fs";
import { access, lstat, mkdir, rm, stat, writeFile } from "node:fs/promises"; import { access, lstat, mkdir, rm, stat, writeFile } from "node:fs/promises";
import { basename, join, resolve } from "node:path"; import { basename, join, resolve } from "node:path";
@@ -24,11 +24,8 @@ import {
AnalysisTimeoutError, AnalysisTimeoutError,
AnalysisUnsupportedTargetError, AnalysisUnsupportedTargetError,
} from "../domain/analysisErrorCore.js"; } from "../domain/analysisErrorCore.js";
import { AnalysisError } from "../domain/analysisErrorBase.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
import { hashStableFile } from "../filesystem/StableFileHash.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import { import {
execFileOutput, execFileOutput,
execFileOutputFailure, execFileOutputFailure,
@@ -100,18 +97,11 @@ const STDERR_EXCERPT_BYTES = 512;
const UNPARSED_LINES_REPORTED = 10; const UNPARSED_LINES_REPORTED = 10;
const STOP_GRACE_MS = 5_000; const STOP_GRACE_MS = 5_000;
const hashAdbFile = async ( /** Stream a local file digest without retaining large files in memory. */
path: string, const sha256File = async (path: string): Promise<string> => {
operation: string, const hash = createHash("sha256");
signal: AbortSignal | undefined, for await (const chunk of createReadStream(path)) hash.update(chunk);
) => { return hash.digest("hex");
try {
return await hashStableFile(path, signal);
} catch (cause: unknown) {
if (signal?.aborted === true)
throw new AnalysisCancelledError(operation, { cause });
throw cause;
}
}; };
/** adb joins shell operands into a remote POSIX command; quote at that boundary. */ /** adb joins shell operands into a remote POSIX command; quote at that boundary. */
@@ -345,48 +335,6 @@ export interface AdbProviderOptions {
readonly environment?: Readonly<Record<string, string | undefined>>; readonly environment?: Readonly<Record<string, string | undefined>>;
} }
interface PendingPackageDirectoryCleanup {
readonly path: string;
readonly identity: PackageDirectoryIdentity | undefined;
readonly primary: AnalysisError;
readonly remove: () => Promise<void>;
}
interface PackageDirectoryIdentity {
readonly dev: number;
readonly ino: number;
readonly mode: number;
}
const removeOwnedPackageDirectory = async (
path: string,
identity: PackageDirectoryIdentity | undefined,
): Promise<void> => {
if (identity === undefined)
throw new Error(
`Cannot verify the created package directory identity; refusing to remove ${path}`,
);
let current: Awaited<ReturnType<typeof lstat>>;
try {
current = await lstat(path);
} catch (cause: unknown) {
if (cause instanceof Error && Reflect.get(cause, "code") === "ENOENT")
return;
throw cause;
}
if (
!current.isDirectory() ||
current.isSymbolicLink() ||
current.dev !== identity.dev ||
current.ino !== identity.ino ||
current.mode !== identity.mode
)
throw new Error(
`The created package directory was replaced or changed; refusing to remove ${path}`,
);
await rm(path, { recursive: true, force: true });
};
/** /**
* ADB device inspection and acquisition over a caller-selected adb binary. * ADB device inspection and acquisition over a caller-selected adb binary.
* *
@@ -398,12 +346,6 @@ const removeOwnedPackageDirectory = async (
*/ */
export class AdbProvider { export class AdbProvider {
private readonly environment: Readonly<Record<string, string | undefined>>; private readonly environment: Readonly<Record<string, string | undefined>>;
readonly #pendingPackageDirectoryCleanup =
new Set<PendingPackageDirectoryCleanup>();
readonly #activeOperations = new Set<Promise<void>>();
#closing = false;
#closePromise: Promise<void> | undefined;
#packageCleanupPromise: Promise<void> | undefined;
constructor(options: AdbProviderOptions = {}) { constructor(options: AdbProviderOptions = {}) {
this.environment = options.environment ?? process.env; this.environment = options.environment ?? process.env;
@@ -436,121 +378,12 @@ export class AdbProvider {
} }
} }
/** Retry removal of package directories retained after failed rollback. */ /** The provider holds no long-lived resources; the adb server is caller-owned. */
close(): Promise<void> { async close(): Promise<void> {}
this.#closing = true;
if (this.#closePromise !== undefined) return this.#closePromise;
const closing = (async () => {
await Promise.all(this.#activeOperations);
await this.#retryPackageDirectoryCleanup();
})().finally(() => {
if (this.#closePromise === closing) this.#closePromise = undefined;
});
this.#closePromise = closing;
return this.#closePromise;
}
#retryPackageDirectoryCleanup(): Promise<void> {
if (this.#packageCleanupPromise !== undefined)
return this.#packageCleanupPromise;
const cleanup = this.#performPackageDirectoryCleanup().finally(() => {
if (this.#packageCleanupPromise === cleanup)
this.#packageCleanupPromise = undefined;
});
this.#packageCleanupPromise = cleanup;
return cleanup;
}
async #performPackageDirectoryCleanup(): Promise<void> {
const failures: AnalysisError[] = [];
for (const owner of this.#pendingPackageDirectoryCleanup) {
try {
await owner.remove();
this.#pendingPackageDirectoryCleanup.delete(owner);
} catch (cause: unknown) {
const cleanup = new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[owner.path],
{
reason: cause instanceof Error ? cause.message : String(cause),
},
{ operation: "pull_adb_package", cause },
);
failures.push(
analysisErrorWithCleanupFailure(
owner.primary,
cleanup,
"pull_adb_package",
),
);
}
}
if (failures.length === 1) throw failures[0];
if (failures.length > 1)
throw new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[...this.#pendingPackageDirectoryCleanup].map(({ path }) => path),
{
reason:
"Failed package output directories remain after cleanup retry",
failures: failures.map(({ message, cleanupResources }) => ({
message,
resources: [...cleanupResources],
})),
},
{
operation: "pull_adb_package",
cause: new AggregateError(
failures,
"Multiple ADB package rollback cleanups failed",
),
},
);
}
async execute( async execute(
request: AdbRequest, request: AdbRequest,
options?: ExecutionOptions, options?: ExecutionOptions,
): Promise<Result<AnalysisExecution, AnalysisError>> {
if (this.#closing)
return err(new AnalysisCancelledError(request.operation));
let release!: () => void;
const active = new Promise<void>((resolve) => {
release = resolve;
});
this.#activeOperations.add(active);
try {
if (this.#pendingPackageDirectoryCleanup.size > 0) {
try {
await this.#retryPackageDirectoryCleanup();
} catch (cause: unknown) {
return err(
cause instanceof AnalysisError
? cause
: new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[...this.#pendingPackageDirectoryCleanup].map(
({ path }) => path,
),
{
reason:
cause instanceof Error ? cause.message : String(cause),
},
{ operation: request.operation, cause },
),
);
}
}
return await this.#executeOperation(request, options);
} finally {
this.#activeOperations.delete(active);
release();
}
}
async #executeOperation(
request: AdbRequest,
options?: ExecutionOptions,
): Promise<Result<AnalysisExecution, AnalysisError>> { ): Promise<Result<AnalysisExecution, AnalysisError>> {
const signal = options?.signal; const signal = options?.signal;
if (signal?.aborted === true) if (signal?.aborted === true)
@@ -1177,11 +1010,9 @@ export class AdbProvider {
); );
if (!directory.ok) return directory; if (!directory.ok) return directory;
const packageDirectory = join(directory.value, packageName); const packageDirectory = join(directory.value, packageName);
let packageDirectoryCreated = false;
let packageDirectoryIdentity: PackageDirectoryIdentity | undefined;
try { try {
if ( if (
await lstat(packageDirectory).then( await stat(packageDirectory).then(
() => true, () => true,
() => false, () => false,
) )
@@ -1190,44 +1021,14 @@ export class AdbProvider {
`The package output directory already exists: ${packageDirectory}`, `The package output directory already exists: ${packageDirectory}`,
); );
await mkdir(packageDirectory); await mkdir(packageDirectory);
packageDirectoryCreated = true;
const created = await lstat(packageDirectory);
if (!created.isDirectory() || created.isSymbolicLink())
throw new Error(
`The created package output is not a regular directory: ${packageDirectory}`,
);
packageDirectoryIdentity = {
dev: created.dev,
ino: created.ino,
mode: created.mode,
};
} catch (cause) { } catch (cause) {
const primary = new AnalysisOutputError( return err(
operation, new AnalysisOutputError(
cause instanceof Error ? cause.message : String(cause), operation,
{ cause }, cause instanceof Error ? cause.message : String(cause),
{ cause },
),
); );
if (!packageDirectoryCreated) return err(primary);
const owner: PendingPackageDirectoryCleanup = {
path: packageDirectory,
identity: packageDirectoryIdentity,
primary,
remove: () =>
removeOwnedPackageDirectory(
packageDirectory,
packageDirectoryIdentity,
),
};
this.#pendingPackageDirectoryCleanup.add(owner);
const cleanup = new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[packageDirectory],
{
reason: "Could not establish an owned package directory identity",
},
{ operation, cause },
);
return err(analysisErrorWithCleanupFailure(primary, cleanup, operation));
} }
const artifacts: { const artifacts: {
device_path: string; device_path: string;
@@ -1243,37 +1044,6 @@ export class AdbProvider {
stage: "resolve_paths" | "pull" | "digest"; stage: "resolve_paths" | "pull" | "digest";
message: string; message: string;
}[] = []; }[] = [];
const cancelledPackagePull = (
error: AnalysisCancelledError,
localPath: string,
devicePath: string,
stage: "pull" | "digest",
) => {
error.retainPartialObservation({
provider_id: "adb",
operation,
result: {
client,
serial,
package_name: packageName,
output_directory: directory.value,
artifacts,
failures: [
...failures,
{
device_path: devicePath,
stage,
message:
stage === "pull"
? `Pull cancelled; any incomplete file is retained at ${localPath}. Remaining APKs were not attempted.`
: `Digesting the pulled file was cancelled; the file is retained at ${localPath}. Remaining APKs were not attempted.`,
},
],
coverage: "partial",
},
});
return err(error);
};
const seenFileNames = new Set<string>(); const seenFileNames = new Set<string>();
for (const entry of parsed.paths) { for (const entry of parsed.paths) {
if (seenFileNames.has(entry.file_name)) { if (seenFileNames.has(entry.file_name)) {
@@ -1295,13 +1065,7 @@ export class AdbProvider {
signal, signal,
); );
if (!pulled.ok) { if (!pulled.ok) {
if (pulled.error instanceof AnalysisCancelledError) if (pulled.error instanceof AnalysisCancelledError) return pulled;
return cancelledPackagePull(
pulled.error,
localPath,
entry.device_path,
"pull",
);
failures.push({ failures.push({
device_path: entry.device_path, device_path: entry.device_path,
stage: "pull", stage: "pull",
@@ -1310,24 +1074,18 @@ export class AdbProvider {
continue; continue;
} }
try { try {
const digest = await hashAdbFile(localPath, operation, signal); const info = await stat(localPath);
const sha256 = await sha256File(localPath);
artifacts.push({ artifacts.push({
device_path: entry.device_path, device_path: entry.device_path,
file_name: entry.file_name, file_name: entry.file_name,
local_path: localPath, local_path: localPath,
bytes: digest.bytes, bytes: info.size,
sha256: digest.sha256, sha256,
role: entry.role, role: entry.role,
role_basis: "file_name", role_basis: "file_name",
}); });
} catch (cause: unknown) { } catch {
if (cause instanceof AnalysisCancelledError)
return cancelledPackagePull(
cause,
localPath,
entry.device_path,
"digest",
);
failures.push({ failures.push({
device_path: entry.device_path, device_path: entry.device_path,
stage: "digest", stage: "digest",
@@ -1336,36 +1094,14 @@ export class AdbProvider {
} }
} }
if (artifacts.length === 0 && failures.length > 0) { if (artifacts.length === 0 && failures.length > 0) {
const primary = new AnalysisProtocolError( await rm(packageDirectory, { recursive: true, force: true }).catch(
`Every APK pull failed for ${packageName} on ${serial}: ${failures[0]!.message}`, () => undefined,
);
return err(
new AnalysisProtocolError(
`Every APK pull failed for ${packageName} on ${serial}: ${failures[0]!.message}`,
),
); );
const owner: PendingPackageDirectoryCleanup = {
path: packageDirectory,
identity: packageDirectoryIdentity,
primary,
remove: () =>
removeOwnedPackageDirectory(
packageDirectory,
packageDirectoryIdentity,
),
};
try {
await owner.remove();
} catch (cause: unknown) {
this.#pendingPackageDirectoryCleanup.add(owner);
const cleanup = new ProviderCleanupError(
ADB_PROVIDER_IDENTITY.id,
[packageDirectory],
{
reason: cause instanceof Error ? cause.message : String(cause),
},
{ operation: operation, cause },
);
return err(
analysisErrorWithCleanupFailure(primary, cleanup, operation),
);
}
return err(primary);
} }
return ok( return ok(
createAnalysisExecution( createAnalysisExecution(
@@ -1420,7 +1156,8 @@ export class AdbProvider {
); );
if (!pulled.ok) return pulled; if (!pulled.ok) return pulled;
try { try {
const digest = await hashAdbFile(localPath, operation, signal); const info = await stat(localPath);
const sha256 = await sha256File(localPath);
return ok( return ok(
createAnalysisExecution( createAnalysisExecution(
{ {
@@ -1428,8 +1165,8 @@ export class AdbProvider {
serial: input.serial, serial: input.serial,
device_path: input.device_path, device_path: input.device_path,
local_path: localPath, local_path: localPath,
bytes: digest.bytes, bytes: info.size,
sha256: digest.sha256, sha256,
}, },
provider, provider,
{ {
@@ -1440,20 +1177,6 @@ export class AdbProvider {
), ),
); );
} catch (cause) { } catch (cause) {
if (cause instanceof AnalysisCancelledError) {
cause.retainPartialObservation({
provider_id: "adb",
operation,
result: {
client,
serial: input.serial,
local_path: localPath,
device_path: input.device_path,
digest_status: "unknown",
},
});
return err(cause);
}
return err( return err(
new AnalysisOutputError( new AnalysisOutputError(
operation, operation,
@@ -1488,13 +1211,7 @@ export class AdbProvider {
cause: new Error(`The local file does not exist: ${localPath}`), cause: new Error(`The local file does not exist: ${localPath}`),
}), }),
); );
let digest: Awaited<ReturnType<typeof hashAdbFile>>; const sha256 = await sha256File(localPath);
try {
digest = await hashAdbFile(localPath, operation, signal);
} catch (cause: unknown) {
if (cause instanceof AnalysisCancelledError) return err(cause);
throw cause;
}
const existsProbe = await runAdb( const existsProbe = await runAdb(
operation, operation,
input.device_path, input.device_path,
@@ -1540,7 +1257,7 @@ export class AdbProvider {
); );
if (deviceDigest.ok) { if (deviceDigest.ok) {
const digestMatch = /^([a-f0-9]{64})\s/u.exec(deviceDigest.value.stdout); const digestMatch = /^([a-f0-9]{64})\s/u.exec(deviceDigest.value.stdout);
if (digestMatch !== null && digestMatch[1] !== digest.sha256) if (digestMatch !== null && digestMatch[1] !== sha256)
return err( return err(
new AnalysisArtifactChangedError( new AnalysisArtifactChangedError(
operation, operation,
@@ -1555,8 +1272,8 @@ export class AdbProvider {
serial: input.serial, serial: input.serial,
local_path: localPath, local_path: localPath,
device_path: input.device_path, device_path: input.device_path,
bytes: digest.bytes, bytes: info.size,
sha256: digest.sha256, sha256,
device_sha256: digestMatch?.[1] ?? null, device_sha256: digestMatch?.[1] ?? null,
overwritten: deviceExists, overwritten: deviceExists,
device_digest_source: "device_sha256sum", device_digest_source: "device_sha256sum",
@@ -1577,8 +1294,8 @@ export class AdbProvider {
serial: input.serial, serial: input.serial,
local_path: localPath, local_path: localPath,
device_path: input.device_path, device_path: input.device_path,
bytes: digest.bytes, bytes: info.size,
sha256: digest.sha256, sha256,
device_sha256: null, device_sha256: null,
overwritten: deviceExists, overwritten: deviceExists,
device_digest_source: "unavailable", device_digest_source: "unavailable",
@@ -1617,13 +1334,7 @@ export class AdbProvider {
cause: new Error(`The local APK does not exist: ${apkPath}`), cause: new Error(`The local APK does not exist: ${apkPath}`),
}), }),
); );
let digest: Awaited<ReturnType<typeof hashAdbFile>>; const sha256 = await sha256File(apkPath);
try {
digest = await hashAdbFile(apkPath, operation, signal);
} catch (cause: unknown) {
if (cause instanceof AnalysisCancelledError) return err(cause);
throw cause;
}
const installed = await runAdb( const installed = await runAdb(
operation, operation,
input.serial, input.serial,
@@ -1652,8 +1363,8 @@ export class AdbProvider {
client, client,
serial: input.serial, serial: input.serial,
apk_path: apkPath, apk_path: apkPath,
bytes: digest.bytes, bytes: info.size,
sha256: digest.sha256, sha256,
replaced: input.replace, replaced: input.replace,
}, },
provider, provider,
@@ -2023,15 +1734,16 @@ export class AdbProvider {
), ),
); );
try { try {
const digest = await hashAdbFile(reportedPath, operation, signal); const info = await stat(reportedPath);
const sha256 = await sha256File(reportedPath);
return ok( return ok(
createAnalysisExecution( createAnalysisExecution(
{ {
client, client,
serial: input.serial, serial: input.serial,
local_path: reportedPath, local_path: reportedPath,
bytes: digest.bytes, bytes: info.size,
sha256: digest.sha256, sha256,
adb_reported_path: reportedPath, adb_reported_path: reportedPath,
}, },
provider, provider,
@@ -2043,20 +1755,6 @@ export class AdbProvider {
), ),
); );
} catch (cause) { } catch (cause) {
if (cause instanceof AnalysisCancelledError) {
cause.retainPartialObservation({
provider_id: "adb",
operation,
result: {
client,
serial: input.serial,
local_path: reportedPath,
adb_reported_path: reportedPath,
digest_status: "unknown",
},
});
return err(cause);
}
return err( return err(
new AnalysisOutputError( new AnalysisOutputError(
operation, operation,
+21 -129
View File
@@ -1,43 +1,40 @@
import { constants, createReadStream } from "node:fs";
import { chmod, copyFile } from "node:fs/promises";
import { createHash } from "node:crypto"; import { createHash } from "node:crypto";
import type { ReadStream, Stats, WriteStream } from "node:fs";
import { chmod, open, rm, stat, type FileHandle } from "node:fs/promises";
import { join } from "node:path"; import { join } from "node:path";
import { Transform, type TransformCallback } from "node:stream";
import { pipeline } from "node:stream/promises";
import { import {
AnalysisInputError, AnalysisInputError,
AnalysisCapabilityUnavailableError, AnalysisCapabilityUnavailableError,
} from "../domain/analysisErrorCore.js"; } from "../domain/analysisErrorCore.js";
import {
openRegularFile,
sameRegularFileState,
} from "../filesystem/RegularFile.js";
import type { AndroidOperation } from "../domain/android/androidAnalysis.js"; import type { AndroidOperation } from "../domain/android/androidAnalysis.js";
/** Stream a local file digest without retaining the APK or JAR in memory. */
export const hashAndroidFile = async (path: string): Promise<string> => {
const hash = createHash("sha256");
for await (const chunk of createReadStream(path)) hash.update(chunk);
return hash.digest("hex");
};
/** Bind decompiler input to the admitted artifact, even if the original changes. */ /** Bind decompiler input to the admitted artifact, even if the original changes. */
export const snapshotAndroidTarget = async ( export const snapshotAndroidTarget = async (
path: string, path: string,
sha256: string, sha256: string,
root: string, root: string,
operation: AndroidOperation, operation: AndroidOperation,
signal?: AbortSignal,
): Promise<string> => { ): Promise<string> => {
const snapshot = join(root, "target.apk"); const snapshot = join(root, "target.apk");
try { await copyFile(path, snapshot, constants.COPYFILE_EXCL);
await copyAndroidSnapshot(path, snapshot, sha256, signal); await chmod(snapshot, 0o400);
signal?.throwIfAborted(); if ((await hashAndroidFile(snapshot)) !== sha256)
await chmod(snapshot, 0o400); throw new AnalysisInputError(operation, undefined, [
return snapshot;
} catch (cause: unknown) {
if (!(cause instanceof AndroidSnapshotChangedError)) throw cause;
throw new AnalysisInputError(operation, { cause }, [
{ {
path: ["path"], path: ["path"],
reason: "invalid_value", reason: "invalid_value",
message: `APK bytes changed after admission at ${path}; retry against a stable file.`, message:
"APK bytes changed after admission; retry against a stable file.",
}, },
]); ]);
} return snapshot;
}; };
/** Fingerprint the actual immutable engine bytes executed for this observation. */ /** Fingerprint the actual immutable engine bytes executed for this observation. */
@@ -46,120 +43,15 @@ export const snapshotAndroidEngine = async (
sha256: string, sha256: string,
root: string, root: string,
operation: AndroidOperation, operation: AndroidOperation,
signal?: AbortSignal,
): Promise<{ path: string; sha256: string }> => { ): Promise<{ path: string; sha256: string }> => {
const path = join(root, "engine.jar"); const path = join(root, "engine.jar");
try { await copyFile(source, path, constants.COPYFILE_EXCL);
const observed = await copyAndroidSnapshot(source, path, sha256, signal); await chmod(path, 0o400);
signal?.throwIfAborted(); if ((await hashAndroidFile(path)) !== sha256)
await chmod(path, 0o400);
return { path, sha256: observed };
} catch (cause: unknown) {
if (!(cause instanceof AndroidSnapshotChangedError)) throw cause;
throw new AnalysisCapabilityUnavailableError( throw new AnalysisCapabilityUnavailableError(
"jadx", "jadx",
operation, operation,
`REA_JADX_MCP_JAR bytes changed during admission at ${source}; retry with a stable engine file.`, "REA_JADX_MCP_JAR bytes changed during admission; retry with a stable engine file.",
{ cause },
); );
} return { path, sha256 };
}; };
const copyAndroidSnapshot = async (
sourcePath: string,
snapshotPath: string,
expectedSha256: string,
signal?: AbortSignal,
): Promise<string> => {
signal?.throwIfAborted();
const source = await openRegularFile(sourcePath, {
symlinks: "follow",
signal,
});
let destination: FileHandle | undefined;
let inputStream: ReadStream | undefined;
let outputStream: WriteStream | undefined;
let removeSnapshot = false;
let copiedBytes = 0;
const hash = createHash("sha256");
try {
const initial = await source.stat();
signal?.throwIfAborted();
const digest = new Transform({
transform(
chunk: Buffer,
_encoding: BufferEncoding,
callback: TransformCallback,
) {
copiedBytes += chunk.length;
if (copiedBytes > initial.size) {
callback(
new AndroidSnapshotChangedError(
`Android input grew while being copied: ${sourcePath}`,
),
);
return;
}
hash.update(chunk);
callback(null, chunk);
},
});
try {
destination = await open(snapshotPath, "wx", 0o600);
signal?.throwIfAborted();
inputStream = source.createReadStream({
start: 0,
end: initial.size,
autoClose: false,
});
outputStream = destination.createWriteStream({ autoClose: false });
await pipeline(inputStream, digest, outputStream, { signal });
signal?.throwIfAborted();
await verifySourceState(source, sourcePath, initial, copiedBytes);
const observedSha256 = hash.digest("hex");
if (observedSha256 !== expectedSha256)
throw new AndroidSnapshotChangedError(
`Android snapshot digest mismatch for ${sourcePath}`,
);
return observedSha256;
} catch (cause: unknown) {
removeSnapshot = destination !== undefined;
throw cause;
}
} finally {
// Non-auto-closing streams retain a reference to their FileHandle even
// after pipeline completion. Release those references before awaiting
// descriptor closure, after the admitted source state has been verified.
inputStream?.destroy();
outputStream?.destroy();
try {
await Promise.all([source.close(), destination?.close()]);
} finally {
if (removeSnapshot)
await rm(snapshotPath, { force: true }).catch(() => undefined);
}
}
};
const verifySourceState = async (
handle: Awaited<ReturnType<typeof openRegularFile>>,
path: string,
initial: Stats,
observedBytes: number,
): Promise<void> => {
const [opened, currentPath] = await Promise.all([handle.stat(), stat(path)]);
if (
observedBytes !== initial.size ||
!sameRegularFileState(initial, opened) ||
!sameRegularFileState(initial, currentPath)
)
throw new AndroidSnapshotChangedError(
`Android input changed while being read: ${path}`,
);
};
class AndroidSnapshotChangedError extends Error {
constructor(message: string, options?: ErrorOptions) {
super(message, options);
}
}
+29 -64
View File
@@ -52,11 +52,7 @@ export const inspectJadxAvailability = async (
signal?: AbortSignal, signal?: AbortSignal,
): Promise<ProviderAvailability> => { ): Promise<ProviderAvailability> => {
try { try {
const configuration = await readJadxConfigurationInputs( const configuration = await readJadxConfiguration(environment, signal);
environment,
signal,
);
await validateJadxConfigurationContents(configuration, environment, signal);
return { return {
status: "available", status: "available",
code: null, code: null,
@@ -81,83 +77,52 @@ export const inspectJadxAvailability = async (
} }
}; };
/** Resolve caller-selected paths and JVM options without probing their contents. */ /** Resolve caller-selected tools and preserve the same prerequisite diagnostics. */
export const resolveJadxConfigurationInputs = async ( export const resolveJadxConfiguration = async (
environment: Readonly<Record<string, string | undefined>>, environment: Readonly<Record<string, string | undefined>>,
operation: AndroidOperation, operation: AndroidOperation,
signal?: AbortSignal, signal?: AbortSignal,
): Promise<JadxConfiguration> => { ): Promise<JadxConfiguration> => {
try { try {
return await readJadxConfigurationInputs(environment, signal); return await readJadxConfiguration(environment, signal);
} catch (cause) { } catch (cause) {
if (signal?.aborted === true) throw cause; if (signal?.aborted === true) throw cause;
if (!(cause instanceof JadxConfigurationFailure)) throw cause; if (!(cause instanceof JadxConfigurationFailure)) throw cause;
throw projectConfigurationFailure(operation, cause); const output = cause.diagnostics;
throw new AnalysisCapabilityUnavailableError(
"jadx",
operation,
cause.message,
{
cause,
userMessage: cause.message,
...(typeof output.stdout === "string" &&
typeof output.stderr === "string"
? {
capturedOutput: {
stdout: output.stdout,
stderr: output.stderr,
truncated: output.output_truncated === true,
},
}
: {}),
},
);
} }
}; };
/** Probe Java and archive prerequisites before starting a new engine session. */ const readJadxConfiguration = async (
export const validateJadxConfiguration = async (
configuration: JadxConfiguration,
environment: Readonly<Record<string, string | undefined>>,
operation: AndroidOperation,
signal?: AbortSignal,
): Promise<void> => {
try {
await validateJadxConfigurationContents(configuration, environment, signal);
} catch (cause) {
if (signal?.aborted === true) throw cause;
if (!(cause instanceof JadxConfigurationFailure)) throw cause;
throw projectConfigurationFailure(operation, cause);
}
};
const readJadxConfigurationInputs = async (
environment: Readonly<Record<string, string | undefined>>, environment: Readonly<Record<string, string | undefined>>,
signal?: AbortSignal, signal?: AbortSignal,
): Promise<JadxConfiguration> => { ): Promise<JadxConfiguration> => {
signal?.throwIfAborted();
requireSupportedHost(); requireSupportedHost();
const jar = await resolveJar(environment); const jar = await resolveJar(environment);
const java = await resolveJava(environment, signal); const java = await resolveJava(environment, signal);
await requireJavaHomeExecutable(environment.JAVA_HOME, java); await requireJavaHomeExecutable(environment.JAVA_HOME, java);
signal?.throwIfAborted(); const jvmArguments = resolveJvmArguments(environment);
return { jar, java, jvmArguments: resolveJvmArguments(environment) }; await inspectJavaRuntime(java, environment, signal);
}; await requireEngineArchive(jar, signal);
return { jar, java, jvmArguments };
const validateJadxConfigurationContents = async (
configuration: JadxConfiguration,
environment: Readonly<Record<string, string | undefined>>,
signal?: AbortSignal,
): Promise<void> => {
signal?.throwIfAborted();
await inspectJavaRuntime(configuration.java, environment, signal);
await requireEngineArchive(configuration.jar, signal);
};
const projectConfigurationFailure = (
operation: AndroidOperation,
cause: JadxConfigurationFailure,
): AnalysisCapabilityUnavailableError => {
const output = cause.diagnostics;
return new AnalysisCapabilityUnavailableError(
"jadx",
operation,
cause.message,
{
cause,
userMessage: cause.message,
...(typeof output.stdout === "string" && typeof output.stderr === "string"
? {
capturedOutput: {
stdout: output.stdout,
stderr: output.stderr,
truncated: output.output_truncated === true,
},
}
: {}),
},
);
}; };
const requireEngineArchive = async ( const requireEngineArchive = async (
+28 -98
View File
@@ -21,14 +21,12 @@ import { err, ok, type Result } from "../domain/result.js";
import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js";
import { import {
snapshotAndroidEngine, snapshotAndroidEngine,
hashAndroidFile,
snapshotAndroidTarget, snapshotAndroidTarget,
} from "./AndroidTargetSnapshot.js"; } from "./AndroidTargetSnapshot.js";
import { hashStableFile } from "../filesystem/StableFileHash.js";
import { RegularFileChangedError } from "../filesystem/RegularFile.js";
import { import {
inspectJadxAvailability, inspectJadxAvailability,
resolveJadxConfigurationInputs, resolveJadxConfiguration,
validateJadxConfiguration,
} from "./JadxConfiguration.js"; } from "./JadxConfiguration.js";
import type { JadxLauncher } from "./JadxMcpTransport.js"; import type { JadxLauncher } from "./JadxMcpTransport.js";
@@ -106,50 +104,6 @@ const executionError = (context: {
}); });
}; };
const androidTargetChanged = (
operation: string,
path: string,
cause?: unknown,
): AnalysisInputError =>
new AnalysisInputError(
operation,
cause === undefined ? undefined : { cause },
[
{
path: ["path"],
reason: "invalid_value",
message: `APK bytes changed after admission at ${path}; retry against a stable file.`,
},
],
);
const androidSourceChanged = (
operation: string,
source: string,
cause?: unknown,
): AnalysisCapabilityUnavailableError =>
new AnalysisCapabilityUnavailableError(
"jadx",
operation,
`Selected JADX input bytes changed during admission at ${source}; retry with a stable installation.`,
cause === undefined ? undefined : { cause },
);
const hashAndroidSource = async (
path: string,
operation: string,
signal: AbortSignal,
): Promise<string> => {
try {
return (await hashStableFile(path, signal)).sha256;
} catch (cause: unknown) {
if (signal.aborted) throw cause;
if (cause instanceof RegularFileChangedError)
throw androidSourceChanged(operation, path, cause);
throw cause;
}
};
interface PendingCleanup { interface PendingCleanup {
session: JadxSession | undefined; session: JadxSession | undefined;
root: PrivateRuntimeRoot | undefined; root: PrivateRuntimeRoot | undefined;
@@ -193,18 +147,17 @@ const cleanup = async (
await root?.close(); await root?.close();
resources.root = undefined; resources.root = undefined;
} catch (cause) { } catch (cause) {
const error = new ProviderCleanupError( return err(
"jadx", new ProviderCleanupError(
[root?.path ?? "unknown"], "jadx",
{ [root?.path ?? "unknown"],
...diagnostics, {
reason: cause instanceof Error ? cause.message : String(cause), ...diagnostics,
}, reason: cause instanceof Error ? cause.message : String(cause),
{ cause: previousError ?? cause }, },
{ cause: previousError ?? cause },
),
); );
if (previousError?.partialObservation !== undefined)
error.retainPartialObservation(previousError.partialObservation);
return err(error);
} }
return ok(undefined); return ok(undefined);
}; };
@@ -334,33 +287,23 @@ export class JadxProvider implements AndroidAnalysisPort {
request.operation, request.operation,
`Target ${target.path} is ${target.format}; provide one standalone APK.`, `Target ${target.path} is ${target.format}; provide one standalone APK.`,
); );
const configuration = await resolveJadxConfigurationInputs( const configuration = await resolveJadxConfiguration(
this.environment, this.environment,
request.operation, request.operation,
signal, signal,
); );
signal.throwIfAborted(); signal.throwIfAborted();
const jarHash = await hashAndroidSource( const jarHash = await hashAndroidFile(configuration.jar);
configuration.jar, const bridgeHash = await hashAndroidFile(BRIDGE_SOURCE);
request.operation, if ((await hashAndroidFile(target.path)) !== target.sha256)
signal, throw new AnalysisInputError(request.operation, undefined, [
); {
const bridgeHash = await hashAndroidSource( path: ["path"],
BRIDGE_SOURCE, reason: "invalid_value",
request.operation, message:
signal, "APK bytes changed after admission; retry against a stable file.",
); },
let targetHash: string; ]);
try {
targetHash = (await hashStableFile(target.path, signal)).sha256;
} catch (cause: unknown) {
if (signal.aborted) throw cause;
if (cause instanceof RegularFileChangedError)
throw androidTargetChanged(request.operation, target.path, cause);
throw cause;
}
if (targetHash !== target.sha256)
throw androidTargetChanged(request.operation, target.path);
const key = JSON.stringify({ const key = JSON.stringify({
path: target.path, path: target.path,
sha256: target.sha256, sha256: target.sha256,
@@ -383,40 +326,27 @@ export class JadxProvider implements AndroidAnalysisPort {
} }
let retained = this.#retained; let retained = this.#retained;
if (retained === undefined) { if (retained === undefined) {
await validateJadxConfiguration(
configuration,
this.environment,
request.operation,
signal,
);
root = await PrivateRuntimeRoot.create({ prefix: "rea-android-" }); root = await PrivateRuntimeRoot.create({ prefix: "rea-android-" });
const engine = await snapshotAndroidEngine( const engine = await snapshotAndroidEngine(
configuration.jar, configuration.jar,
jarHash, jarHash,
root.path, root.path,
request.operation, request.operation,
signal,
); );
const snapshot = await snapshotAndroidTarget( const snapshot = await snapshotAndroidTarget(
target.path, target.path,
target.sha256, target.sha256,
root.path, root.path,
request.operation, request.operation,
signal,
); );
const bridge = join(root.path, "ReaJadxBridge.java"); const bridge = join(root.path, "ReaJadxBridge.java");
await copyFile(BRIDGE_SOURCE, bridge); await copyFile(BRIDGE_SOURCE, bridge);
await chmod(bridge, 0o400); await chmod(bridge, 0o400);
const copiedBridgeHash = await hashAndroidSource( if ((await hashAndroidFile(bridge)) !== bridgeHash)
bridge, throw new AnalysisCapabilityUnavailableError(
request.operation, "jadx",
signal,
);
if (copiedBridgeHash !== bridgeHash)
throw androidSourceChanged(
request.operation, request.operation,
BRIDGE_SOURCE, "REA Android bridge bytes changed during admission; retry with a stable installation.",
undefined,
); );
signal.throwIfAborted(); signal.throwIfAborted();
session = new JadxSession( session = new JadxSession(
+1 -7
View File
@@ -373,13 +373,7 @@ const readExecutableMetadata = async (
if (!abiFlags.ok) return abiFlags; if (!abiFlags.ok) return abiFlags;
return ok({ return ok({
...parsed.value, ...parsed.value,
mips: { mips: { ...parsed.value.mips, abiFlags: abiFlags.value },
...parsed.value.mips,
abiFlags: abiFlags.value.abiFlags,
...(abiFlags.value.limitations.length === 0
? {}
: { limitations: abiFlags.value.limitations }),
},
}); });
}; };
+3 -20
View File
@@ -1,22 +1,17 @@
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
import { import { readFile } from "node:fs/promises";
RegularFileCleanupFailure,
readRegularFileText,
} from "./RegularFileRead.js";
import { import {
effectiveClientServer, effectiveClientServer,
parseClientConfiguration, parseClientConfiguration,
type ClientConfigurationDocument, type ClientConfigurationDocument,
} from "./ClientConfigurationDocument.js"; } from "./ClientConfigurationDocument.js";
import type { SetupClient } from "./SupportedClients.js"; import type { SetupClient } from "./SupportedClients.js";
import { clientRegistrationEnvironment } from "./ClientRegistrationEnvironment.js";
/** A named source failure that setup can explain without losing the bad path. */ /** A named source failure that setup can explain without losing the bad path. */
export interface ClientConfigurationFileError { export interface ClientConfigurationFileError {
readonly kind: "unreadable" | "malformed"; readonly kind: "unreadable" | "malformed";
readonly path: string; readonly path: string;
readonly detail: string; readonly detail: string;
readonly cleanupFailure?: RegularFileCleanupFailure;
} }
/** Preserve named read failures and distinguish a missing source from an empty file. */ /** Preserve named read failures and distinguish a missing source from an empty file. */
@@ -24,7 +19,7 @@ export const readConfigurationText = async (
path: string, path: string,
): Promise<Result<string | undefined, ClientConfigurationFileError>> => { ): Promise<Result<string | undefined, ClientConfigurationFileError>> => {
try { try {
return ok(await readRegularFileText(path)); return ok(await readFile(path, "utf8"));
} catch (cause: unknown) { } catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT") if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
return ok(undefined); return ok(undefined);
@@ -32,9 +27,6 @@ export const readConfigurationText = async (
kind: "unreadable", kind: "unreadable",
path, path,
detail: `Configuration ${path} could not be read: ${cause instanceof Error ? cause.message : String(cause)}`, detail: `Configuration ${path} could not be read: ${cause instanceof Error ? cause.message : String(cause)}`,
...(cause instanceof RegularFileCleanupFailure
? { cleanupFailure: cause }
: {}),
}); });
} }
}; };
@@ -71,16 +63,7 @@ export const readClientConfigurationFiles = async (
for (const path of client.configPaths ?? [client.configPath]) { for (const path of client.configPaths ?? [client.configPath]) {
const document = await readConfigurationFile(path, client.format); const document = await readConfigurationFile(path, client.format);
if (!document.ok) return document; if (!document.ok) return document;
if (document.value !== undefined) { if (document.value !== undefined) documents.push(document.value);
const environment = clientRegistrationEnvironment(document.value);
if (!environment.ok)
return err({
kind: "malformed",
path,
detail: `Configuration ${path} has an invalid REA server environment: ${environment.error.settings.map(({ setting, constraint }) => `${setting}: ${constraint}`).join("; ")}. Repair it before rerunning setup.`,
});
documents.push(document.value);
}
} }
return ok(documents); return ok(documents);
}; };
@@ -1,54 +0,0 @@
import { z } from "zod";
import { parseEnvironment } from "../config/environment.js";
import { ConfigurationError } from "../domain/configurationErrors.js";
import { err, ok, type Result } from "../domain/result.js";
import {
effectiveClientServer,
type ClientConfigurationDocument,
} from "./ClientConfigurationDocument.js";
import { PRODUCT_IDENTITY } from "../identity.js";
/** Read explicit REA overrides without dropping peers when one value is invalid. */
export const clientRegistrationEnvironment = (
parsed: ClientConfigurationDocument | undefined,
overrides: Readonly<Record<string, string>> = {},
): Result<Readonly<Record<string, string>>, ConfigurationError> => {
const key =
parsed?.dialect === "opencode" || parsed?.dialect === "opencode_v2"
? "environment"
: "env";
const registration =
parsed === undefined
? undefined
: effectiveClientServer(parsed, PRODUCT_IDENTITY.mcpServerKey);
const raw =
typeof registration === "object" && registration !== null
? Reflect.get(registration, key)
: undefined;
const existing = z
.record(z.string(), z.string())
.safeParse(raw === undefined ? {} : raw);
if (!existing.success)
return err(
new ConfigurationError("Invalid REA server environment", {
settings: existing.error.issues.map((issue) => ({
setting: [key, ...issue.path].join("."),
constraint:
"Server environment must be an object containing only string values",
})),
}),
);
// Validate known settings with the runtime's parser, but retain the original
// string overrides, including unknown client-specific keys and no defaults.
const environment = { ...existing.data, ...overrides };
const validated = parseEnvironment(environment);
if (!validated.ok) return validated;
return ok(
Object.fromEntries(
Object.entries(environment).sort(([left], [right]) =>
left.localeCompare(right),
),
),
);
};
+13 -6
View File
@@ -5,22 +5,27 @@ import type { JsonValue } from "../domain/jsonValue.js";
import { jsonValueSchema } from "../domain/jsonValue.js"; import { jsonValueSchema } from "../domain/jsonValue.js";
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
import { compareBundles } from "../domain/bundleComparison.js"; import { compareBundles } from "../domain/bundleComparison.js";
import { EvidenceLedger } from "./investigation/EvidenceLedger.js";
import { import {
readEvidenceBundle, readEvidenceBundle,
writeEvidenceBundle, writeEvidenceBundle,
} from "./EvidenceBundleFiles.js"; } from "./EvidenceBundleFiles.js";
/** Validate a standalone bundle and report its complete record/revision counts. */ /** Validate and merge one bundle using the session Evidence ledger. */
export const importEvidenceBundleCommand = async ( export const importEvidenceBundleCommand = async (
path: string, path: string,
): Promise<Result<JsonValue, AnalysisError>> => { ): Promise<Result<JsonValue, AnalysisError>> => {
const loaded = await readEvidenceBundle(path); const loaded = await readEvidenceBundle(path);
if (!loaded.ok) return loaded; if (!loaded.ok) return loaded;
return ok({ const ledger = createLedger();
imported: loaded.value.records.length, const imported = ledger.import(loaded.value);
unknowns_added: loaded.value.unknowns.length, return imported.ok
total: loaded.value.records.length, ? ok({
}); imported: imported.value.recordsAdded,
unknowns_added: imported.value.unknownsAdded,
total: ledger.export().records.length,
})
: imported;
}; };
/** Validate a source bundle and atomically export canonical bytes. */ /** Validate a source bundle and atomically export canonical bytes. */
@@ -59,6 +64,8 @@ export const compareEvidenceBundlesCommand = async (input: {
} }
}; };
const createLedger = (): EvidenceLedger => new EvidenceLedger();
const projectWrite = ( const projectWrite = (
bundle: EvidenceBundle, bundle: EvidenceBundle,
written: Awaited<ReturnType<typeof writeEvidenceBundle>>, written: Awaited<ReturnType<typeof writeEvidenceBundle>>,
+5 -12
View File
@@ -16,7 +16,8 @@ import {
AnalysisResourceConstraintError, AnalysisResourceConstraintError,
} from "../domain/analysisErrorCore.js"; } from "../domain/analysisErrorCore.js";
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
import { readJsonInputFile } from "./JsonInputFile.js"; import { parseUtf8Json } from "./Utf8JsonInput.js";
import { readRegularFile } from "./RegularFileRead.js";
import { NonRegularFileReadError } from "../filesystem/RegularFile.js"; import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
/** Request control and its owning operation for an interruptible atomic write. */ /** Request control and its owning operation for an interruptible atomic write. */
@@ -36,20 +37,12 @@ export const readJsonFile = async (
> => { > => {
const requestedPath = resolve(path); const requestedPath = resolve(path);
try { try {
const decoded = await readJsonInputFile( const encoded = await readRegularFile(requestedPath);
requestedPath, const decoded = parseUtf8Json(encoded, "read_evidence_file", requestedPath);
"read_evidence_file",
);
if (!decoded.ok) { if (!decoded.ok) {
if (decoded.error instanceof AnalysisResourceConstraintError)
return err(decoded.error);
return err( return err(
new EvidenceFileError("read", "invalid-json", { new EvidenceFileError("read", "invalid-json", {
cause: cause: decoded.cause ?? new SyntaxError(decoded.error),
decoded.error.cause ??
new SyntaxError(
decoded.error.issues[0]?.message ?? decoded.error.message,
),
path: requestedPath, path: requestedPath,
}), }),
); );
+6 -27
View File
@@ -5,12 +5,6 @@ import type {
} from "../domain/binaryTargetTypes.js"; } from "../domain/binaryTargetTypes.js";
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
/** Observed ABI declarations and coverage of their source tables. */
export interface MipsAbiFlagsInspection {
readonly abiFlags: MipsAbiFlags | null;
readonly limitations: readonly string[];
}
/** /**
* Read ELF32 MIPS ABI declarations from the caller's already-open file. * Read ELF32 MIPS ABI declarations from the caller's already-open file.
* Inspect both typed tables (not section names), including records beyond the * Inspect both typed tables (not section names), including records beyond the
@@ -21,7 +15,7 @@ export const readMipsElfAbiFlags = async (
handle: FileHandle, handle: FileHandle,
metadata: MipsElfMetadata, metadata: MipsElfMetadata,
checkCancelled: () => void, checkCancelled: () => void,
): Promise<Result<MipsAbiFlagsInspection, string>> => { ): Promise<Result<MipsAbiFlags | null, string>> => {
checkCancelled(); checkCancelled();
if (metadata.elfClass !== 32) if (metadata.elfClass !== 32)
return err("MIPS ABI record inspection currently supports ELF32 only"); return err("MIPS ABI record inspection currently supports ELF32 only");
@@ -87,7 +81,6 @@ export const readMipsElfAbiFlags = async (
dataOffset: 4, dataOffset: 4,
sizeOffset: 16, sizeOffset: 16,
recordType: 0x70000003, // PT_MIPS_ABIFLAGS recordType: 0x70000003, // PT_MIPS_ABIFLAGS
optional: false,
}, },
{ {
label: "section header table", label: "section header table",
@@ -99,7 +92,6 @@ export const readMipsElfAbiFlags = async (
dataOffset: 16, dataOffset: 16,
sizeOffset: 20, sizeOffset: 20,
recordType: 0x7000002a, // SHT_MIPS_ABIFLAGS recordType: 0x7000002a, // SHT_MIPS_ABIFLAGS
optional: true,
}, },
]; ];
// Extended numbering needs a separate section-zero reader. Refuse it rather // Extended numbering needs a separate section-zero reader. Refuse it rather
@@ -107,7 +99,6 @@ export const readMipsElfAbiFlags = async (
if (u16(bytes, 44) === 0xffff || u16(bytes, 48) >= 0xff00) if (u16(bytes, 44) === 0xffff || u16(bytes, 48) >= 0xff00)
return err("extended or reserved MIPS ELF table numbering is unsupported"); return err("extended or reserved MIPS ELF table numbering is unsupported");
let record: Buffer | undefined; let record: Buffer | undefined;
const limitations: string[] = [];
for (const table of tables) { for (const table of tables) {
if (table.count === 0) { if (table.count === 0) {
if (table.offset !== 0) if (table.offset !== 0)
@@ -116,27 +107,16 @@ export const readMipsElfAbiFlags = async (
} }
if (table.offset < 52 || table.stride !== table.entrySize) if (table.offset < 52 || table.stride !== table.entrySize)
return err(`invalid MIPS ${table.label} offset or entry size`); return err(`invalid MIPS ${table.label} offset or entry size`);
const count = table.optional
? Math.min(
table.count,
Math.max(0, Math.floor((fileSize - table.offset) / table.entrySize)),
)
: table.count;
if (count < table.count)
limitations.push(
`MIPS ${table.label} at file offset ${table.offset} extends beyond the captured file: examined ${count} of ${table.count} complete headers. ABI declarations in unavailable headers remain unknown.`,
);
if (count === 0) continue;
// Counts are bounded by the ELF16 fields and strides by their exact ELF32 // Counts are bounded by the ELF16 fields and strides by their exact ELF32
// structure sizes. Never allocate according to an untrusted section size. // structure sizes. Never allocate according to an untrusted section size.
const contents = await read( const contents = await read(
table.offset, table.offset,
count * table.entrySize, table.count * table.entrySize,
table.label, table.label,
); );
if (!contents.ok) return contents; if (!contents.ok) return contents;
let found = false; let found = false;
for (let i = 0; i < count; i += 1) { for (let i = 0; i < table.count; i += 1) {
checkCancelled(); checkCancelled();
const position = i * table.entrySize; const position = i * table.entrySize;
if (u32(contents.value, position + table.typeOffset) !== table.recordType) if (u32(contents.value, position + table.typeOffset) !== table.recordType)
@@ -154,10 +134,10 @@ export const readMipsElfAbiFlags = async (
record = candidate.value; record = candidate.value;
} }
} }
if (record === undefined) return ok({ abiFlags: null, limitations }); if (record === undefined) return ok(null);
// Preserve declarations verbatim, including unknown versions/values. Provider // Preserve declarations verbatim, including unknown versions/values. Provider
// admission, not parsing, decides which interpretations have been verified. // admission, not parsing, decides which interpretations have been verified.
const abiFlags: MipsAbiFlags = { return ok({
version: u16(record, 0), version: u16(record, 0),
isaLevel: record.readUInt8(2), isaLevel: record.readUInt8(2),
isaRevision: record.readUInt8(3), isaRevision: record.readUInt8(3),
@@ -169,6 +149,5 @@ export const readMipsElfAbiFlags = async (
ases: u32(record, 12), ases: u32(record, 12),
flags1: u32(record, 16), flags1: u32(record, 16),
flags2: u32(record, 20), flags2: u32(record, 20),
}; });
return ok({ abiFlags, limitations });
}; };
@@ -1,11 +1,3 @@
import { javascriptApplicationAnalysisResultSchema } from "../domain/javascript/javascriptApplicationAnalysis.js";
import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../contracts/javascript/javascriptRuntimeReconciliationExample.js";
import {
createJavaScriptSemanticGraph,
createJavaScriptSemanticGraphNode,
JavaScriptSemanticEvidenceContextRegistry,
} from "../domain/javascript/javascriptSemanticGraph.js";
import { proofEvidence } from "./ReconstructionObligationLedger.fixture.js";
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { createEvidence } from "../domain/evidence.js"; import { createEvidence } from "../domain/evidence.js";
@@ -179,115 +171,3 @@ describe("reconstruction obligation candidates", () => {
); );
}); });
}); });
it("keeps each semantic candidate bound to its own file and authority context", () => {
const contexts = new JavaScriptSemanticEvidenceContextRegistry();
const base = javascriptApplicationAnalysisResultSchema.parse(
JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE.normalized_result,
);
const sourceRecords = [
proofEvidence("first-file"),
proofEvidence("second-file"),
];
const module = createJavaScriptSemanticGraphNode(
{
kind: "module",
identity: {
artifact_sha256: base.root_artifact_sha256,
module_path: "app.js",
source_range: null,
role_key: "module",
},
function_node_id: null,
application_node_ids: [],
label: "app.js",
properties: {},
evidence: artifactEvidence(base.root_artifact_sha256, "app.js"),
},
contexts,
);
const requests = sourceRecords.flatMap((source, fileIndex) =>
[1, 2].map((line) => {
const sha256 = String(fileIndex + 4).repeat(64);
const path = `file-${String(fileIndex)}.js`;
return createJavaScriptSemanticGraphNode(
{
kind: "request",
identity: {
artifact_sha256: sha256,
module_path: path,
source_range: {
start: { line, column: 0 },
end: { line, column: 8 },
},
role_key: `request-${String(line)}`,
},
function_node_id: null,
application_node_ids: [],
label: `${path}:${String(line)}`,
properties: {},
evidence: {
...artifactEvidence(sha256, path, "ast-static-analysis"),
evidence_ids: [source.evidence_id],
},
},
contexts,
);
}),
);
const { graph_id: _graphId, ...semantic } = base.semantic_graph;
const semanticGraph = createJavaScriptSemanticGraph({
...semantic,
root_node_ids: [module.node_id],
nodes: [module, ...requests],
evidence_contexts: contexts.contexts,
});
const example = JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE;
const application = createEvidence(
{
path: base.input_path,
sha256: base.root_artifact_sha256,
format: base.format,
},
example.provider,
{
predicateType: example.predicate_type,
operation: example.operation,
parameters: example.parameters,
result: jsonValueSchema.parse({ ...base, semantic_graph: semanticGraph }),
confidence: "derived",
authority: "shipped-artifact",
evidenceLinks: sourceRecords.map(({ evidence_id }) => evidence_id),
},
);
const generated = deriveReconstructionObligationCandidates(
createEvidenceBundle([...sourceRecords, application]),
[],
);
expect(generated.limitations).toEqual([]);
expect(generated.candidates).toHaveLength(requests.length);
for (const [index, node] of requests.entries()) {
const source = sourceRecords[Math.floor(index / 2)];
expect(
generated.candidates.find(
({ target }) => target.semantic_node_id === node.node_id,
),
).toMatchObject({
title: node.label,
family: "request",
source_state: "candidate",
target: {
artifact_sha256: node.identity.artifact_sha256,
semantic_node_id: node.node_id,
},
authority_references: [
{
evidence_id: source?.evidence_id,
authority: "controlled-replay",
state: "candidate",
location: `${semanticGraph.graph_id}/node/${node.node_id}`,
},
],
});
}
});
@@ -12,6 +12,7 @@ import type {
ReconstructionObligation, ReconstructionObligation,
ReviewedReconstructionObligation, ReviewedReconstructionObligation,
} from "../domain/reconstructionObligationLedgerSchemas.js"; } from "../domain/reconstructionObligationLedgerSchemas.js";
import { resolveJavaScriptSemanticEvidence } from "../domain/javascript/javascriptSemanticGraph.js";
import { parseApplicationGraphEvidence } from "./javascript/JavaScriptApplicationEvidenceGraph.js"; import { parseApplicationGraphEvidence } from "./javascript/JavaScriptApplicationEvidenceGraph.js";
import { import {
applicationObligationPolicy, applicationObligationPolicy,
@@ -114,21 +115,14 @@ const deriveApplicationCandidates = (
}), }),
); );
} }
if (source.semanticGraph !== null) { if (source.semanticGraph !== null)
const evidenceContexts = new Map(
source.semanticGraph.evidence_contexts.map((context) => [
context.context_id,
context,
]),
);
for (const node of source.semanticGraph.nodes) { for (const node of source.semanticGraph.nodes) {
const candidatePolicy = semanticObligationPolicy(node.kind); const candidatePolicy = semanticObligationPolicy(node.kind);
if (candidatePolicy === undefined) continue; if (candidatePolicy === undefined) continue;
const evidenceContext = evidenceContexts.get(node.evidence.context_id); const evidenceContext = resolveJavaScriptSemanticEvidence(
if (evidenceContext === undefined) source.semanticGraph,
throw new TypeError( node.evidence,
"Semantic evidence reference names an absent context", );
);
addCandidate( addCandidate(
candidates, candidates,
generatedCandidate({ generatedCandidate({
@@ -150,7 +144,6 @@ const deriveApplicationCandidates = (
}), }),
); );
} }
}
if (source.graph.coverage.status !== "complete") if (source.graph.coverage.status !== "complete")
limitations.add( limitations.add(
`Application graph ${source.graph.graph_id} coverage is ${source.graph.coverage.status}; omitted candidates remain unknown.`, `Application graph ${source.graph.graph_id} coverage is ${source.graph.coverage.status}; omitted candidates remain unknown.`,
+28 -121
View File
@@ -8,24 +8,14 @@ import {
type HistoricalSourceGraphInput, type HistoricalSourceGraphInput,
} from "../domain/referenceSourceGraph.js"; } from "../domain/referenceSourceGraph.js";
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import type { AnalysisCleanupObservation } from "../domain/analysisErrorBase.js";
import { readReferenceSource } from "../reference/ReferenceSourceReader.js"; import { readReferenceSource } from "../reference/ReferenceSourceReader.js";
import type {
ReferenceSourceEntryKind,
ReferenceSourceRead,
ReferenceSourceReaderError,
} from "../reference/ReferenceSourceReaderTypes.js";
import { parseReferenceSourceEntries } from "./ReferenceSourceImportEntries.js"; import { parseReferenceSourceEntries } from "./ReferenceSourceImportEntries.js";
import { readReferenceSourceVcs } from "./ReferenceSourceVcsAdapter.js"; import { readReferenceSourceVcs } from "./ReferenceSourceVcsAdapter.js";
import { import {
type ReferenceSourceImportError, type ReferenceSourceImportError,
type ReferenceSourceImportOptions, type ReferenceSourceImportOptions,
} from "./ReferenceSourceImportTypes.js"; } from "./ReferenceSourceImportTypes.js";
import { import { prepareReferenceSourceImport } from "./ReferenceSourceImportPolicy.js";
prepareReferenceSourceImport,
type PreparedReferenceSourceImport,
} from "./ReferenceSourceImportPolicy.js";
const failure = ( const failure = (
code: ReferenceSourceImportError["code"], code: ReferenceSourceImportError["code"],
@@ -36,12 +26,8 @@ const failure = (
message, message,
}); });
const cancelled = ( const cancelled = (): ReferenceSourceImportError =>
partial?: ReferenceSourceRead, failure("cancelled", "Reference source import cancelled");
): ReferenceSourceImportError => ({
...failure("cancelled", "Reference source import cancelled"),
...(partial === undefined ? {} : { partial }),
});
const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true; const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true;
@@ -129,48 +115,9 @@ const sortExclusions = (
[...exclusions].sort((left, right) => { [...exclusions].sort((left, right) => {
const byPath = compareUnicodeCodePoints(left.path, right.path); const byPath = compareUnicodeCodePoints(left.path, right.path);
if (byPath !== 0) return byPath; if (byPath !== 0) return byPath;
const byReason = compareUnicodeCodePoints(left.reason, right.reason); return compareUnicodeCodePoints(left.reason, right.reason);
if (byReason !== 0) return byReason;
return compareUnicodeCodePoints(
"pattern" in left ? left.pattern : "",
"pattern" in right ? right.pattern : "",
);
}); });
const createShouldExclude =
(
exclusions: HistoricalSourceGraphInput["exclusions"],
secrets: PreparedReferenceSourceImport["secrets"],
ignored: PreparedReferenceSourceImport["ignored"],
): ((path: string, kind: ReferenceSourceEntryKind) => boolean) =>
(path, kind) => {
const patternPath = kind === "directory" ? `${path}/` : path;
const secretMatch = secrets.test(patternPath);
if (secretMatch.ignored) {
if (!secretMatch.rule)
throw new Error(`Ignored secret path has no matching rule: ${path}`);
exclusions.push({
path,
reason: "configured-secret",
pattern: secretMatch.rule.pattern,
});
return true;
}
const match = ignored.test(patternPath);
if (!match.ignored) return false;
if (!match.rule)
throw new Error(`Ignored path has no matching rule: ${path}`);
const reason = match.rule.mark;
if (
reason !== "project-ignored" &&
reason !== "default-ignored" &&
reason !== "caller-excluded"
)
throw new Error(`Ignored path has unknown rule origin: ${path}`);
exclusions.push({ path, reason, pattern: match.rule.pattern });
return true;
};
/** /**
* Import a reference source directory into a committed historical source graph. * Import a reference source directory into a committed historical source graph.
* *
@@ -180,41 +127,43 @@ const createShouldExclude =
*/ */
export const importReferenceSource = async ( export const importReferenceSource = async (
options: ReferenceSourceImportOptions, options: ReferenceSourceImportOptions,
resources: ArtifactResourceScope,
): Promise<Result<HistoricalSourceGraph, ReferenceSourceImportError>> => { ): Promise<Result<HistoricalSourceGraph, ReferenceSourceImportError>> => {
if (isAborted(options.signal)) return err(cancelled()); if (isAborted(options.signal)) return err(cancelled());
const prepared = await prepareReferenceSourceImport(options, resources); const prepared = await prepareReferenceSourceImport(options);
if (!prepared.ok) return prepared; if (!prepared.ok) return prepared;
const { ignored, root, secrets } = prepared.value; const { ignored, root, secrets } = prepared.value;
if (isAborted(options.signal)) return err(cancelled()); if (isAborted(options.signal)) return err(cancelled());
const exclusions: HistoricalSourceGraphInput["exclusions"] = []; const exclusions: HistoricalSourceGraphInput["exclusions"] = [];
const shouldExclude = createShouldExclude(exclusions, secrets, ignored); const shouldExclude = (path: string): boolean => {
if (secrets.ignores(path)) {
exclusions.push({ path, reason: "configured-secret" });
return true;
}
if (ignored.ignores(path)) {
exclusions.push({ path, reason: "caller-excluded" });
return true;
}
return false;
};
const [readResult, vcsResult] = await Promise.all([ const [readResult, vcs] = await Promise.all([
readReferenceSource(root, resources, { readReferenceSource(root, {
...(options.signal === undefined ? {} : { signal: options.signal }), ...(options.signal === undefined ? {} : { signal: options.signal }),
shouldExclude, shouldExclude,
}), }),
readReferenceSourceVcs(root, resources, options.signal), readReferenceSourceVcs(root, options.signal),
]); ]);
if (!readResult.ok) if (!readResult.ok) {
return err( const error = readResult.error;
combineReadFailures( if (error.code === "cancelled") return err(cancelled());
readResult.error, if (error.code === "unsupported")
vcsResult.ok ? undefined : vcsResult.error, return err(failure("unsupported", error.message));
readResult.error.partial, return err(failure("io", error.message));
), }
);
if (!vcsResult.ok)
return err(
combineReadFailures(vcsResult.error, undefined, readResult.value),
);
const vcs = vcsResult.value; if (isAborted(options.signal)) return err(cancelled());
if (isAborted(options.signal)) return err(cancelled(readResult.value));
const read = readResult.value; const read = readResult.value;
const filePaths = new Set( const filePaths = new Set(
@@ -226,7 +175,7 @@ export const importReferenceSource = async (
const { entries, relationships, parseFailures, limitations } = const { entries, relationships, parseFailures, limitations } =
parseReferenceSourceEntries(read, filePaths, options.signal); parseReferenceSourceEntries(read, filePaths, options.signal);
if (isAborted(options.signal)) return err(cancelled(read)); if (isAborted(options.signal)) return err(cancelled());
const uniqueRelationships = deduplicateRelationships(relationships); const uniqueRelationships = deduplicateRelationships(relationships);
const uniqueFailures = normalizeHistoricalSourceParseFailures(parseFailures); const uniqueFailures = normalizeHistoricalSourceParseFailures(parseFailures);
@@ -272,45 +221,3 @@ export const importReferenceSource = async (
); );
} }
}; };
const combineReadFailures = (
primary: ReferenceSourceReaderError,
other: ReferenceSourceReaderError | undefined,
partial: ReferenceSourceRead | undefined,
): ReferenceSourceImportError => {
const code =
primary.code === "cancelled" || other?.code === "cancelled"
? "cancelled"
: primary.code;
const cleanup = mergeCleanup(primary.cleanup, other?.cleanup);
const message =
other === undefined
? primary.message
: `${primary.message}; additional reference metadata failure: ${other.message}`;
const cause =
other === undefined
? primary.cause
: new AggregateError(
[primary, other],
"Reference source inventory and metadata both failed",
{ cause: primary },
);
return {
...failure(code, message),
...(cleanup === undefined ? {} : { cleanup }),
...(partial === undefined ? {} : { partial }),
...(cause === undefined ? {} : { cause }),
};
};
const mergeCleanup = (
first: AnalysisCleanupObservation | undefined,
second: AnalysisCleanupObservation | undefined,
): AnalysisCleanupObservation | undefined => {
if (first === undefined) return second;
if (second === undefined) return first;
return {
reason: `${first.reason}; ${second.reason}`,
resources: [...new Set([...first.resources, ...second.resources])],
};
};
+21 -133
View File
@@ -1,22 +1,14 @@
import { lstat, realpath, stat } from "node:fs/promises"; import { readFile, realpath, stat } from "node:fs/promises";
import { join, resolve } from "node:path"; import { join, resolve } from "node:path";
import ignore from "ignore"; import ignore from "ignore";
import {
readRegularFileText,
RegularFileCleanupFailure,
retryRegularFileCleanup,
} from "./RegularFileRead.js";
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import { import {
DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS, DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS,
type ReferenceSourceImportError, type ReferenceSourceImportError,
type ReferenceSourceImportOptions, type ReferenceSourceImportOptions,
} from "./ReferenceSourceImportTypes.js"; } from "./ReferenceSourceImportTypes.js";
import { rootFilesystemFailure } from "../reference/ReferenceSourceReaderErrors.js";
/** Validated inputs ready for filesystem traversal. */ /** Validated inputs ready for filesystem traversal. */
export interface PreparedReferenceSourceImport { export interface PreparedReferenceSourceImport {
@@ -34,158 +26,54 @@ const failure = (
message, message,
}); });
const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true;
const addMarkedPatterns = (
matcher: ReturnType<typeof ignore>,
patterns: readonly string[],
mark: "project-ignored" | "default-ignored" | "caller-excluded",
): void => {
for (const pattern of patterns)
for (const line of pattern.split(/\r?\n/u))
matcher.add({ pattern: line, mark });
};
const resolveRoot = async ( const resolveRoot = async (
requestedRoot: string, requestedRoot: string,
signal?: AbortSignal,
): Promise<Result<string, ReferenceSourceImportError>> => { ): Promise<Result<string, ReferenceSourceImportError>> => {
if (isAborted(signal))
return err(failure("cancelled", "Reference source import cancelled"));
try { try {
const metadata = await stat(requestedRoot); if (!(await stat(requestedRoot)).isDirectory())
if (isAborted(signal))
return err(failure("cancelled", "Reference source import cancelled"));
if (!metadata.isDirectory())
return err( return err(
failure( failure("invalid-root", "Reference source root is not a directory"),
"invalid-root",
`Reference source root is not a directory: ${requestedRoot}`,
),
); );
const canonicalRoot = await realpath(resolve(requestedRoot)); const canonicalRoot = await realpath(resolve(requestedRoot));
if (isAborted(signal))
return err(failure("cancelled", "Reference source import cancelled"));
return ok(canonicalRoot); return ok(canonicalRoot);
} catch (cause: unknown) { } catch (cause: unknown) {
if (isAborted(signal)) void cause;
return err(failure("cancelled", "Reference source import cancelled")); return err(
const rootFailure = rootFilesystemFailure( failure("invalid-root", "Reference source root could not be resolved"),
cause,
"Reference source root could not be resolved",
); );
if (rootFailure === undefined) throw cause;
return err({ ...failure(rootFailure.code, rootFailure.message), cause });
} }
}; };
const buildIgnored = async ( const buildIgnored = async (
root: string, root: string,
excludePaths: readonly string[], excludePaths: readonly string[],
resources: ArtifactResourceScope, ): Promise<ReturnType<typeof ignore>> => {
signal?: AbortSignal,
): Promise<Result<ReturnType<typeof ignore>, ReferenceSourceImportError>> => {
const ignored = ignore(); const ignored = ignore();
const policyPath = join(root, ".gitignore");
let present = true;
try { try {
await lstat(policyPath); ignored.add(await readFile(join(root, ".gitignore"), "utf8"));
} catch (cause: unknown) { } catch (cause: unknown) {
if ( // Missing or unreadable ignore file does not authorize broader access.
!( void cause;
typeof cause === "object" &&
cause !== null &&
"code" in cause &&
cause.code === "ENOENT"
)
)
throw cause;
present = false;
} }
signal?.throwIfAborted(); ignored.add([...DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS]);
if (present) {
let text: string;
try {
text = await readRegularFileText(policyPath, { signal });
} catch (cause: unknown) {
if (!(cause instanceof RegularFileCleanupFailure)) throw cause;
const retry = await retryRegularFileCleanup(cause, resources);
if (retry.cleanup !== undefined) {
const primary =
cause.outcome.kind === "failed"
? cause.outcome.cause
: cause.cleanupCause;
return err({
...failure(
signal?.aborted === true ? "cancelled" : "io",
`Reference source ignore policy could not be read at ${policyPath}: ${errorMessage(primary)}`,
),
cleanup: retry.cleanup,
cause: primary,
});
}
if (retry.outcome.kind === "failed") throw retry.outcome.cause;
text = retry.outcome.value;
}
addMarkedPatterns(ignored, [text], "project-ignored");
}
addMarkedPatterns(
ignored,
DEFAULT_REFERENCE_SOURCE_IGNORE_PATTERNS,
"default-ignored",
);
for (const path of excludePaths) { for (const path of excludePaths) {
addMarkedPatterns(ignored, [path, `${path}/`], "caller-excluded"); ignored.add(path);
ignored.add(`${path}/`);
} }
return ok(ignored); return ignored;
}; };
/** Resolve the caller-selected directory and build path filters. */ /** Resolve the caller-selected directory and build path filters. */
export const prepareReferenceSourceImport = async ( export const prepareReferenceSourceImport = async (
options: ReferenceSourceImportOptions, options: ReferenceSourceImportOptions,
resources: ArtifactResourceScope,
): Promise< ): Promise<
Result<PreparedReferenceSourceImport, ReferenceSourceImportError> Result<PreparedReferenceSourceImport, ReferenceSourceImportError>
> => { > => {
try { const root = await resolveRoot(options.root);
return await resources.run(async () => { if (!root.ok) return root;
const root = await resolveRoot(options.root, options.signal); return ok({
if (!root.ok) return root; root: root.value,
try { ignored: await buildIgnored(root.value, options.excludePaths ?? []),
const ignored = await buildIgnored( secrets: ignore().add([...options.policy.secretPatterns]),
root.value, });
options.excludePaths ?? [],
resources,
options.signal,
);
if (!ignored.ok) return ignored;
return ok({
root: root.value,
ignored: ignored.value,
secrets: ignore().add([...options.policy.secretPatterns]),
});
} catch (cause: unknown) {
return err({
...failure(
options.signal?.aborted === true ? "cancelled" : "io",
`Reference source ignore policy could not be read at ${join(root.value, ".gitignore")}: ${errorMessage(cause)}`,
),
cause,
});
}
});
} catch (cause: unknown) {
if (!(cause instanceof ArtifactReaderFailure)) throw cause;
return err({
...failure(
cause.reason === "cancelled" ? "cancelled" : "io",
cause.message,
),
...(cause.cleanup === undefined ? {} : { cleanup: cause.cleanup }),
cause,
});
}
}; };
const errorMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
@@ -1,80 +0,0 @@
import { describe, expect, it } from "vitest";
import {
projectReferenceSourceImportError,
type ReferenceSourceImportError,
} from "./ReferenceSourceImportTypes.js";
import type { ReferenceSourceRead } from "../reference/ReferenceSourceReaderTypes.js";
describe("reference-source import error projection", () => {
it("preserves primary evidence with category-specific recovery guidance", () => {
const expectedCategories = {
cancelled: "cancelled",
"invalid-root": "invalid_input",
unsupported: "unsupported_host",
io: "execution_failure",
parse: "execution_failure",
} as const;
for (const [code, category] of Object.entries(expectedCategories)) {
const projected = projectReferenceSourceImportError({
tag: "reference-source-import",
code: code as ReferenceSourceImportError["code"],
message: "EIO while reading /selected/source/src/main.ts",
});
expect(projected.category).toBe(category);
expect(projected.message).toContain(
"EIO while reading /selected/source/src/main.ts",
);
expect(projected.message).toMatch(
/try again|when ready|Check that|REA on Linux/u,
);
}
});
it("preserves cancellation, partial facts, and cleanup diagnostics", () => {
const bytes = Buffer.from("export const observed = true;\n");
const partial: ReferenceSourceRead = {
root: "/selected/source",
entries: [
{
status: "read",
kind: "file",
path: "src/main.ts",
bytes,
size: bytes.length,
},
],
bytesRead: bytes.length,
limitations: [],
};
const projected = projectReferenceSourceImportError({
tag: "reference-source-import",
code: "cancelled",
message: "Traversal cancelled after reading src/main.ts",
cleanup: {
reason: "directory close rejected",
resources: ["/selected/source"],
},
partial,
});
expect(projected).toMatchObject({
category: "cancelled",
message: expect.stringContaining(
"Traversal cancelled after reading src/main.ts",
),
cleanup: {
reason: "directory close rejected",
resources: ["/selected/source"],
},
partial: {
root: "/selected/source",
entries: [{ path: "src/main.ts", bytes }],
},
});
expect(projected.message).toContain("Cleanup failed during import");
expect(projected.message).toContain("directory close rejected");
expect(projected.message).toContain("/selected/source");
expect(projected.message).not.toContain("remains unconfirmed");
});
});
+28 -51
View File
@@ -1,67 +1,44 @@
import type { ReferenceSourcePolicy } from "../domain/referenceSourcePolicy.js"; import type { ReferenceSourcePolicy } from "../domain/referenceSourcePolicy.js";
import type { AnalysisCleanupObservation } from "../domain/analysisErrorBase.js";
import type { ReferenceSourceRead } from "../reference/ReferenceSourceReaderTypes.js";
/** Typed expected failure returned by historical-source imports. */ /** Typed expected failure returned by historical-source imports. */
export interface ReferenceSourceImportError { export interface ReferenceSourceImportError {
readonly tag: "reference-source-import"; readonly tag: "reference-source-import";
readonly code: "cancelled" | "invalid-root" | "unsupported" | "io" | "parse"; readonly code: "cancelled" | "invalid-root" | "unsupported" | "io" | "parse";
readonly message: string; readonly message: string;
readonly cleanup?: AnalysisCleanupObservation;
readonly partial?: ReferenceSourceRead;
readonly cause?: unknown;
} }
const IMPORT_FAILURE_GUIDANCE = {
cancelled: {
category: "cancelled",
message:
"Reference source import was cancelled. Start it again when ready.",
},
"invalid-root": {
category: "invalid_input",
message:
"Reference source directory could not be opened. Check that the path exists, is readable, and points to a directory.",
},
unsupported: {
category: "unsupported_host",
message:
"Safe no-follow file opens are unavailable on this host. Import the source tree with REA on Linux (including WSL) or macOS.",
},
io: {
category: "execution_failure",
message:
"Reference source files could not be read. Check directory permissions and try again.",
},
parse: {
category: "execution_failure",
message:
"Reference source could not be indexed. Check that the source tree is readable, then try again.",
},
} satisfies Record<
ReferenceSourceImportError["code"],
{ readonly category: string; readonly message: string }
>;
/** Safe CLI projection for a historical-source import failure. */ /** Safe CLI projection for a historical-source import failure. */
export const projectReferenceSourceImportError = ( export const projectReferenceSourceImportError = (
error: ReferenceSourceImportError, error: ReferenceSourceImportError,
): Readonly<{ ): Readonly<{ category: string; message: string }> => {
category: string; if (error.code === "cancelled")
message: string; return {
cleanup?: AnalysisCleanupObservation; category: "cancelled",
partial?: ReferenceSourceRead; message:
}> => { "Reference source import was cancelled. Start it again when ready.",
const primary = IMPORT_FAILURE_GUIDANCE[error.code]; };
const cleanupMessage = if (error.code === "invalid-root")
error.cleanup === undefined return {
? "" category: "invalid_input",
: ` Cleanup failed during import for ${error.cleanup.resources.join(", ")}: ${error.cleanup.reason}.`; message:
"Reference source directory could not be opened. Check that the path exists, is readable, and points to a directory.",
};
if (error.code === "unsupported")
return {
category: "unsupported_host",
message:
"Safe no-follow file opens are unavailable on this host. Import the source tree with REA on Linux (including WSL) or macOS.",
};
if (error.code === "io")
return {
category: "execution_failure",
message:
"Reference source files could not be read. Check directory permissions and try again.",
};
return { return {
category: primary.category, category: "execution_failure",
message: `${error.message} ${primary.message}${cleanupMessage}`, message:
...(error.cleanup === undefined ? {} : { cleanup: error.cleanup }), "Reference source could not be indexed. Check that the source tree is readable, then try again.",
...(error.partial === undefined ? {} : { partial: error.partial }),
}; };
}; };
+28 -177
View File
@@ -1,15 +1,6 @@
import fs from "node:fs"; import fs from "node:fs";
import { lstat } from "node:fs/promises"; import { lstat, readFile } from "node:fs/promises";
import { join, resolve } from "node:path"; import { join, resolve } from "node:path";
import { err, ok, type Result } from "../domain/result.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import {
readRegularFile,
RegularFileCleanupFailure,
retryRegularFileCleanup,
} from "./RegularFileRead.js";
import type { ReferenceSourceReaderError } from "../reference/ReferenceSourceReaderTypes.js";
export type ReferenceSourceVcsInfo = export type ReferenceSourceVcsInfo =
| { | {
@@ -27,174 +18,34 @@ export type ReferenceSourceVcsInfo =
*/ */
export const readReferenceSourceVcs = async ( export const readReferenceSourceVcs = async (
root: string, root: string,
resources: ArtifactResourceScope,
signal?: AbortSignal, signal?: AbortSignal,
): Promise<Result<ReferenceSourceVcsInfo, ReferenceSourceReaderError>> => { ): Promise<ReferenceSourceVcsInfo> => {
if (isAborted(signal)) return { kind: "unknown", head: null, dirty: null };
try { try {
return await resources.run(async () => { await lstat(join(root, ".git"));
if (isAborted(signal)) return ok(unknownVcs());
try {
await lstat(join(root, ".git"));
} catch (cause: unknown) {
return ok(
errorCode(cause) === "ENOENT"
? { kind: "none", head: null, dirty: null }
: unknownVcs(),
);
}
try {
const head = await resolveSourceHead(root, resources, signal);
if (isAborted(signal)) return ok(unknownVcs());
return ok({ kind: "git", head, dirty: null });
} catch (cause: unknown) {
if (cause instanceof ArtifactReaderFailure) throw cause;
return ok(unknownVcs());
}
});
} catch (cause: unknown) { } catch (cause: unknown) {
if (!(cause instanceof ArtifactReaderFailure)) throw cause; return errorCode(cause) === "ENOENT"
return err({ ? { kind: "none", head: null, dirty: null }
tag: "reference-source-reader", : { kind: "unknown", head: null, dirty: null };
code: cause.reason === "cancelled" ? "cancelled" : "io", }
message: cause.message, try {
...(cause.cleanup === undefined ? {} : { cleanup: cause.cleanup }), const head = await resolveSourceHead(root);
cause, if (isAborted(signal)) return { kind: "unknown", head: null, dirty: null };
}); return { kind: "git", head, dirty: null };
} catch (cause: unknown) {
// Unresolvable refs mean VCS state is unknown, not absent.
void cause;
return { kind: "unknown", head: null, dirty: null };
} }
}; };
const unknownVcs = (): ReferenceSourceVcsInfo => ({ const resolveSourceHead = async (root: string): Promise<string> => {
kind: "unknown",
head: null,
dirty: null,
});
const resolveSourceHead = async (
root: string,
resources: ArtifactResourceScope,
signal?: AbortSignal,
): Promise<string> => {
// isomorphic-git is loaded on first use so CLI and MCP startup skip it. // isomorphic-git is loaded on first use so CLI and MCP startup skip it.
const { resolveRef } = await import("isomorphic-git"); const { resolveRef } = await import("isomorphic-git");
let failedRead: { readonly cause: unknown } | undefined;
const cleanupFailures: ArtifactReaderFailure[] = [];
const readMetadataBytes = async (path: string): Promise<Buffer> => {
try {
return await readRegularFile(path, { signal });
} catch (cause: unknown) {
if (!(cause instanceof RegularFileCleanupFailure)) throw cause;
const retry = await retryRegularFileCleanup(cause, resources);
if (
retry.outcome.kind === "failed" &&
errorCode(retry.outcome.cause) !== "ENOENT"
)
failedRead ??= { cause: retry.outcome.cause };
if (
retry.cleanup === undefined &&
retry.outcome.kind === "failed" &&
!isAborted(signal)
)
throw retry.outcome.cause;
const primary =
retry.outcome.kind === "failed"
? retry.outcome.cause
: cause.cleanupCause;
throw new ArtifactReaderFailure(
isAborted(signal) ? "cancelled" : "io",
`Git metadata could not be read safely: ${errorMessage(primary)}`,
{
cause: primary,
...(retry.cleanup === undefined ? {} : { cleanup: retry.cleanup }),
},
);
}
};
const readMetadataText = async (path: string): Promise<string> =>
(await readMetadataBytes(path)).toString("utf8");
const throwCleanupFailures = (primary: unknown): never => {
const observations = cleanupFailures.flatMap((failure) =>
failure.cleanup === undefined ? [] : [failure.cleanup],
);
throw new ArtifactReaderFailure(
isAborted(signal) ? "cancelled" : "io",
`Git metadata could not be read safely: ${errorMessage(primary)}`,
{
cause: new AggregateError(
[primary, ...cleanupFailures],
"Git metadata reads and cleanup failed",
{ cause: primary },
),
...(observations.length === 0
? {}
: {
cleanup: {
reason: observations.map(({ reason }) => reason).join("; "),
resources: [
...new Set(
observations.flatMap(({ resources }) => resources),
),
],
},
}),
},
);
};
const referenceFs = {
...fs,
promises: {
...fs.promises,
// Let isomorphic-git continue from a failed loose ref to packed-refs,
// then report the original read failure after resolution. This preserves
// both observations when packed metadata also fails cleanup.
readFile: async (
path: string | undefined,
options:
| BufferEncoding
| { readonly encoding?: BufferEncoding | null }
| null = null,
) => {
// The library probes readFile() without a path to detect promise APIs.
// Reject that invocation without attributing it to a source read.
if (path === undefined)
throw new TypeError("Git filesystem read requires a path");
try {
const bytes = await readMetadataBytes(path);
const encoding =
typeof options === "string" ? options : options?.encoding;
return encoding === undefined || encoding === null
? bytes
: bytes.toString(encoding);
} catch (cause: unknown) {
if (cause instanceof ArtifactReaderFailure) {
cleanupFailures.push(cause);
} else if (errorCode(cause) !== "ENOENT") failedRead ??= { cause };
throw cause;
}
},
},
};
const readRef = async (options: {
readonly gitdir?: string;
readonly ref: string;
}) => {
let head: string;
try {
head = await resolveRef({ fs: referenceFs, dir: root, ...options });
} catch (cause: unknown) {
if (cleanupFailures.length > 0)
throwCleanupFailures(failedRead?.cause ?? cause);
if (failedRead !== undefined) throw failedRead.cause;
throw cause;
}
if (cleanupFailures.length > 0)
throwCleanupFailures(failedRead?.cause ?? cleanupFailures[0]?.cause);
signal?.throwIfAborted();
if (failedRead !== undefined) throw failedRead.cause;
return head;
};
const dotgit = join(root, ".git"); const dotgit = join(root, ".git");
if (!(await lstat(dotgit)).isFile()) return readRef({ ref: "HEAD" }); if (!(await lstat(dotgit)).isFile())
const pointer = (await readMetadataText(dotgit)).replace(/\r?\n$/u, ""); return resolveRef({ fs, dir: root, ref: "HEAD" });
const pointer = (await readFile(dotgit, "utf8")).replace(/\r?\n$/u, "");
if (!pointer.startsWith("gitdir: ") || pointer.length === 8) if (!pointer.startsWith("gitdir: ") || pointer.length === 8)
throw new Error("Invalid Git directory pointer"); throw new Error("Invalid Git directory pointer");
const gitdir = resolve(root, pointer.slice(8)); const gitdir = resolve(root, pointer.slice(8));
@@ -202,22 +53,25 @@ const resolveSourceHead = async (
try { try {
commonDirectory = resolve( commonDirectory = resolve(
gitdir, gitdir,
(await readMetadataText(join(gitdir, "commondir"))).replace( (await readFile(join(gitdir, "commondir"), "utf8")).replace(
/\r?\n$/u, /\r?\n$/u,
"", "",
), ),
); );
} catch (cause: unknown) { } catch (cause: unknown) {
if (errorCode(cause) !== "ENOENT") throw cause; if (errorCode(cause) !== "ENOENT") throw cause;
return readRef({ gitdir, ref: "HEAD" }); return resolveRef({ fs, dir: root, gitdir, ref: "HEAD" });
} }
const head = (await readMetadataText(join(gitdir, "HEAD"))).trim(); const head = (await readFile(join(gitdir, "HEAD"), "utf8")).trim();
if (!head.startsWith("ref: ")) return readRef({ gitdir, ref: "HEAD" }); if (!head.startsWith("ref: "))
return resolveRef({ fs, dir: root, gitdir, ref: "HEAD" });
const ref = head.slice(5); const ref = head.slice(5);
const privateRef = ["refs/bisect/", "refs/rewritten/", "refs/worktree/"].some( const privateRef = ["refs/bisect/", "refs/rewritten/", "refs/worktree/"].some(
(prefix) => ref.startsWith(prefix), (prefix) => ref.startsWith(prefix),
); );
return readRef({ return resolveRef({
fs,
dir: root,
gitdir: privateRef ? gitdir : commonDirectory, gitdir: privateRef ? gitdir : commonDirectory,
ref, ref,
}); });
@@ -228,7 +82,4 @@ const errorCode = (cause: unknown): string | undefined =>
? String(cause.code) ? String(cause.code)
: undefined; : undefined;
const errorMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true; const isAborted = (signal?: AbortSignal): boolean => signal?.aborted === true;
+9 -138
View File
@@ -1,158 +1,29 @@
import type { Stats } from "node:fs"; import type { Stats } from "node:fs";
import { lstat, stat, type FileHandle } from "node:fs/promises"; import type { FileHandle } from "node:fs/promises";
import { import { openRegularFile } from "../filesystem/RegularFile.js";
RegularFileAdmissionFailure,
openRegularFile,
sameRegularFileState,
RegularFileChangedError,
} from "../filesystem/RegularFile.js";
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
import { readBoundedFileBytes } from "../process/BoundedFileBytes.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import type { AnalysisCleanupObservation } from "../domain/analysisErrorBase.js";
/** Filesystem admission policy for one selected regular input. */
export interface RegularFileReadOptions {
readonly signal?: AbortSignal | undefined;
readonly symlinks?: "follow" | "reject";
}
export type RegularFileReadOutcome<Value> =
| { readonly kind: "completed"; readonly value: Value }
| { readonly kind: "failed"; readonly cause: unknown };
/** A close failed after reading; this error transfers the exact handle owner. */
export class RegularFileCleanupFailure<Value = unknown> extends Error {
constructor(
readonly path: string,
readonly owner: OwnedFileHandle,
readonly outcome: RegularFileReadOutcome<Value>,
readonly cleanupCause: unknown,
) {
super(`Regular file cleanup failed: ${path}`, {
cause: outcome.kind === "failed" ? outcome.cause : cleanupCause,
});
this.name = "RegularFileCleanupFailure";
}
}
/** Retry a failed close through its retained owner and report any remaining cleanup. */
export const retryRegularFileCleanup = async <Value>(
failure: RegularFileCleanupFailure<Value>,
resources: ArtifactResourceScope,
): Promise<{
readonly outcome: RegularFileReadOutcome<Value>;
readonly cleanup?: AnalysisCleanupObservation;
}> => {
const released = await resources.release({
kind: "file-handle",
handle: failure.owner,
resource: failure.path,
});
if (released.kind === "released") return { outcome: failure.outcome };
return {
outcome: failure.outcome,
cleanup: {
reason: `${failureMessage(failure.cleanupCause)}; retry: ${failureMessage(released.cause)}`,
resources: [failure.path],
},
};
};
const failureMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
/** Admit one regular-file handle, retain it through the consumer, and always close it. */ /** Admit one regular-file handle, retain it through the consumer, and always close it. */
export const withRegularFile = async <Value>( export const withRegularFile = async <Value>(
path: string, path: string,
read: (handle: FileHandle, stats: Stats) => Promise<Value>, read: (handle: FileHandle, stats: Stats) => Promise<Value>,
options: RegularFileReadOptions = {}, signal?: AbortSignal,
): Promise<Value> => { ): Promise<Value> => {
const { signal } = options; const handle = await openRegularFile(path, { symlinks: "follow", signal });
let handle: FileHandle;
try {
handle = await openRegularFile(path, {
symlinks: options.symlinks ?? "follow",
signal,
});
} catch (cause: unknown) {
if (!(cause instanceof RegularFileAdmissionFailure)) throw cause;
throw new RegularFileCleanupFailure(
path,
cause.owner,
{ kind: "failed", cause: cause.cause },
cause.cleanupCause,
);
}
const owner = new OwnedFileHandle(handle);
let outcome: RegularFileReadOutcome<Value>;
try { try {
const stats = await handle.stat(); const stats = await handle.stat();
signal?.throwIfAborted(); signal?.throwIfAborted();
const value = await read(handle, stats); const value = await read(handle, stats);
signal?.throwIfAborted(); signal?.throwIfAborted();
const [opened, currentPath] = await Promise.all([ return value;
handle.stat(), } finally {
options.symlinks === "reject" ? lstat(path) : stat(path), await handle.close();
]);
if (
!sameRegularFileState(stats, opened) ||
!sameRegularFileState(stats, currentPath)
)
throw new RegularFileChangedError(path);
outcome = { kind: "completed", value };
} catch (cause: unknown) {
outcome = { kind: "failed", cause };
} }
try {
await owner.close();
} catch (cleanupCause: unknown) {
throw new RegularFileCleanupFailure(path, owner, outcome, cleanupCause);
}
if (outcome.kind === "failed") throw outcome.cause;
return outcome.value;
}; };
/** Admit a regular file without waiting for a pipe, then read its verified handle. */ /** Admit a regular file without waiting for a pipe, then read its verified handle. */
export const readRegularFile = ( export const readRegularFile = (
path: string, path: string,
options: RegularFileReadOptions = {}, signal?: AbortSignal,
): Promise<Buffer> => ): Promise<Buffer> =>
withRegularFile( withRegularFile(path, (handle) => handle.readFile({ signal }), signal);
path,
async (handle, stats) => {
const bytes = await readBoundedFileBytes(
handle,
stats.size,
options.signal,
);
if (bytes === undefined || bytes.length !== stats.size)
throw new RegularFileChangedError(path);
return bytes;
},
options,
);
/** Read one admitted regular file as UTF-8 without waiting on a pipe. */
export const readRegularFileText = async (
path: string,
options: RegularFileReadOptions = {},
): Promise<string> => {
try {
return (await readRegularFile(path, options)).toString("utf8");
} catch (cause: unknown) {
if (
cause instanceof RegularFileCleanupFailure &&
cause.outcome.kind === "completed" &&
Buffer.isBuffer(cause.outcome.value)
)
throw new RegularFileCleanupFailure(
cause.path,
cause.owner,
{ kind: "completed", value: cause.outcome.value.toString("utf8") },
cause.cleanupCause,
);
throw cause;
}
};
-73
View File
@@ -1,8 +1,5 @@
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { AnalysisError } from "../domain/analysisErrorBase.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import { ConfigurationError } from "../domain/configurationErrors.js";
import { SUPPORTED_NODE_VERSION_PROSE } from "../domain/runtimeVersion.js"; import { SUPPORTED_NODE_VERSION_PROSE } from "../domain/runtimeVersion.js";
import { FakeSetupHost, options } from "./Setup.fixture.js"; import { FakeSetupHost, options } from "./Setup.fixture.js";
import { runSetup } from "./Setup.js"; import { runSetup } from "./Setup.js";
@@ -110,76 +107,6 @@ describe("setup workflow", () => {
}); });
}); });
describe("setup failure cleanup projection", () => {
it("projects a configuration failure together with failed host cleanup", async () => {
const configPath = "/fixture/home/vscode/settings.json";
const primary = new ConfigurationError("Client config could not be read", {
settings: [
{
setting: configPath,
constraint: "The selected config could not be read.",
},
],
cleanup: {
reason: "The admitted config handle remains open.",
resources: [configPath],
},
});
class SetupHostWithCleanupFailure extends FakeSetupHost {
close = async (): Promise<string> =>
"Setup could not release its client configuration resources.";
}
const host = new SetupHostWithCleanupFailure();
host.clients = [{ name: "codex", configPath }];
host.inspectClientConfiguration = async () => {
throw primary;
};
const failure = await runSetup(
{ ...options(true), clientIds: ["codex"] },
host,
).then(
() => undefined,
(cause: unknown) => cause,
);
expect(failure).toBeInstanceOf(AnalysisError);
if (!(failure instanceof AnalysisError))
throw new Error("Expected setup to preserve a typed analysis failure");
const projected = projectAnalysisError(failure);
expect(projected).toMatchObject({
code: "cleanup_incomplete",
details: {
cleanup: "incomplete",
resources: [configPath],
diagnostics: {
primary_error: {
code: "cleanup_incomplete",
details: { resources: [configPath] },
},
cleanup_error: {
code: "cleanup_incomplete",
details: { resources: [configPath] },
},
},
},
});
expect(projected.details).toMatchObject({
diagnostics: {
primary_error: { message: expect.stringContaining("selected config") },
cleanup_error: {
details: {
diagnostics: {
reason:
"Setup could not release its client configuration resources.",
},
},
},
},
});
});
});
describe("setup scoped readiness", () => { describe("setup scoped readiness", () => {
it("limits Hopper host checks to workflows that request Hopper", async () => { it("limits Hopper host checks to workflows that request Hopper", async () => {
const platformHost = new FakeSetupHost("win32"); const platformHost = new FakeSetupHost("win32");
-14
View File
@@ -23,9 +23,6 @@ import {
hostRemediation, hostRemediation,
systemSetupHost, systemSetupHost,
} from "./SetupHost.js"; } from "./SetupHost.js";
import { AnalysisError } from "../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
/** Discover, approve, and apply setup actions idempotently. */ /** Discover, approve, and apply setup actions idempotently. */
export const runSetup = async ( export const runSetup = async (
@@ -51,17 +48,6 @@ export const runSetup = async (
} }
if (threw) { if (threw) {
if (cleanupFailure !== undefined && primaryFailure instanceof AnalysisError)
throw analysisErrorWithCleanupFailure(
primaryFailure,
new ProviderCleanupError(
"setup",
primaryFailure.cleanupResources,
{ reason: cleanupFailure },
{ operation: "setup" },
),
"setup",
);
if (cleanupFailure !== undefined) if (cleanupFailure !== undefined)
throw new AggregateError( throw new AggregateError(
[primaryFailure, new Error(cleanupFailure)], [primaryFailure, new Error(cleanupFailure)],
+47 -120
View File
@@ -11,22 +11,17 @@ import {
serializeClientConfiguration, serializeClientConfiguration,
withClientServers, withClientServers,
type ClientConfigurationDocument, type ClientConfigurationDocument,
type ClientRegistrationDialect,
} from "./ClientConfigurationDocument.js"; } from "./ClientConfigurationDocument.js";
import { constants as fsConstants } from "node:fs"; import { constants as fsConstants } from "node:fs";
import { copyFile, mkdir, rm, writeFile } from "node:fs/promises"; import { copyFile, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { dirname } from "node:path"; import { dirname } from "node:path";
import writeFileAtomic from "write-file-atomic"; import writeFileAtomic from "write-file-atomic";
import { clientRegistrationEnvironment } from "./ClientRegistrationEnvironment.js";
import { PRODUCT_IDENTITY } from "../identity.js"; import { PRODUCT_IDENTITY } from "../identity.js";
import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js"; import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js";
import { MCP_STARTUP_POLICY } from "../mcpStartupPolicy.js"; import { MCP_STARTUP_POLICY } from "../mcpStartupPolicy.js";
import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js"; import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js";
import {
readRegularFileText,
RegularFileCleanupFailure,
} from "./RegularFileRead.js";
import type { ClientConfigurationFileError } from "./ClientConfigurationFiles.js";
import type { import type {
ClientConfigurationInspection, ClientConfigurationInspection,
ClientConfigurationResult, ClientConfigurationResult,
@@ -42,37 +37,25 @@ export const configureClientConfiguration = (
): Promise<ClientConfigurationResult> => { ): Promise<ClientConfigurationResult> => {
if (client.configPathError !== undefined) if (client.configPathError !== undefined)
return Promise.resolve({ status: "failed", reason: "path" }); return Promise.resolve({ status: "failed", reason: "path" });
return configureClientDocument(client, environment, command); if (client.format === undefined || client.format === "unsupported")
return Promise.resolve({ status: "failed", reason: "readback" });
return configureClientDocument(client, environment, command, client.format);
}; };
const configureClientDocument = async ( const configureClientDocument = async (
client: SetupClient, client: SetupClient,
environment: SetupProviderEnvironment, environment: SetupProviderEnvironment,
command: readonly string[], command: readonly string[],
format: NonNullable<SetupClient["format"]>,
): Promise<ClientConfigurationResult> => { ): Promise<ClientConfigurationResult> => {
const format = client.format;
if (format === undefined || format === "unsupported")
return { status: "failed", reason: "readback" };
const files = await readClientConfigurationFiles(client); const files = await readClientConfigurationFiles(client);
if (!files.ok) { if (!files.ok) return { status: "failed", reason: "readback" };
throwCleanupFailure(files.error);
return { status: "failed", reason: "readback" };
}
const policy = await readClientPolicyBlock( const policy = await readClientPolicyBlock(
client, client,
undefined, undefined,
clientConfigurationDesired( clientConfigurationDesired(client, environment, command, client.format),
client,
environment,
command,
files.value.at(-1),
),
); );
if (!policy.ok) { if (!policy.ok || policy.value !== undefined)
throwCleanupFailure(policy.error);
return { status: "failed", reason: "readback" };
}
if (policy.value !== undefined)
return { status: "failed", reason: "readback" }; return { status: "failed", reason: "readback" };
const transactionPath = await resolveClientConfigTransactionPath( const transactionPath = await resolveClientConfigTransactionPath(
client.configPath, client.configPath,
@@ -81,9 +64,8 @@ const configureClientDocument = async (
return { status: "failed", reason: "path" }; return { status: "failed", reason: "path" };
let original: string | undefined; let original: string | undefined;
try { try {
original = await readRegularFileText(transactionPath); original = await readFile(transactionPath, "utf8");
} catch (cause: unknown) { } catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure) throw cause;
if (!isMissing(cause)) return { status: "failed", reason: "readback" }; if (!isMissing(cause)) return { status: "failed", reason: "readback" };
} }
let parsed: ClientConfigurationDocument; let parsed: ClientConfigurationDocument;
@@ -95,16 +77,11 @@ const configureClientDocument = async (
const policy = await readClientPolicyBlock( const policy = await readClientPolicyBlock(
client, client,
parsed, parsed,
clientConfigurationDesired(client, environment, command, parsed), clientConfigurationDesired(client, environment, command, parsed.dialect),
); );
if (!policy.ok) { if (!policy.ok || policy.value !== undefined)
throwCleanupFailure(policy.error);
return { status: "failed", reason: "readback" };
}
if (policy.value !== undefined)
return { status: "failed", reason: "readback" }; return { status: "failed", reason: "readback" };
} catch (cause: unknown) { } catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure) throw cause;
// Malformed existing configuration fails the readback gate. // Malformed existing configuration fails the readback gate.
void cause; void cause;
return { status: "failed", reason: "readback" }; return { status: "failed", reason: "readback" };
@@ -113,36 +90,9 @@ const configureClientDocument = async (
client, client,
environment, environment,
command, command,
parsed, parsed.dialect,
); );
if (registrationCurrent(parsed, desired)) return { status: "unchanged" }; if (registrationCurrent(parsed, desired)) return { status: "unchanged" };
return persistClientConfiguration({
client,
transactionPath,
format,
original,
parsed,
desired,
});
};
interface ClientConfigurationUpdate {
readonly client: SetupClient;
readonly transactionPath: string;
readonly format: NonNullable<SetupClient["format"]>;
readonly original: string | undefined;
readonly parsed: ClientConfigurationDocument;
readonly desired: unknown;
}
const persistClientConfiguration = async ({
client,
transactionPath,
format,
original,
parsed,
desired,
}: ClientConfigurationUpdate): Promise<ClientConfigurationResult> => {
const backupPath = const backupPath =
original === undefined ? undefined : `${client.configPath}.rea.backup`; original === undefined ? undefined : `${client.configPath}.rea.backup`;
if ( if (
@@ -199,7 +149,7 @@ const persistClientConfiguration = async ({
} }
try { try {
const readback = parseClientConfiguration( const readback = parseClientConfiguration(
await readRegularFileText(transactionPath), await readFile(transactionPath, "utf8"),
format, format,
); );
if (!registrationCurrent(readback, desired)) { if (!registrationCurrent(readback, desired)) {
@@ -207,7 +157,6 @@ const persistClientConfiguration = async ({
return { status: "failed", reason: "readback" }; return { status: "failed", reason: "readback" };
} }
} catch (cause: unknown) { } catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure) throw cause;
// Readback failure restores the transaction before reporting. // Readback failure restores the transaction before reporting.
void cause; void cause;
await restoreConfig(transactionPath, original); await restoreConfig(transactionPath, original);
@@ -228,22 +177,21 @@ export const clientConfigurationAligned = async (
if (client.configPathError !== undefined) return false; if (client.configPathError !== undefined) return false;
try { try {
const files = await readClientConfigurationFiles(client); const files = await readClientConfigurationFiles(client);
if (!files.ok) { if (!files.ok) return false;
throwCleanupFailure(files.error); const original = await readFile(client.configPath, "utf8");
return false;
}
const original = await readRegularFileText(client.configPath);
const parsed = parseClientConfiguration(original, client.format); const parsed = parseClientConfiguration(original, client.format);
const policy = await readClientPolicyBlock( const policy = await readClientPolicyBlock(
client, client,
parsed, parsed,
clientConfigurationDesired(client, providerEnvironment, command, parsed), clientConfigurationDesired(
client,
providerEnvironment,
command,
parsed.dialect,
),
); );
if (!policy.ok) {
throwCleanupFailure(policy.error);
return false;
}
return ( return (
policy.ok &&
policy.value === undefined && policy.value === undefined &&
registrationCurrent( registrationCurrent(
parsed, parsed,
@@ -251,12 +199,11 @@ export const clientConfigurationAligned = async (
client, client,
providerEnvironment, providerEnvironment,
command, command,
parsed, parsed.dialect,
), ),
) )
); );
} catch (cause: unknown) { } catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure) throw cause;
// Unreadable configuration is treated as not aligned so setup repairs it. // Unreadable configuration is treated as not aligned so setup repairs it.
void cause; void cause;
return false; return false;
@@ -273,13 +220,7 @@ export const inspectClientConfiguration = async (
return { status: "invalid", remediation: client.configPathError }; return { status: "invalid", remediation: client.configPathError };
if (client.format === "unsupported") return { status: "already_current" }; if (client.format === "unsupported") return { status: "already_current" };
const files = await readClientConfigurationFiles(client); const files = await readClientConfigurationFiles(client);
if (!files.ok) { if (!files.ok) return { status: "invalid", remediation: files.error.detail };
throwCleanupFailure(files.error);
return {
status: "invalid",
remediation: files.error.detail,
};
}
const policy = await readClientPolicyBlock( const policy = await readClientPolicyBlock(
client, client,
undefined, undefined,
@@ -287,16 +228,11 @@ export const inspectClientConfiguration = async (
client, client,
providerEnvironment, providerEnvironment,
command, command,
files.value.at(-1), client.format,
), ),
); );
if (!policy.ok) { if (!policy.ok)
throwCleanupFailure(policy.error); return { status: "invalid", remediation: policy.error.detail };
return {
status: "invalid",
remediation: policy.error.detail,
};
}
if (policy.value !== undefined) if (policy.value !== undefined)
return { status: "invalid", remediation: policy.value }; return { status: "invalid", remediation: policy.value };
const transactionPath = await resolveClientConfigTransactionPath( const transactionPath = await resolveClientConfigTransactionPath(
@@ -310,13 +246,13 @@ export const inspectClientConfiguration = async (
}; };
let original: string; let original: string;
try { try {
original = await readRegularFileText(transactionPath); original = await readFile(transactionPath, "utf8");
} catch (cause: unknown) { } catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure) throw cause;
if (isMissing(cause)) return { status: "create" }; if (isMissing(cause)) return { status: "create" };
return { return {
status: "invalid", status: "invalid",
remediation: `The configuration file could not be read. Check its permissions before rerunning setup. ${cause instanceof Error ? cause.message : String(cause)}`, remediation:
"The configuration file could not be read. Check its permissions before rerunning setup.",
}; };
} }
try { try {
@@ -324,27 +260,26 @@ export const inspectClientConfiguration = async (
const policy = await readClientPolicyBlock( const policy = await readClientPolicyBlock(
client, client,
parsed, parsed,
clientConfigurationDesired(client, providerEnvironment, command, parsed), clientConfigurationDesired(
client,
providerEnvironment,
command,
parsed.dialect,
),
); );
if (!policy.ok) { if (!policy.ok)
throwCleanupFailure(policy.error); return { status: "invalid", remediation: policy.error.detail };
return {
status: "invalid",
remediation: policy.error.detail,
};
}
if (policy.value !== undefined) if (policy.value !== undefined)
return { status: "invalid", remediation: policy.value }; return { status: "invalid", remediation: policy.value };
const desired = clientConfigurationDesired( const desired = clientConfigurationDesired(
client, client,
providerEnvironment, providerEnvironment,
command, command,
parsed, parsed.dialect,
); );
if (registrationCurrent(parsed, desired)) if (registrationCurrent(parsed, desired))
return { status: "already_current" }; return { status: "already_current" };
} catch (cause: unknown) { } catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure) throw cause;
// Malformed configuration is reported with the invalid remediation. // Malformed configuration is reported with the invalid remediation.
void cause; void cause;
return { return {
@@ -361,11 +296,6 @@ export const inspectClientConfiguration = async (
const isMissing = (cause: unknown): boolean => const isMissing = (cause: unknown): boolean =>
cause instanceof Error && "code" in cause && cause.code === "ENOENT"; cause instanceof Error && "code" in cause && cause.code === "ENOENT";
const throwCleanupFailure = (error: ClientConfigurationFileError): void => {
if (error.cleanupFailure !== undefined) throw error.cleanupFailure;
};
const preserveConfigBackup = async ( const preserveConfigBackup = async (
source: string, source: string,
destination: string, destination: string,
@@ -407,20 +337,17 @@ const clientConfigurationDesired = (
client: SetupClient, client: SetupClient,
providerEnvironment: SetupProviderEnvironment, providerEnvironment: SetupProviderEnvironment,
command: readonly string[], command: readonly string[],
parsed?: ClientConfigurationDocument, format: ClientRegistrationDialect | undefined,
) => { ) => {
const format = parsed?.dialect ?? client.format ?? "json"; const environment = Object.fromEntries(
// Preserve explicitly configured server settings, never the ambient process Object.entries(providerEnvironment).sort(([left], [right]) =>
// environment. Newly discovered provider paths win only for their own keys. left.localeCompare(right),
const environment = clientRegistrationEnvironment( ),
parsed,
providerEnvironment,
); );
if (!environment.ok) throw environment.error;
const registration = clientRegistrationEntry( const registration = clientRegistrationEntry(
format, format ?? "json",
command.length === 0 ? [PRODUCT_IDENTITY.cliBinary, "mcp"] : command, command.length === 0 ? [PRODUCT_IDENTITY.cliBinary, "mcp"] : command,
environment.value, environment,
); );
return { return {
...registration, ...registration,
+12 -110
View File
@@ -1,9 +1,6 @@
import { access } from "node:fs/promises"; import { access } from "node:fs/promises";
import { resolve } from "node:path"; import { resolve } from "node:path";
import { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import { ConfigurationError } from "../domain/configurationErrors.js";
import { PRODUCT_IDENTITY } from "../identity.js"; import { PRODUCT_IDENTITY } from "../identity.js";
import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js"; import { npxRegistrationCommand } from "./ClientRegistrationIdentity.js";
import { import {
@@ -33,10 +30,6 @@ import {
inspectClientConfiguration, inspectClientConfiguration,
} from "./SetupClientConfiguration.js"; } from "./SetupClientConfiguration.js";
import { setupInstallFailure } from "./SetupInstallFailure.js"; import { setupInstallFailure } from "./SetupInstallFailure.js";
import {
RegularFileCleanupFailure,
retryRegularFileCleanup,
} from "./RegularFileRead.js";
import { providerRegistrationEnvironment } from "./SetupRegistrationEnvironment.js"; import { providerRegistrationEnvironment } from "./SetupRegistrationEnvironment.js";
import type { import type {
SetupHost, SetupHost,
@@ -105,7 +98,6 @@ export const systemSetupHost = (
platform === "darwin" platform === "darwin"
? systemMacHopperInstallHost(homeDirectory, environment) ? systemMacHopperInstallHost(homeDirectory, environment)
: undefined; : undefined;
const resources = new ArtifactResourceScope();
return { return {
platform, platform,
homeDirectory, homeDirectory,
@@ -118,7 +110,7 @@ export const systemSetupHost = (
nodeVersion: process.versions.node, nodeVersion: process.versions.node,
macosVersion: () => doctorHost.macosVersion(), macosVersion: () => doctorHost.macosVersion(),
linuxDistribution: readLinuxDistribution, linuxDistribution: readLinuxDistribution,
close: () => closeSetupResources(resources, macHopperHost), close: () => macHopperHost?.close() ?? Promise.resolve(undefined),
initialSetupState: async ( initialSetupState: async (
scope?: DoctorScope, scope?: DoctorScope,
): Promise<SetupInitialState> => { ): Promise<SetupInitialState> => {
@@ -170,7 +162,17 @@ export const systemSetupHost = (
detectedClients: () => detectClients(homeDirectory, platform, environment), detectedClients: () => detectClients(homeDirectory, platform, environment),
supportedClients: () => supportedClients: () =>
Promise.resolve(supportedClients(homeDirectory, platform, environment)), Promise.resolve(supportedClients(homeDirectory, platform, environment)),
...clientConfigurationOperations(resources), configureClient: (client, providerEnvironment, command) =>
client.format === "unsupported"
? Promise.resolve({ status: "skipped" })
: configureClientConfiguration(client, providerEnvironment, command),
clientNeedsConfigure: (client, providerEnvironment, command) =>
client.format === "unsupported"
? Promise.resolve(false)
: clientConfigurationAligned(client, providerEnvironment, command).then(
(aligned) => !aligned,
),
inspectClientConfiguration: inspectClientConfiguration,
skillNeedsInstall: (clientIds) => skillNeedsInstall: (clientIds) =>
canonicalSkillNeedsInstall( canonicalSkillNeedsInstall(
homeDirectory, homeDirectory,
@@ -184,106 +186,6 @@ export const systemSetupHost = (
}; };
}; };
const clientConfigurationOperations = (resources: ArtifactResourceScope) => ({
configureClient: (
client: SetupClient,
providerEnvironment: SetupProviderEnvironment,
command: readonly string[],
) =>
runClientConfigurationOperation(resources, async () =>
client.format === "unsupported"
? { status: "skipped" as const }
: configureClientConfiguration(client, providerEnvironment, command),
),
clientNeedsConfigure: (
client: SetupClient,
providerEnvironment: SetupProviderEnvironment,
command: readonly string[],
) =>
runClientConfigurationOperation(resources, async () =>
client.format === "unsupported"
? false
: !(await clientConfigurationAligned(
client,
providerEnvironment,
command,
)),
),
inspectClientConfiguration: (
client: SetupClient,
providerEnvironment: SetupProviderEnvironment,
command: readonly string[],
) =>
runClientConfigurationOperation(resources, () =>
inspectClientConfiguration(client, providerEnvironment, command),
),
});
const closeSetupResources = async (
resources: ArtifactResourceScope,
macHopperHost: ReturnType<typeof systemMacHopperInstallHost> | undefined,
): Promise<string | undefined> => {
const [resourceCleanup, hopperCleanup] = await Promise.allSettled([
resources.close(),
Promise.resolve().then(() => macHopperHost?.close()),
]);
const failures: string[] = [];
if (hopperCleanup.status === "rejected")
failures.push(
`Hopper host cleanup failed: ${errorMessage(hopperCleanup.reason)}`,
);
else if (hopperCleanup.value !== undefined)
failures.push(hopperCleanup.value);
if (resourceCleanup.status === "rejected")
failures.push(errorMessage(resourceCleanup.reason));
return failures.length === 0 ? undefined : failures.join("; ");
};
const runClientConfigurationOperation = async <Value>(
resources: ArtifactResourceScope,
operation: () => Promise<Value>,
): Promise<Value> => {
try {
return await resources.run(async () => {
try {
return await operation();
} catch (cause: unknown) {
if (!(cause instanceof RegularFileCleanupFailure)) throw cause;
const cleanup = await retryRegularFileCleanup(cause, resources);
const primary =
cleanup.outcome.kind === "failed"
? cleanup.outcome.cause
: cause.cleanupCause;
throw configurationCleanupError(
`Could not safely access client configuration at ${cause.path}: ${errorMessage(primary)}`,
cause,
cleanup.cleanup,
);
}
});
} catch (cause: unknown) {
if (cause instanceof ArtifactReaderFailure)
throw configurationCleanupError(cause.message, cause, cause.cleanup);
throw cause;
}
};
const configurationCleanupError = (
message: string,
cause: unknown,
cleanup:
| { readonly reason: string; readonly resources: readonly string[] }
| undefined,
): ConfigurationError =>
new ConfigurationError(message, {
cause,
settings: [{ setting: "client configuration", constraint: message }],
...(cleanup === undefined ? {} : { cleanup }),
});
const errorMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
/** Detect supported agents from their config files or stable installation markers. */ /** Detect supported agents from their config files or stable installation markers. */
export const detectClients = async ( export const detectClients = async (
home: string, home: string,
+2 -2
View File
@@ -312,8 +312,8 @@ const clientConfigurationDetail = (
.map(([name, value]) => `${name}=${value}`); .map(([name, value]) => `${name}=${value}`);
const environmentDetail = const environmentDetail =
entries.length === 0 entries.length === 0
? " No additional provider environment variables. Existing server environment overrides are retained." ? " No additional environment variables."
: ` Detected provider environment: ${entries.join(", ")}. Existing server environment overrides are retained; detected values replace matching keys.`; : ` Environment: ${entries.join(", ")}.`;
const hopperDetail = installHopper const hopperDetail = installHopper
? environment.HOPPER_LAUNCHER_PATH === undefined ? environment.HOPPER_LAUNCHER_PATH === undefined
? " If this plan installs Hopper, REA will add HOPPER_LAUNCHER_PATH using the launcher path reported by that installation." ? " If this plan installs Hopper, REA will add HOPPER_LAUNCHER_PATH using the launcher path reported by that installation."
+14 -7
View File
@@ -4,13 +4,14 @@ import { dirname, join } from "node:path";
import writeFileAtomic from "write-file-atomic"; import writeFileAtomic from "write-file-atomic";
import { PRODUCT_IDENTITY } from "../identity.js"; import { PRODUCT_IDENTITY } from "../identity.js";
import { NonRegularFileReadError } from "../filesystem/RegularFile.js"; import {
import { readRegularFileText } from "./RegularFileRead.js"; NonRegularFileReadError,
openRegularFile,
} from "../filesystem/RegularFile.js";
import { clientSkillDirectories } from "./SupportedClients.js"; import { clientSkillDirectories } from "./SupportedClients.js";
export const MANAGED_SKILL_FILES = [ export const MANAGED_SKILL_FILES = [
"SKILL.md", "SKILL.md",
"references/connection-and-recovery.md",
"references/native-and-artifacts.md", "references/native-and-artifacts.md",
"references/javascript-applications.md", "references/javascript-applications.md",
"references/android-applications.md", "references/android-applications.md",
@@ -22,8 +23,14 @@ export const isManagedSkillManifest = (content: string): boolean =>
content.startsWith(`---\nname: ${PRODUCT_IDENTITY.skillName}\n`) || content.startsWith(`---\nname: ${PRODUCT_IDENTITY.skillName}\n`) ||
content.startsWith(`---\r\nname: ${PRODUCT_IDENTITY.skillName}\r\n`); content.startsWith(`---\r\nname: ${PRODUCT_IDENTITY.skillName}\r\n`);
export const readSkillFile = (path: string): Promise<string> => export const readSkillFile = async (path: string): Promise<string> => {
readRegularFileText(path, { symlinks: "reject" }); const handle = await openRegularFile(path, { symlinks: "reject" });
try {
return await handle.readFile("utf8");
} finally {
await handle.close();
}
};
interface CanonicalSkillFile { interface CanonicalSkillFile {
readonly content: string; readonly content: string;
@@ -192,7 +199,7 @@ export const installCanonicalSkill = async (
for (const { destination, content } of changed) for (const { destination, content } of changed)
await writeText(destination, content); await writeText(destination, content);
for (const { destination, content } of changed) for (const { destination, content } of changed)
if ((await readSkillFile(destination)) !== content) if ((await readFile(destination, "utf8")) !== content)
throw new Error(`skill readback mismatch: ${destination}`); throw new Error(`skill readback mismatch: ${destination}`);
return "installed"; return "installed";
} catch (cause: unknown) { } catch (cause: unknown) {
@@ -224,7 +231,7 @@ export const readInstalledSkillIdentity = async (
let content: string | undefined; let content: string | undefined;
for (const destination of destinations) { for (const destination of destinations) {
try { try {
content = await readSkillFile(join(destination.path, "SKILL.md")); content = await readFile(join(destination.path, "SKILL.md"), "utf8");
break; break;
} catch { } catch {
// Another selected copy may provide metadata; canonical validation below // Another selected copy may provide metadata; canonical validation below
+2 -3
View File
@@ -6,7 +6,7 @@ import {
withClientServers, withClientServers,
type ClientConfigurationDocument, type ClientConfigurationDocument,
} from "./ClientConfigurationDocument.js"; } from "./ClientConfigurationDocument.js";
import { copyFile, lstat, realpath, rm } from "node:fs/promises"; import { copyFile, lstat, readFile, realpath, rm } from "node:fs/promises";
import { constants as fsConstants } from "node:fs"; import { constants as fsConstants } from "node:fs";
import { homeDirectoryFromEnvironment } from "../config/homeDirectory.js"; import { homeDirectoryFromEnvironment } from "../config/homeDirectory.js";
import { join } from "node:path"; import { join } from "node:path";
@@ -23,7 +23,6 @@ import {
} from "./SetupSkill.js"; } from "./SetupSkill.js";
import { isOwnedClientRegistrationCommand } from "./ClientRegistrationIdentity.js"; import { isOwnedClientRegistrationCommand } from "./ClientRegistrationIdentity.js";
import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js"; import { resolveClientConfigTransactionPath } from "./ClientConfigPath.js";
import { readRegularFileText } from "./RegularFileRead.js";
import { import {
manualRegistrationRemediation, manualRegistrationRemediation,
supportedClients, supportedClients,
@@ -50,7 +49,7 @@ export interface UninstallFileSystem {
} }
const systemFileSystem: UninstallFileSystem = { const systemFileSystem: UninstallFileSystem = {
readText: (path) => readRegularFileText(path), readText: (path) => readFile(path, "utf8"),
readRegularText: readSkillFile, readRegularText: readSkillFile,
copy: (source, destination) => copy: (source, destination) =>
copyFile(source, destination, fsConstants.COPYFILE_EXCL), copyFile(source, destination, fsConstants.COPYFILE_EXCL),
+2 -2
View File
@@ -1,3 +1,4 @@
import { readFile } from "node:fs/promises";
import { z } from "zod"; import { z } from "zod";
import { PRODUCT_IDENTITY } from "../identity.js"; import { PRODUCT_IDENTITY } from "../identity.js";
@@ -13,7 +14,6 @@ import { supportedClients } from "./SupportedClients.js";
import { existingSkillDestinations } from "./SetupSkill.js"; import { existingSkillDestinations } from "./SetupSkill.js";
import type { SetupAction } from "./SetupTypes.js"; import type { SetupAction } from "./SetupTypes.js";
import type { Result } from "../domain/result.js"; import type { Result } from "../domain/result.js";
import { readRegularFileText } from "./RegularFileRead.js";
/** Existing REA integrations selected for maintenance, without discovering new targets. */ /** Existing REA integrations selected for maintenance, without discovering new targets. */
export interface MaintenanceScope { export interface MaintenanceScope {
@@ -73,7 +73,7 @@ export const existingMaintenanceScope = async (
const client = supported.find(({ name }) => name === registration.client); const client = supported.find(({ name }) => name === registration.client);
if (client === undefined) continue; if (client === undefined) continue;
const parsed = parseClientConfiguration( const parsed = parseClientConfiguration(
await readRegularFileText(client.configPath), await readFile(client.configPath, "utf8"),
client.format, client.format,
); );
const enabled = z const enabled = z
+85 -244
View File
@@ -4,7 +4,6 @@ import { isAbsolute } from "node:path";
import { z } from "zod"; import { z } from "zod";
import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js"; import { ArtifactReaderFailure } from "../artifacts/ArtifactReader.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
import { publishWebScripts } from "../browser/assets/PublishWebScripts.js"; import { publishWebScripts } from "../browser/assets/PublishWebScripts.js";
import { selectScriptCapture } from "../browser/assets/ScriptCaptureAdapters.js"; import { selectScriptCapture } from "../browser/assets/ScriptCaptureAdapters.js";
import { import {
@@ -15,11 +14,9 @@ import {
} from "../domain/analysisErrorCore.js"; } from "../domain/analysisErrorCore.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js"; import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { createEvidence, type Evidence } from "../domain/evidence.js"; import { createEvidence, type Evidence } from "../domain/evidence.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js"; import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js";
import { jsonValueSchema } from "../domain/jsonValue.js"; import { jsonValueSchema } from "../domain/jsonValue.js";
import { ProviderAdapterError } from "../domain/providerAdapterError.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
import { safeParseJson } from "../domain/safeJson.js"; import { safeParseJson } from "../domain/safeJson.js";
import { import {
@@ -29,41 +26,19 @@ import {
import { WebScriptExportError } from "../domain/webScriptExportError.js"; import { WebScriptExportError } from "../domain/webScriptExportError.js";
import type { ExecutionOptions } from "./AnalysisProvider.js"; import type { ExecutionOptions } from "./AnalysisProvider.js";
import { WEB_SCRIPT_EXPORT_PROVIDER as PROVIDER } from "./InvestigationProviders.js"; import { WEB_SCRIPT_EXPORT_PROVIDER as PROVIDER } from "./InvestigationProviders.js";
import { import { readRegularFile } from "./RegularFileRead.js";
readRegularFile, import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
RegularFileCleanupFailure,
retryRegularFileCleanup,
} from "./RegularFileRead.js";
import {
NonRegularFileReadError,
RegularFileChangedError,
} from "../filesystem/RegularFile.js";
const OPERATION = "export_web_scripts"; const OPERATION = "export_web_scripts";
interface CaptureRead {
readonly bytes: Buffer;
readonly cleanup?: {
readonly reason: string;
readonly incomplete: boolean;
readonly cause: unknown;
};
}
interface PreparedCapture {
readonly bytes: Buffer;
readonly loaded: ReturnType<typeof selectScriptCapture>;
}
/** Export one local capture through the shared CLI/MCP application workflow. */ /** Export one local capture through the shared CLI/MCP application workflow. */
export const exportWebScripts = async ( export const exportWebScripts = async (
rawInput: unknown, rawInput: unknown,
resources: ArtifactResourceScope,
options: ExecutionOptions = {}, options: ExecutionOptions = {},
): Promise<Result<Evidence, AnalysisError>> => { ): Promise<Result<Evidence, AnalysisError>> => {
const input = exportWebScriptsInputSchema.safeParse(rawInput); const input = exportWebScriptsInputSchema.safeParse(rawInput);
return input.success return input.success
? exportWebScriptsValidated(input.data, resources, options) ? exportWebScriptsValidated(input.data, options)
: err( : err(
analysisInputErrorFromIssues(OPERATION, input.error.issues, rawInput), analysisInputErrorFromIssues(OPERATION, input.error.issues, rawInput),
); );
@@ -72,7 +47,6 @@ export const exportWebScripts = async (
/** Publish input already parsed by a named adapter contract. */ /** Publish input already parsed by a named adapter contract. */
export const exportWebScriptsValidated = async ( export const exportWebScriptsValidated = async (
input: ExportWebScriptsInput, input: ExportWebScriptsInput,
resources: ArtifactResourceScope,
options: ExecutionOptions = {}, options: ExecutionOptions = {},
): Promise<Result<Evidence, AnalysisError>> => { ): Promise<Result<Evidence, AnalysisError>> => {
if (!isAbsolute(input.capture_path) || !isAbsolute(input.output_directory)) if (!isAbsolute(input.capture_path) || !isAbsolute(input.output_directory))
@@ -88,19 +62,19 @@ export const exportWebScriptsValidated = async (
); );
try { try {
options.signal?.throwIfAborted(); options.signal?.throwIfAborted();
const prepared = await prepareCapture( const read = await readCapture(input.capture_path, options.signal);
input.capture_path, if (!read.ok) return read;
resources, const bytes = read.value;
options.signal, const loaded = parseCapture(bytes);
); if (!loaded.ok) return loaded;
if (!prepared.ok) return prepared;
const { bytes, loaded } = prepared.value;
options.signal?.throwIfAborted(); options.signal?.throwIfAborted();
const sha256 = createHash("sha256").update(bytes).digest("hex"); const sha256 = createHash("sha256").update(bytes).digest("hex");
const result = await publishWebScripts(input, loaded, sha256, { const result = await publishWebScripts(
resources, input,
...(options.signal === undefined ? {} : { signal: options.signal }), loaded.value,
}); sha256,
options.signal,
);
return ok( return ok(
createEvidence( createEvidence(
{ path: input.capture_path, sha256, format: "file" }, { path: input.capture_path, sha256, format: "file" },
@@ -119,109 +93,55 @@ export const exportWebScriptsValidated = async (
limitations: result.limitations, limitations: result.limitations,
locations: [{ kind: "artifact-path", path: result.manifest.path }], locations: [{ kind: "artifact-path", path: result.manifest.path }],
evidenceLinks: evidenceLinks:
loaded.sourceEvidenceId === null ? [] : [loaded.sourceEvidenceId], loaded.value.sourceEvidenceId === null
? []
: [loaded.value.sourceEvidenceId],
}, },
), ),
); );
} catch (cause: unknown) { } catch (cause: unknown) {
return err(mapExportFailure(cause, input, options.signal)); if (cause instanceof WebScriptExportError) return err(cause);
} if (options.signal?.aborted === true)
}; return err(new AnalysisCancelledError(OPERATION));
if (cause instanceof ArtifactReaderFailure)
const prepareCapture = async ( return err(
path: string, new WebScriptExportError(
resources: ArtifactResourceScope, cause.reason,
signal: AbortSignal | undefined, input.output_directory,
): Promise<Result<PreparedCapture, AnalysisError>> => { cause.message,
const read = await resources.run(() => readCapture(path, resources, signal)); ),
if (!read.ok) return read; );
const cleanup = read.value.cleanup; if (cause instanceof z.ZodError)
const loaded = parseCapture( return err(new AnalysisOutputError(OPERATION, cause.message, { cause }));
read.value.bytes, if (
cleanup === undefined || cleanup.incomplete cause instanceof Error &&
? undefined "code" in cause &&
: { cause: cleanup.cause }, typeof cause.code === "string"
); )
if (!loaded.ok) return err(
new WebScriptExportError(
"io",
`${input.capture_path} → ${input.output_directory}`,
cause.message,
),
);
return err( return err(
cleanup?.incomplete === true new ProviderAdapterError(PROVIDER.id, OPERATION, {
? incompleteCaptureCleanup(loaded.error, path, cleanup) cause,
: loaded.error, diagnostics: {
); capture_path: input.capture_path,
if (cleanup !== undefined) { output_directory: input.output_directory,
const primary = new WebScriptExportError( error_name: cause instanceof Error ? cause.name : "UnknownError",
"io", error_message:
path, cause instanceof Error
`Capture handle close failed: ${cleanup.reason}`, ? cause.message
{ cause: cleanup.cause }, : typeof cause === "string"
); ? cause
return err( : "Unknown script export failure",
cleanup.incomplete },
? incompleteCaptureCleanup(primary, path, cleanup) }),
: primary,
); );
} }
return ok({ bytes: read.value.bytes, loaded: loaded.value });
};
const mapExportFailure = (
cause: unknown,
input: ExportWebScriptsInput,
signal: AbortSignal | undefined,
): AnalysisError => {
if (cause instanceof WebScriptExportError) return cause;
if (cause instanceof ArtifactReaderFailure && cause.cleanup !== undefined)
return analysisErrorWithCleanupFailure(
new WebScriptExportError(
cause.reason,
input.output_directory,
cause.message,
{ cause },
),
new ProviderCleanupError(
PROVIDER.id,
cause.cleanup.resources,
{ reason: cause.cleanup.reason },
{ cause, operation: OPERATION },
),
OPERATION,
);
if (signal?.aborted === true)
return new AnalysisCancelledError(OPERATION, { cause });
if (cause instanceof ArtifactReaderFailure)
return new WebScriptExportError(
cause.reason,
input.output_directory,
cause.message,
{ cause },
);
if (cause instanceof z.ZodError)
return new AnalysisOutputError(OPERATION, cause.message, { cause });
if (
cause instanceof Error &&
"code" in cause &&
typeof cause.code === "string"
)
return new WebScriptExportError(
"io",
`${input.capture_path} → ${input.output_directory}`,
cause.message,
{ cause },
);
return new ProviderAdapterError(PROVIDER.id, OPERATION, {
cause,
diagnostics: {
capture_path: input.capture_path,
output_directory: input.output_directory,
error_name: cause instanceof Error ? cause.name : "UnknownError",
error_message:
cause instanceof Error
? cause.message
: typeof cause === "string"
? cause
: "Unknown script export failure",
},
});
}; };
/** /**
@@ -230,115 +150,44 @@ const mapExportFailure = (
*/ */
const readCapture = async ( const readCapture = async (
path: string, path: string,
resources: ArtifactResourceScope,
signal: AbortSignal | undefined, signal: AbortSignal | undefined,
): Promise<Result<CaptureRead, AnalysisError>> => { ): Promise<Result<Buffer, AnalysisError>> => {
try { try {
return ok({ bytes: await readRegularFile(path, { signal }) }); return ok(await readRegularFile(path, signal));
} catch (cause: unknown) { } catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure) { const code =
const retry = await retryRegularFileCleanup(cause, resources); cause instanceof Error && "code" in cause ? String(cause.code) : "";
const cleanup = { if (code === "EACCES" || code === "EPERM")
reason: retry.cleanup?.reason ?? errorMessage(cause.cleanupCause),
incomplete: retry.cleanup !== undefined,
cause: cause.cleanupCause,
};
if (cause.outcome.kind === "completed")
return ok({ bytes: cause.outcome.value as Buffer, cleanup });
const primary = captureReadFailure(
path,
cause.outcome.cause,
signal,
retry.cleanup === undefined ? { cause: cause.cleanupCause } : undefined,
);
return err( return err(
retry.cleanup === undefined new AnalysisAccessDeniedError(OPERATION, path, code, { cause }),
? primary
: incompleteCaptureCleanup(primary, path, cleanup),
); );
} if (
return err(captureReadFailure(path, cause, signal)); cause instanceof NonRegularFileReadError ||
code === "ENOENT" ||
code === "ENOTDIR" ||
code === "EISDIR" ||
code === "ENXIO"
)
return err(
new AnalysisInputError(OPERATION, { cause }, [
{
path: ["capture_path"],
reason: "invalid_value",
message:
cause instanceof NonRegularFileReadError && code !== "EISDIR"
? cause.message
: code === "EISDIR"
? `Selected capture is a directory, not a file: ${path}`
: `Selected capture could not be read (${code}): ${path}`,
},
]),
);
throw cause;
} }
}; };
const captureReadFailure = (
path: string,
cause: unknown,
signal: AbortSignal | undefined,
closeFailure?: { readonly cause: unknown },
): AnalysisError => {
const causeWithClose =
closeFailure === undefined
? cause
: new AggregateError(
[cause, closeFailure.cause],
"Capture read and file-handle close both failed",
{ cause },
);
if (signal?.aborted === true)
return new AnalysisCancelledError(OPERATION, { cause: causeWithClose });
const code =
cause instanceof Error && "code" in cause ? String(cause.code) : "";
if (code === "EACCES" || code === "EPERM")
return new AnalysisAccessDeniedError(OPERATION, path, code, {
cause: causeWithClose,
});
if (
cause instanceof NonRegularFileReadError ||
cause instanceof RegularFileChangedError ||
code === "ENOENT" ||
code === "ENOTDIR" ||
code === "EISDIR" ||
code === "ENXIO"
)
return new AnalysisInputError(OPERATION, { cause: causeWithClose }, [
{
path: ["capture_path"],
reason: "invalid_value",
message:
cause instanceof NonRegularFileReadError && code !== "EISDIR"
? cause.message
: code === "EISDIR"
? `Selected capture is a directory, not a file: ${path}`
: `Selected capture could not be read (${code}): ${path}`,
},
]);
if (code !== "")
return new WebScriptExportError("io", path, errorMessage(cause), {
cause: causeWithClose,
});
return new ProviderAdapterError(PROVIDER.id, OPERATION, {
cause: causeWithClose,
diagnostics: {
capture_path: path,
error_name: cause instanceof Error ? cause.name : "UnknownError",
error_message: cause instanceof Error ? cause.message : String(cause),
},
});
};
const incompleteCaptureCleanup = (
primary: AnalysisError,
path: string,
cleanup: NonNullable<CaptureRead["cleanup"]>,
): AnalysisError =>
analysisErrorWithCleanupFailure(
primary,
new ProviderCleanupError(
PROVIDER.id,
[path],
{ reason: cleanup.reason },
{ cause: cleanup.cause, operation: OPERATION },
),
OPERATION,
);
const errorMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const parseCapture = ( const parseCapture = (
bytes: Buffer, bytes: Buffer,
closeFailure?: { readonly cause: unknown },
): Result<ReturnType<typeof selectScriptCapture>, AnalysisError> => { ): Result<ReturnType<typeof selectScriptCapture>, AnalysisError> => {
try { try {
const text = new TextDecoder("utf-8", { const text = new TextDecoder("utf-8", {
@@ -349,16 +198,8 @@ const parseCapture = (
if (!json.ok) throw new TypeError(json.error); if (!json.ok) throw new TypeError(json.error);
return ok(selectScriptCapture(json.value)); return ok(selectScriptCapture(json.value));
} catch (cause: unknown) { } catch (cause: unknown) {
const causeWithClose =
closeFailure === undefined
? cause
: new AggregateError(
[cause, closeFailure.cause],
"Capture parse and file-handle close both failed",
{ cause },
);
return err( return err(
new AnalysisInputError(OPERATION, { cause: causeWithClose }, [ new AnalysisInputError(OPERATION, { cause }, [
{ {
path: ["capture_path"], path: ["capture_path"],
reason: "invalid_format", reason: "invalid_format",
+1 -1
View File
@@ -14,7 +14,7 @@ import type { Result } from "../../domain/result.js";
export interface AdbDeviceAnalysisPort { export interface AdbDeviceAnalysisPort {
/** Probe the selected adb binary without contacting devices. */ /** Probe the selected adb binary without contacting devices. */
inspectAvailability(signal?: AbortSignal): Promise<ProviderAvailability>; inspectAvailability(signal?: AbortSignal): Promise<ProviderAvailability>;
/** Terminally stop admission, drain work, and retry retained package rollback. */ /** The provider holds no long-lived resources; safe to call always. */
close(): Promise<void>; close(): Promise<void>;
execute( execute(
request: AdbRequest, request: AdbRequest,
@@ -11,7 +11,6 @@ import {
} from "../../../tests/fixtures/analysisView.js"; } from "../../../tests/fixtures/analysisView.js";
import { import {
inspectAnalysisView, inspectAnalysisView,
inspectAnalysisViewValidated,
summarizeRetainedAnalysis, summarizeRetainedAnalysis,
} from "./AnalysisViewService.js"; } from "./AnalysisViewService.js";
import type { Evidence } from "../../domain/evidence.js"; import type { Evidence } from "../../domain/evidence.js";
@@ -59,35 +58,6 @@ it.each(["/fixtures/unknown.exe", ""])(
}, },
); );
it("reports inline Evidence identity failures at the complete source path", () => {
const parent = analysisViewLayoutEvidence();
const input = {
source: {
kind: "inline",
evidence: { ...parent, evidence_id: `ev_${"0".repeat(64)}` },
},
view: { kind: "summary" },
} as const;
for (const result of [
inspectAnalysisView(input),
inspectAnalysisViewValidated(input),
])
expect(result).toMatchObject({
ok: false,
error: {
_tag: "AnalysisInputError",
issues: [
{
path: ["source", "evidence", "evidence_id"],
reason: "invalid_value",
message: "Evidence semantic identifier does not match its record",
},
],
},
});
});
it("projects a small native view from an authenticated >10 MiB retained dossier", () => { it("projects a small native view from an authenticated >10 MiB retained dossier", () => {
const original = functionDossierSchema.parse(ghidraFunctionDossier()); const original = functionDossierSchema.parse(ghidraFunctionDossier());
const parent = createEvidence( const parent = createEvidence(
@@ -132,11 +132,8 @@ const resolveAnalysisViewSource = (
return err( return err(
analysisInputErrorFromIssues( analysisInputErrorFromIssues(
INSPECT_ANALYSIS_VIEW_OPERATION, INSPECT_ANALYSIS_VIEW_OPERATION,
cause.issues.map((issue) => ({ cause.issues,
...issue, source.evidence,
path: ["source", "evidence", ...issue.path],
})),
{ source },
{ cause }, { cause },
), ),
); );
@@ -2,10 +2,9 @@ import type { ApplicationNode } from "../../domain/javascript/javascriptApplicat
import { sha256Text } from "../../domain/javascript/javascriptStaticAnalysisHelpers.js"; import { sha256Text } from "../../domain/javascript/javascriptStaticAnalysisHelpers.js";
import type { JavaScriptSourceRange } from "../../domain/javascript/javascriptStaticAnalysisTypes.js"; import type { JavaScriptSourceRange } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js"; import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import { import type {
sourceNodeFor, JavaScriptArtifactGraphContext,
type JavaScriptArtifactGraphContext, JavaScriptArtifactGraphCoverage,
type JavaScriptArtifactGraphCoverage,
} from "./JavaScriptArtifactGraphContext.js"; } from "./JavaScriptArtifactGraphContext.js";
import { import {
astObservationEvidence, astObservationEvidence,
@@ -39,11 +38,10 @@ export const electronFindingSourceNode = (
context: JavaScriptArtifactGraphContext, context: JavaScriptArtifactGraphContext,
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
moduleKey: string | null, moduleKey: string | null,
runtime?: string,
): ApplicationNode | undefined => ): ApplicationNode | undefined =>
moduleKey === null moduleKey === null
? (context.assetNodes.get(file.path) ?? context.fileNodes.get(file.path)) ? (context.assetNodes.get(file.path) ?? context.fileNodes.get(file.path))
: sourceNodeFor(context, file.path, moduleKey, runtime); : context.moduleNodes.get(`${file.path}\0${moduleKey}`);
/** Observation-scoped identity shared safely across different artifact files. */ /** Observation-scoped identity shared safely across different artifact files. */
export const electronObservationIdentity = ( export const electronObservationIdentity = (
@@ -73,7 +73,6 @@ const addIpcRecord = (
context, context,
record.file, record.file,
record.finding.module_key, record.finding.module_key,
record.finding.module_runtime,
); );
if (source === undefined) return; if (source === undefined) return;
state.sources.set(record.key, source); state.sources.set(record.key, source);
@@ -256,7 +255,6 @@ const addValidationObservations = (state: IpcGraphState): void => {
state.context, state.context,
file, file,
validation.module_key, validation.module_key,
validation.module_runtime,
) )
: state.handlers.get(handlerRecord.key); : state.handlers.get(handlerRecord.key);
if (target !== undefined) if (target !== undefined)
@@ -38,12 +38,7 @@ export const addElectronNativeBoundaries = (
const addNativeBinding = (input: NativeBindingInput): void => { const addNativeBinding = (input: NativeBindingInput): void => {
const { context, file, value, coverage } = input; const { context, file, value, coverage } = input;
const source = electronFindingSourceNode( const source = electronFindingSourceNode(context, file, value.module_key);
context,
file,
value.module_key,
value.module_runtime,
);
if (source === undefined) return; if (source === undefined) return;
const resolved = resolveArtifactPath( const resolved = resolveArtifactPath(
value.specifier, value.specifier,
@@ -39,32 +39,17 @@ export const addElectronWindowBoundaries = (
if (javascript === null) continue; if (javascript === null) continue;
const coverage = javascriptAnalysisCoverage(javascript); const coverage = javascriptAnalysisCoverage(javascript);
for (const value of javascript.electron.browser_windows) { for (const value of javascript.electron.browser_windows) {
const source = electronFindingSourceNode( const source = electronFindingSourceNode(context, file, value.module_key);
context,
file,
value.module_key,
value.module_runtime,
);
if (source !== undefined) if (source !== undefined)
addBrowserWindow({ context, file, source, value, coverage }); addBrowserWindow({ context, file, source, value, coverage });
} }
for (const value of javascript.electron.context_bridge_apis) { for (const value of javascript.electron.context_bridge_apis) {
const source = electronFindingSourceNode( const source = electronFindingSourceNode(context, file, value.module_key);
context,
file,
value.module_key,
value.module_runtime,
);
if (source !== undefined) if (source !== undefined)
addContextBridge({ context, file, source, value, coverage }); addContextBridge({ context, file, source, value, coverage });
} }
for (const value of javascript.electron.utility_processes) { for (const value of javascript.electron.utility_processes) {
const source = electronFindingSourceNode( const source = electronFindingSourceNode(context, file, value.module_key);
context,
file,
value.module_key,
value.module_runtime,
);
if (source !== undefined) if (source !== undefined)
addUtilityProcess({ context, file, source, value, coverage }); addUtilityProcess({ context, file, source, value, coverage });
} }
@@ -2,11 +2,11 @@ import { setImmediate } from "node:timers";
import { writeFile } from "node:fs/promises"; import { writeFile } from "node:fs/promises";
import { join } from "node:path"; import { join } from "node:path";
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import { analyzeJavaScriptApplication } from "../../support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
describe("JavaScript application cancellation", () => { describe("JavaScript analysis cancellation before publication", () => {
it("interrupts semantic graph commitment before result validation", async () => { it("interrupts semantic graph commitment before result validation", async () => {
const root = await createTestTempDirectory("rea-js-cancel-semantic-"); const root = await createTestTempDirectory("rea-js-cancel-semantic-");
await writeFile( await writeFile(
@@ -33,9 +33,7 @@ describe("JavaScript application cancellation", () => {
}, },
); );
if (result.ok) if (result.ok)
throw new Error( throw new Error("Cancelled commitment must not publish Evidence");
"Cancelled commitment must not publish a successful analysis",
);
expect(projectAnalysisError(result.error)).toMatchObject({ expect(projectAnalysisError(result.error)).toMatchObject({
code: "cancelled", code: "cancelled",
details: { reason: "cancelled" }, details: { reason: "cancelled" },
@@ -44,7 +42,7 @@ describe("JavaScript application cancellation", () => {
expect(phases).not.toContain("validate_javascript_application_result"); expect(phases).not.toContain("validate_javascript_application_result");
}); });
it("honors control turns and cancellation while preparing one large source for owned analysis (#1462)", async () => { it("handles control turns and cancellation inside one large source file (#1462)", async () => {
const root = await createTestTempDirectory("rea-js-cancel-in-file-"); const root = await createTestTempDirectory("rea-js-cancel-in-file-");
await writeFile( await writeFile(
join(root, "index.js"), join(root, "index.js"),
@@ -76,7 +74,7 @@ describe("JavaScript application cancellation", () => {
}, },
}, },
); );
if (result.ok) throw new Error("Cancelled analysis must not succeed"); if (result.ok) throw new Error("Cancelled analysis must not publish");
expect(projectAnalysisError(result.error)).toMatchObject({ expect(projectAnalysisError(result.error)).toMatchObject({
code: "cancelled", code: "cancelled",
}); });
@@ -120,21 +118,12 @@ describe("JavaScript application cancellation", () => {
}, },
); );
expect(requested).toBe(true); expect(requested).toBe(true);
if (result.ok) throw new Error("Cancelled analysis must return a failure"); if (result.ok)
const failure = projectAnalysisError(result.error); throw new Error("Cancelled analysis must not publish Evidence");
expect(failure).toMatchObject({ code: "cancelled" }); expect(projectAnalysisError(result.error)).toMatchObject({
if (phase === "parse_javascript_source") code: "cancelled",
expect(failure).toMatchObject({ details: { reason: "cancelled" },
details: { });
operation: "analyze_javascript_application",
cleanup: "complete",
partial_observation: {
normalized_result: {
statistics: { relevant_files: 1, parse_failures: 0 },
},
},
},
});
expect(terminal).toEqual([]); expect(terminal).toEqual([]);
}); });
@@ -10,7 +10,7 @@ import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascri
import { matchJavaScriptApplicationVersions } from "../../domain/javascript/javascriptApplicationVersionKeys.js"; import { matchJavaScriptApplicationVersions } from "../../domain/javascript/javascriptApplicationVersionKeys.js";
import { findApplicationFeatureSeeds } from "../../domain/javascript/javascriptFeatureSeed.js"; import { findApplicationFeatureSeeds } from "../../domain/javascript/javascriptFeatureSeed.js";
import { traceApplicationFeatureInputSchema } from "../../domain/javascript/javascriptFeatureTraceSchemas.js"; import { traceApplicationFeatureInputSchema } from "../../domain/javascript/javascriptFeatureTraceSchemas.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
const analyzeSource = async ( const analyzeSource = async (
source: string, source: string,
@@ -5,7 +5,7 @@ import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
it.each([ it.each([
{ {
@@ -4,11 +4,9 @@ import { join } from "node:path";
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { z } from "zod"; import { z } from "zod";
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import { scanCanonicalArtifactInventory } from "../../../tests/fixtures/artifactInventory.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js"; import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
describe("JavaScript application failure diagnostics", () => { describe("JavaScript application failure diagnostics", () => {
it("identifies the rejected result field in caller-visible diagnostics", async () => { it("identifies the rejected result field in caller-visible diagnostics", async () => {
@@ -155,41 +153,3 @@ describe("JavaScript application failure diagnostics", () => {
}, },
); );
}); });
describe("JavaScript application cleanup diagnostics", () => {
it("preserves cleanup and a complete inventory when translating reader failures", async () => {
const inputPath = await createTestTempDirectory("rea-js-cleanup-failure-");
await writeFile(join(inputPath, "main.js"), "export const value = 1;\n");
const inventory = await scanCanonicalArtifactInventory(inputPath);
const cleanup = {
reason: "reader remains open",
resources: [`JavaScript artifact reader for ${inputPath}`],
};
const partialObservation = {
kind: "artifact-inventory" as const,
inventory,
};
const failure = new ArtifactReaderFailure("io", "Reader cleanup failed", {
cleanup,
partialObservation,
});
const result = await analyzeJavaScriptApplication(
{ input_path: inputPath, format: "directory" },
{
progress: {
report: async () => {
throw failure;
},
},
},
);
if (result.ok) throw new Error("Expected reader failure");
expect(result.error).toMatchObject({
_tag: "ArtifactOperationError",
cleanup,
partialObservation,
});
});
});
@@ -1,14 +1,18 @@
import { z } from "zod"; import { z } from "zod";
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js"; import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
import { import {
analyzeJavaScriptApplicationInputSchema, analyzeJavaScriptApplicationInputSchema,
parseOwnedJavaScriptApplicationAnalysisSteps, parseOwnedJavaScriptApplicationAnalysisSteps,
} from "../../domain/javascript/javascriptApplicationAnalysis.js"; } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { AnalysisOutputError } from "../../domain/analysisErrorCore.js"; import {
AnalysisAccessDeniedError,
AnalysisInputError,
AnalysisOutputError,
AnalysisUnsupportedTargetError,
} from "../../domain/analysisErrorCore.js";
import { ArtifactOperationError } from "../../domain/artifactOperationError.js"; import { ArtifactOperationError } from "../../domain/artifactOperationError.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js"; import { type AnalysisError } from "../../domain/analysisErrorBase.js";
import type { Evidence } from "../../domain/evidence.js"; import type { Evidence } from "../../domain/evidence.js";
import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js"; import { analysisInputErrorFromIssues } from "../../domain/inputIssueProjection.js";
import { err, ok, type Result } from "../../domain/result.js"; import { err, ok, type Result } from "../../domain/result.js";
@@ -28,7 +32,6 @@ const OPERATION = "analyze_javascript_application" as const;
/** Statically analyze one local JavaScript/Electron application. */ /** Statically analyze one local JavaScript/Electron application. */
export const analyzeJavaScriptApplication = async ( export const analyzeJavaScriptApplication = async (
rawInput: unknown, rawInput: unknown,
scope: ArtifactResourceScope,
options: ExecutionOptions = {}, options: ExecutionOptions = {},
): Promise<Result<Evidence, AnalysisError>> => { ): Promise<Result<Evidence, AnalysisError>> => {
const parsed = analyzeJavaScriptApplicationInputSchema.safeParse(rawInput); const parsed = analyzeJavaScriptApplicationInputSchema.safeParse(rawInput);
@@ -36,13 +39,12 @@ export const analyzeJavaScriptApplication = async (
return err( return err(
analysisInputErrorFromIssues(OPERATION, parsed.error.issues, rawInput), analysisInputErrorFromIssues(OPERATION, parsed.error.issues, rawInput),
); );
return analyzeJavaScriptApplicationValidated(parsed.data, scope, options); return analyzeJavaScriptApplicationValidated(parsed.data, options);
}; };
/** Analyze input already parsed by a trusted adapter boundary. */ /** Analyze input already parsed by a trusted adapter boundary. */
export const analyzeJavaScriptApplicationValidated = async ( export const analyzeJavaScriptApplicationValidated = async (
input: z.output<typeof analyzeJavaScriptApplicationInputSchema>, input: z.output<typeof analyzeJavaScriptApplicationInputSchema>,
scope: ArtifactResourceScope,
options: ExecutionOptions = {}, options: ExecutionOptions = {},
): Promise<Result<Evidence, AnalysisError>> => { ): Promise<Result<Evidence, AnalysisError>> => {
try { try {
@@ -59,14 +61,7 @@ export const analyzeJavaScriptApplicationValidated = async (
input_path: input.input_path, input_path: input.input_path,
format: input.format, format: input.format,
integrity_policy: input.integrity_policy, integrity_policy: input.integrity_policy,
...(input.max_heap_mb === undefined
? {}
: { max_heap_mb: input.max_heap_mb }),
...(input.analysis_timeout_ms === undefined
? {}
: { analysis_timeout_ms: input.analysis_timeout_ms }),
}, },
scope,
options.signal, options.signal,
options.progress, options.progress,
); );
@@ -111,10 +106,22 @@ export const analyzeJavaScriptApplicationValidated = async (
assertJavaScriptAnalysisActive(options.signal); assertJavaScriptAnalysisActive(options.signal);
return ok(evidence); return ok(evidence);
} catch (cause: unknown) { } catch (cause: unknown) {
// Preserve execution constraints and their completed partial Evidence. // Input selection failures are already typed by their cause.
if (cause instanceof AnalysisError) return err(cause); if (
cause instanceof AnalysisInputError ||
cause instanceof AnalysisUnsupportedTargetError ||
cause instanceof AnalysisAccessDeniedError
)
return err(cause);
if (cause instanceof ArtifactReaderFailure) if (cause instanceof ArtifactReaderFailure)
return err(artifactFailureToOperationError(cause)); return err(
new ArtifactOperationError(
OPERATION,
cause.reason,
cause.details,
cause.message,
),
);
if (cause instanceof z.ZodError) if (cause instanceof z.ZodError)
return err( return err(
new AnalysisOutputError(OPERATION, describeResultSchemaFailure(cause), { new AnalysisOutputError(OPERATION, describeResultSchemaFailure(cause), {
@@ -146,23 +153,6 @@ export const analyzeJavaScriptApplicationValidated = async (
} }
}; };
const artifactFailureToOperationError = (
cause: ArtifactReaderFailure,
): ArtifactOperationError =>
new ArtifactOperationError(
OPERATION,
cause.reason,
cause.details,
cause.message,
{
cause,
...(cause.cleanup === undefined ? {} : { cleanup: cause.cleanup }),
...(cause.partialObservation === undefined
? {}
: { partialObservation: cause.partialObservation }),
},
);
const describeResultSchemaFailure = (cause: z.ZodError): string => { const describeResultSchemaFailure = (cause: z.ZodError): string => {
const issue = cause.issues[0]; const issue = cause.issues[0];
if (issue === undefined) return "Result schema rejected an unknown issue"; if (issue === undefined) return "Result schema rejected an unknown issue";
@@ -7,7 +7,7 @@ import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirect
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { javaScriptSemanticQueryResultSchema } from "../../domain/javascript/javascriptSemanticQuerySchemas.js"; import { javaScriptSemanticQueryResultSchema } from "../../domain/javascript/javascriptSemanticQuerySchemas.js";
import { queryJavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticQuery.js"; import { queryJavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticQuery.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
it.each([ it.each([
{ {
@@ -5,7 +5,7 @@ import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
const graphIdentities = async (source: string) => { const graphIdentities = async (source: string) => {
const inputPath = await createTestTempDirectory("rea-js-template-literal-"); const inputPath = await createTestTempDirectory("rea-js-template-literal-");
@@ -31,16 +31,7 @@ const graphIdentities = async (source: string) => {
...(identity.strategy === "artifact-local-key" ...(identity.strategy === "artifact-local-key"
? { ? {
namespace: identity.namespace, namespace: identity.namespace,
// Module-export identity is a JSON tuple whose first entry key: identity.key.replace(/^[0-9a-f]{64}:/u, ""),
// binds the whole container; only that digest changes here.
key:
identity.namespace === "module-export"
? JSON.stringify(
(JSON.parse(identity.key) as unknown[]).slice(1),
)
: identity.namespace === "source-module"
? identity.key.replace(/^[0-9a-f]{64}:/u, "")
: identity.key,
} }
: {}), : {}),
})) }))
@@ -7,7 +7,7 @@ import { canonicalJson } from "../../domain/comparisonSemantics.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js"; import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
type ApplicationAnalysis = ReturnType< type ApplicationAnalysis = ReturnType<
typeof javascriptApplicationAnalysisResultSchema.parse typeof javascriptApplicationAnalysisResultSchema.parse
@@ -7,6 +7,7 @@ import type { ApplicationVersionComparisonResult } from "../../domain/javascript
import type { JavaScriptExportShapeComparisonResult } from "../../domain/javascript/javascriptExportShapeComparisonSchemas.js"; import type { JavaScriptExportShapeComparisonResult } from "../../domain/javascript/javascriptExportShapeComparisonSchemas.js";
import type { ApplicationFeatureTraceResult } from "../../domain/javascript/javascriptFeatureTraceSchemas.js"; import type { ApplicationFeatureTraceResult } from "../../domain/javascript/javascriptFeatureTraceSchemas.js";
import type { JavaScriptSemanticTraceResult } from "../../domain/javascript/javascriptSemanticTraceSchemas.js"; import type { JavaScriptSemanticTraceResult } from "../../domain/javascript/javascriptSemanticTraceSchemas.js";
import { jsonValueSchema } from "../../domain/jsonValue.js";
import type { SourceToBundleComparisonResult } from "../../domain/javascript/sourceToBundleComparisonSchemas.js"; import type { SourceToBundleComparisonResult } from "../../domain/javascript/sourceToBundleComparisonSchemas.js";
import { JAVASCRIPT_APPLICATION_WORKFLOW_PROVIDER } from "../InvestigationProviders.js"; import { JAVASCRIPT_APPLICATION_WORKFLOW_PROVIDER } from "../InvestigationProviders.js";
@@ -19,7 +20,7 @@ export const createApplicationFeatureTraceEvidence = (
predicateType: "rea.application-feature-trace", predicateType: "rea.application-feature-trace",
operation: "trace_application_feature", operation: "trace_application_feature",
parameters, parameters,
result, result: jsonValueSchema.parse(result),
rawResult: null, rawResult: null,
confidence: "inferred", confidence: "inferred",
authority: "analyst-inference", authority: "analyst-inference",
@@ -37,7 +38,7 @@ export const createJavaScriptSemanticTraceEvidence = (
predicateType: "rea.javascript-semantic-trace", predicateType: "rea.javascript-semantic-trace",
operation: "trace_javascript_semantics", operation: "trace_javascript_semantics",
parameters, parameters,
result, result: jsonValueSchema.parse(result),
rawResult: null, rawResult: null,
confidence: "inferred", confidence: "inferred",
authority: "analyst-inference", authority: "analyst-inference",
@@ -55,7 +56,7 @@ export const createApplicationVersionComparisonEvidence = (
predicateType: "rea.application-version-comparison", predicateType: "rea.application-version-comparison",
operation: "compare_application_versions", operation: "compare_application_versions",
parameters, parameters,
result, result: jsonValueSchema.parse(result),
rawResult: null, rawResult: null,
confidence: "inferred", confidence: "inferred",
authority: "analyst-inference", authority: "analyst-inference",
@@ -73,7 +74,7 @@ export const createSourceToBundleComparisonEvidence = (
predicateType: "rea.source-to-bundle-comparison", predicateType: "rea.source-to-bundle-comparison",
operation: "compare_source_to_bundle", operation: "compare_source_to_bundle",
parameters, parameters,
result, result: jsonValueSchema.parse(result),
rawResult: null, rawResult: null,
confidence: "inferred", confidence: "inferred",
authority: "analyst-inference", authority: "analyst-inference",
@@ -91,7 +92,7 @@ export const createJavaScriptExportShapeComparisonEvidence = (
predicateType: "rea.javascript-export-shape-comparison", predicateType: "rea.javascript-export-shape-comparison",
operation: "compare_javascript_export_shapes", operation: "compare_javascript_export_shapes",
parameters, parameters,
result, result: jsonValueSchema.parse(result),
rawResult: null, rawResult: null,
confidence: "inferred", confidence: "inferred",
authority: "analyst-inference", authority: "analyst-inference",
@@ -8,7 +8,6 @@ import {
classifyParsedJavaScriptOpenReceiversSteps, classifyParsedJavaScriptOpenReceiversSteps,
} from "../../domain/javascript/javascriptSemanticAnalysis.js"; } from "../../domain/javascript/javascriptSemanticAnalysis.js";
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js"; import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
import { projectJavaScriptModuleSemantics } from "../../domain/javascript/javascriptModuleSemanticIr.js";
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js"; import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js"; import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js";
import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js"; import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js";
@@ -36,15 +35,6 @@ import type {
import { analyzeJavaScriptJsonModule } from "./JavaScriptJsonModules.js"; import { analyzeJavaScriptJsonModule } from "./JavaScriptJsonModules.js";
import { completeJavaScriptAnalysisSteps } from "./JavaScriptAnalysisControl.js"; import { completeJavaScriptAnalysisSteps } from "./JavaScriptAnalysisControl.js";
import { htmlArtifactReferences } from "../../domain/javascript/htmlArtifactReferences.js"; import { htmlArtifactReferences } from "../../domain/javascript/htmlArtifactReferences.js";
import type { JavaScriptSourceAnalysisPort } from "../../domain/javascript/javascriptSourceAnalysis.js";
import type { JavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import {
AnalysisCancelledError,
AnalysisResourceConstraintError,
AnalysisTimeoutError,
} from "../../domain/analysisErrorCore.js";
interface MutableArtifactAnalysis< interface MutableArtifactAnalysis<
SemanticIr extends JavaScriptModuleSemanticIr, SemanticIr extends JavaScriptModuleSemanticIr,
@@ -105,7 +95,27 @@ export const analyzeAndProjectJavaScriptArtifactFiles = async (
): Generator<void, JavaScriptModuleSemanticIr> { ): Generator<void, JavaScriptModuleSemanticIr> {
const projection = projectSemantics(file, ir); const projection = projectSemantics(file, ir);
if (projection !== undefined) while (projection.next().done !== true) yield; if (projection !== undefined) while (projection.next().done !== true) yield;
return projectJavaScriptModuleSemantics(ir); const programScopes = ir.scopes.filter(({ kind }) => kind === "program");
const programScopeId = programScopes[0]?.scopeId;
const localNames = new Set(
ir.moduleLinks.map(({ localName }) => localName),
);
const callableIds = new Set(
ir.moduleLinks.map(({ callableId }) => callableId),
);
return {
scopes: programScopes,
bindings: ir.bindings.filter(
({ name, scopeId }) =>
scopeId === programScopeId && localNames.has(name),
),
callables: ir.callables.filter(({ callableId }) =>
callableIds.has(callableId),
),
moduleLinks: ir.moduleLinks,
coverage: ir.coverage,
limitations: ir.limitations,
};
}; };
const context = { state, projectSemantics: projectFileSemantics }; const context = { state, projectSemantics: projectFileSemantics };
for (const [index, file] of fileSet.files.entries()) { for (const [index, file] of fileSet.files.entries()) {
@@ -120,174 +130,6 @@ export const analyzeAndProjectJavaScriptArtifactFiles = async (
return finalizeArtifactAnalysis(state); return finalizeArtifactAnalysis(state);
}; };
/** Analyze sources behind a hard heap boundary, retaining collected facts on interruption. */
export const analyzeAndAdoptJavaScriptArtifactFiles = async (
fileSet: JavaScriptArtifactFileSet,
worker: Pick<JavaScriptSourceAnalysisPort, "analyze">,
projection: JavaScriptSemanticGraphProjection,
beforeFile: (
file: JavaScriptArtifactFile,
completed: number,
total: number,
) => Promise<void>,
signal?: AbortSignal,
): Promise<{
readonly analysis: JavaScriptModuleArtifactAnalysis;
readonly interruption: AnalysisError | null;
}> => {
const state = emptyArtifactAnalysis<JavaScriptModuleSemanticIr>();
let interruption: AnalysisError | null = null;
for (const [index, file] of fileSet.files.entries()) {
try {
await setImmediate();
await beforeFile(file, index, fileSet.files.length);
if (signal?.aborted === true)
throw new AnalysisCancelledError("analyze_javascript_application");
if (file.kind !== "javascript" || !file.text.included) {
const steps = analyzeArtifactFileSteps(file, {
state,
projectSemantics: function* (_file, ir) {
yield;
return projectJavaScriptModuleSemantics(ir);
},
});
await completeJavaScriptAnalysisSteps(steps, signal);
continue;
}
const result = await worker.analyze(
file,
projection.remainingFileNodeBudget(),
signal,
);
const facts = result.ok ? result.value : result.error;
const applicationProjectionFailure = facts.applicationProjectionFailure;
if (facts.projection !== null)
projection.adoptFileProjection(facts.projection);
const error = result.ok ? null : result.error.error;
const limits =
error instanceof AnalysisResourceConstraintError
? error.reportedLimits
: error instanceof AnalysisTimeoutError
? { worker_timeout_ms: error.timeoutMs }
: null;
const failure =
error === null
? undefined
: {
reason: `${file.path}: ${error.message}`,
error: projectAnalysisError(error),
limits,
};
state.files.push({
file,
javascript: facts.javascript,
semantic: facts.module === null ? null : { ir: facts.module },
...(failure === undefined ? {} : { analysis_failure: failure }),
...(applicationProjectionFailure === undefined
? {}
: {
application_projection_failure: {
reason: applicationProjectionFailure.message,
error: projectAnalysisError(applicationProjectionFailure),
limits: applicationProjectionFailure.reportedLimits,
},
}),
});
if (failure !== undefined && facts.projection === null)
projection.recordUnavailableFile(
file,
failure.reason,
failure.limits,
error instanceof AnalysisResourceConstraintError ||
error instanceof AnalysisTimeoutError
? "resource-limit"
: "incomplete-module",
);
if (facts.javascript !== null) {
state.visitedNodes += facts.javascript.visited_ast_nodes;
state.findings +=
findingCount(facts.javascript) +
(facts.module?.moduleLinks.length ?? 0);
state.modules += facts.javascript.bundler_registrations.reduce(
(count, item) => count + item.modules.length,
0,
);
if (facts.javascript.parse_status === "failed")
state.parseFailures += 1;
}
if (
error !== null &&
(!(error instanceof AnalysisResourceConstraintError) ||
error.reportedLimits?.parent_result_budget_exhausted === true)
) {
interruption = error;
break;
}
} catch (cause: unknown) {
if (signal?.aborted !== true && !(cause instanceof AnalysisError))
throw cause;
interruption =
signal?.aborted === true
? new AnalysisCancelledError("analyze_javascript_application")
: cause instanceof AnalysisError
? cause
: null;
if (interruption !== null) {
state.files.push({
file,
javascript: null,
semantic: null,
analysis_failure: {
reason: `${file.path}: ${interruption.message}`,
error: projectAnalysisError(interruption),
limits: null,
},
});
projection.recordUnavailableFile(
file,
interruption.message,
null,
"incomplete-module",
);
}
break;
}
}
if (interruption !== null) {
const analyzedPaths = new Set(state.files.map(({ file }) => file.path));
const failure = {
reason: `Source analysis was interrupted: ${interruption.message}`,
error: projectAnalysisError(interruption),
limits:
interruption instanceof AnalysisResourceConstraintError
? interruption.reportedLimits
: interruption instanceof AnalysisTimeoutError
? { worker_timeout_ms: interruption.timeoutMs }
: null,
};
for (const file of fileSet.files) {
if (analyzedPaths.has(file.path)) continue;
state.files.push({
file,
javascript: null,
semantic: null,
analysis_failure: failure,
});
if (file.kind === "javascript")
projection.recordUnavailableFile(
file,
failure.reason,
failure.limits,
interruption instanceof AnalysisResourceConstraintError ||
interruption instanceof AnalysisTimeoutError
? "resource-limit"
: "incomplete-module",
);
}
}
return { analysis: finalizeArtifactAnalysis(state), interruption };
};
const analyzeArtifactFiles = <SemanticIr extends JavaScriptModuleSemanticIr>( const analyzeArtifactFiles = <SemanticIr extends JavaScriptModuleSemanticIr>(
fileSet: JavaScriptArtifactFileSet, fileSet: JavaScriptArtifactFileSet,
projectSemantics: ArtifactAnalysisContext<SemanticIr>["projectSemantics"], projectSemantics: ArtifactAnalysisContext<SemanticIr>["projectSemantics"],
@@ -319,12 +161,6 @@ const finalizeArtifactAnalysis = <
limitations: [ limitations: [
"JavaScript and HTML were parsed as inert text; bundle bootstrap code was never executed.", "JavaScript and HTML were parsed as inert text; bundle bootstrap code was never executed.",
"Static paths and relationships may remain unresolved when expressions are dynamic or obfuscated.", "Static paths and relationships may remain unresolved when expressions are dynamic or obfuscated.",
...state.files.flatMap(
({ analysis_failure, application_projection_failure }) =>
[analysis_failure, application_projection_failure].flatMap(
(failure) => (failure === undefined ? [] : [failure.reason]),
),
),
], ],
}; };
}; };
@@ -4,9 +4,7 @@ import type {
JavaScriptStaticAnalysis, JavaScriptStaticAnalysis,
} from "../../domain/javascript/javascriptStaticAnalysisTypes.js"; } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js"; import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSemanticIr.js";
import type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js"; import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import type { JsonValue } from "../../domain/jsonValue.js";
interface JavaScriptStructuredObservation { interface JavaScriptStructuredObservation {
readonly path: string; readonly path: string;
@@ -91,7 +89,15 @@ export type JavaScriptJsonModuleObservation = JavaScriptStructuredObservation &
); );
/** Module facts needed after file-local semantic projection has finished. */ /** Module facts needed after file-local semantic projection has finished. */
export type { JavaScriptModuleSemanticIr } from "../../domain/javascript/javascriptModuleSemanticIr.js"; export type JavaScriptModuleSemanticIr = Pick<
JavaScriptSemanticIr,
| "scopes"
| "bindings"
| "callables"
| "moduleLinks"
| "coverage"
| "limitations"
>;
/** One relevant file plus optional AST-only JavaScript facts. */ /** One relevant file plus optional AST-only JavaScript facts. */
export interface AnalyzedJavaScriptArtifactFile< export interface AnalyzedJavaScriptArtifactFile<
@@ -102,18 +108,6 @@ export interface AnalyzedJavaScriptArtifactFile<
readonly semantic: { readonly semantic: {
readonly ir: SemanticIr; readonly ir: SemanticIr;
} | null; } | null;
/** Operational incompleteness, distinct from a malformed source file. */
readonly analysis_failure?: {
readonly reason: string;
readonly error: JsonValue;
readonly limits: Readonly<Record<string, JsonValue>> | null;
};
/** Facts retained inline when their expanded graph exceeds available memory. */
readonly application_projection_failure?: {
readonly reason: string;
readonly error: JsonValue;
readonly limits: Readonly<Record<string, JsonValue>> | null;
};
} }
/** Complete static-analysis projection before graph construction. */ /** Complete static-analysis projection before graph construction. */
@@ -12,7 +12,7 @@ export class JavaScriptArtifactGraphAccumulator {
readonly #nodes = new Map<string, ApplicationNode>(); readonly #nodes = new Map<string, ApplicationNode>();
readonly #edges = new Map<string, ApplicationEdge>(); readonly #edges = new Map<string, ApplicationEdge>();
/** Create or merge one canonical node, retaining a stable live construction reference. */ /** Create or merge one canonical node. */
addNode<Input extends DisplayableNodeInput>(input: Input): ApplicationNode { addNode<Input extends DisplayableNodeInput>(input: Input): ApplicationNode {
const created = createJavaScriptApplicationNode(displayableNode(input)); const created = createJavaScriptApplicationNode(displayableNode(input));
const existing = this.#nodes.get(created.node_id); const existing = this.#nodes.get(created.node_id);
@@ -23,23 +23,19 @@ export class JavaScriptArtifactGraphAccumulator {
const bySemanticId = new Map( const bySemanticId = new Map(
[...existing.observations, ...created.observations].map((observation) => [ [...existing.observations, ...created.observations].map((observation) => [
observation.observation_id, observation.observation_id,
observation, observationInput(observation),
]), ]),
); );
const observations = [...bySemanticId.values()].sort((left, right) => const observations = [...bySemanticId.entries()]
left.observation_id < right.observation_id .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0))
? -1 .map(([, observation]) => observation);
: left.observation_id > right.observation_id const merged = createJavaScriptApplicationNode({
? 1 kind: existing.kind,
: 0, identity: existing.identity,
); observations,
// Both factories already validated and identified these observations. });
// Replacing the node retained every previous observation array through this.#nodes.set(merged.node_id, merged);
// path/chunk/IPC indexes when multiple paths shared an entity identity. return merged;
existing.observations.length = 0;
for (const observation of observations)
existing.observations.push(observation);
return existing;
} }
/** Create or replace one canonical edge by semantic identifier. */ /** Create or replace one canonical edge by semantic identifier. */
@@ -60,6 +56,17 @@ export class JavaScriptArtifactGraphAccumulator {
} }
} }
const observationInput = (
observation: ApplicationNode["observations"][number],
) => {
const {
observation_id: _observationId,
identifier_strategy: _strategy,
...input
} = observation;
return input;
};
/** Node fields whose display form is normalized before validation. */ /** Node fields whose display form is normalized before validation. */
interface DisplayableNodeInput { interface DisplayableNodeInput {
readonly observations: readonly { readonly label: string | null }[]; readonly observations: readonly { readonly label: string | null }[];
@@ -107,33 +107,19 @@ function* buildJavaScriptArtifactGraphInputSteps(
yield; yield;
addJavaScriptArtifactFiles(context); addJavaScriptArtifactFiles(context);
yield; yield;
// Raw observations remain on the file nodes. Only admitted expansions feed const packageRoots = addJavaScriptPackageNodes(context);
// the relationship builders, including their Electron and bundler callers.
const projectionContext = {
...context,
analysis: {
...analysis,
files: analysis.files.map((file) =>
file.application_projection_failure === undefined
? file
: { ...file, javascript: null, semantic: null },
),
},
};
const packageRoots = addJavaScriptPackageNodes(projectionContext);
yield; yield;
addJavaScriptSourceModules(projectionContext); addJavaScriptSourceModules(context);
yield; yield;
const bundlerLimitations = addJavaScriptBundlerNodes(projectionContext); const bundlerLimitations = addJavaScriptBundlerNodes(context);
const relationshipOmissions = const relationshipOmissions =
yield* addJavaScriptModuleRelationshipsSteps(projectionContext); yield* addJavaScriptModuleRelationshipsSteps(context);
const findingLimitations = const findingLimitations = yield* addJavaScriptStaticFindingsSteps(context);
yield* addJavaScriptStaticFindingsSteps(projectionContext); addElectronBoundaries(context);
addElectronBoundaries(projectionContext);
yield; yield;
addJavaScriptHtmlRoles(projectionContext); addJavaScriptHtmlRoles(context);
yield; yield;
addJavaScriptSourceMapOriginals(projectionContext); addJavaScriptSourceMapOriginals(context);
yield; yield;
const coverage = graphCoverage(context); const coverage = graphCoverage(context);
return { return {
@@ -167,34 +153,14 @@ const graphCoverage = (context: JavaScriptArtifactGraphContext) => {
javascript !== null && javascript.parse_status === "partial", javascript !== null && javascript.parse_status === "partial",
); );
const unknownGap = const unknownGap =
context.analysis.files.some(
({ analysis_failure, application_projection_failure }) =>
analysis_failure !== undefined ||
application_projection_failure !== undefined,
) ||
context.analysis.parse_failures > 0 || context.analysis.parse_failures > 0 ||
context.fileSet.invalid_utf8_files > 0 || context.fileSet.invalid_utf8_files > 0 ||
sourceMapPolicyGap || sourceMapPolicyGap ||
malformedStructuredData || malformedStructuredData ||
partialJavaScript; partialJavaScript;
const executionLimits = context.analysis.files.flatMap( if (resourceLimits.length > 0)
({ analysis_failure, application_projection_failure }) =>
[analysis_failure, application_projection_failure].flatMap((failure) =>
failure === undefined
? []
: javaScriptAnalysisCoverageLimits(failure.limits),
),
);
if (resourceLimits.length > 0 || executionLimits.length > 0)
return partialApplicationCoverage( return partialApplicationCoverage(
[ semanticResourceLimitCoverage(resourceLimits),
...new Map(
[
...semanticResourceLimitCoverage(resourceLimits),
...executionLimits,
].map((limit) => [JSON.stringify(limit), limit]),
).values(),
],
null, null,
); );
if (unknownGap) return partialApplicationCoverage([], null); if (unknownGap) return partialApplicationCoverage([], null);
@@ -294,4 +260,3 @@ const graphLimitations = (
]), ]),
]; ];
}; };
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
@@ -161,7 +161,7 @@ const addBundlerModuleNodes = (
], ],
}); });
context.moduleNodes.set( context.moduleNodes.set(
moduleLookupKey(file.path, registration.runtime, moduleValue.module_key), moduleLookupKey(file.path, moduleValue.module_key),
module, module,
); );
addAstContainsEdge(context, { addAstContainsEdge(context, {
@@ -195,7 +195,7 @@ const addBundlerRuntimeEdges = (
const { file, registration, chunk, coverage } = record; const { file, registration, chunk, coverage } = record;
for (const moduleKey of registration.entry_module_keys) { for (const moduleKey of registration.entry_module_keys) {
const resolved = context.moduleNodes.get( const resolved = context.moduleNodes.get(
moduleLookupKey(file.path, registration.runtime, moduleKey), moduleLookupKey(file.path, moduleKey),
); );
const target = const target =
resolved ?? unresolvedBundlerModuleNode(context, record, moduleKey); resolved ?? unresolvedBundlerModuleNode(context, record, moduleKey);
@@ -1,5 +1,4 @@
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js"; import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import type { JsonValue } from "../../domain/jsonValue.js";
import { compositeKey } from "../../domain/unicodeCodePointOrder.js"; import { compositeKey } from "../../domain/unicodeCodePointOrder.js";
import { import {
completeApplicationCoverage, completeApplicationCoverage,
@@ -100,8 +99,6 @@ export interface UnavailableParseInput {
readonly asset: ApplicationNode; readonly asset: ApplicationNode;
readonly operation: string; readonly operation: string;
readonly limitation: string; readonly limitation: string;
readonly failure?: JsonValue;
readonly limits?: JavaScriptArtifactGraphCoverage["limits"];
} }
/** Add a direct artifact-containment edge. */ /** Add a direct artifact-containment edge. */
@@ -185,15 +182,12 @@ export const addUnavailableStaticParseScope = (
path: input.file.path, path: input.file.path,
operation: input.operation, operation: input.operation,
limitation: input.limitation, limitation: input.limitation,
...(input.failure === undefined
? {}
: { analysis_failure: input.failure }),
}, },
evidence: unavailableAstEvidence({ evidence: unavailableAstEvidence({
sha256: input.file.sha256, sha256: input.file.sha256,
path: input.file.path, path: input.file.path,
operation: input.operation, operation: input.operation,
coverage: partialApplicationCoverage(input.limits ?? [], null), coverage: partialApplicationCoverage([], null),
limitation: input.limitation, limitation: input.limitation,
}), }),
}, },
@@ -329,11 +323,8 @@ export const selfReferenceOmissions = (
]; ];
/** Deterministic key for one recovered module inside a bundle asset. */ /** Deterministic key for one recovered module inside a bundle asset. */
export const moduleLookupKey = ( export const moduleLookupKey = (path: string, moduleKey: string): string =>
path: string, `${path}\0${moduleKey}`;
runtime: string,
moduleKey: string,
): string => compositeKey([path, runtime, moduleKey]);
/** Deterministic key for one recovered bundler chunk inside a bundle asset. */ /** Deterministic key for one recovered bundler chunk inside a bundle asset. */
export const chunkLookupKey = ( export const chunkLookupKey = (
@@ -362,10 +353,7 @@ export const sourceNodeFor = (
context: JavaScriptArtifactGraphContext, context: JavaScriptArtifactGraphContext,
path: string, path: string,
moduleKey: string | null, moduleKey: string | null,
runtime?: string,
): ApplicationNode | undefined => ): ApplicationNode | undefined =>
moduleKey === null moduleKey === null
? context.sourceModuleNodes.get(path) ? context.sourceModuleNodes.get(path)
: runtime === undefined : context.moduleNodes.get(moduleLookupKey(path, moduleKey));
? undefined
: context.moduleNodes.get(moduleLookupKey(path, runtime, moduleKey));
@@ -78,12 +78,8 @@ const addReference = (
omissions: { selfReferences: number }, omissions: { selfReferences: number },
): void => { ): void => {
const source = const source =
sourceNodeFor( sourceNodeFor(context, input.file.path, input.value.module_key) ??
context, input.asset;
input.file.path,
input.value.module_key,
input.value.module_runtime,
) ?? input.asset;
const resolved = const resolved =
input.value.specifier === null input.value.specifier === null
? null ? null
@@ -166,12 +162,8 @@ const addEndpoint = (
input: FindingInput<StaticEndpoint>, input: FindingInput<StaticEndpoint>,
): void => { ): void => {
const source = const source =
sourceNodeFor( sourceNodeFor(context, input.file.path, input.value.module_key) ??
context, input.asset;
input.file.path,
input.value.module_key,
input.value.module_runtime,
) ?? input.asset;
const key = compactStringIdentityKey(input.value.value); const key = compactStringIdentityKey(input.value.value);
const label = `${input.value.kind} endpoint`; const label = `${input.value.kind} endpoint`;
const node = context.accumulator.addNode({ const node = context.accumulator.addNode({
@@ -212,12 +204,8 @@ const addStorage = (
input: FindingInput<StaticStorage>, input: FindingInput<StaticStorage>,
): void => { ): void => {
const source = const source =
sourceNodeFor( sourceNodeFor(context, input.file.path, input.value.module_key) ??
context, input.asset;
input.file.path,
input.value.module_key,
input.value.module_runtime,
) ?? input.asset;
const node = context.accumulator.addNode({ const node = context.accumulator.addNode({
kind: "storage", kind: "storage",
identity: artifactLocalIdentity( identity: artifactLocalIdentity(
@@ -274,12 +262,8 @@ const addDiscoveredRole = (
coverage: input.coverage, coverage: input.coverage,
}); });
const source = const source =
sourceNodeFor( sourceNodeFor(context, input.file.path, input.value.module_key) ??
context, input.asset;
input.file.path,
input.value.module_key,
input.value.module_runtime,
) ?? input.asset;
addStaticInferenceEdge(context, { addStaticInferenceEdge(context, {
source, source,
target: role, target: role,
@@ -375,12 +359,9 @@ const resolveReference = (
if (specifier === null) return null; if (specifier === null) return null;
const chunk = resolveBundlerChunkReference(context, sourcePath, reference); const chunk = resolveBundlerChunkReference(context, sourcePath, reference);
if (chunk !== null) return chunk; if (chunk !== null) return chunk;
const module = const module = context.moduleNodes.get(
reference.module_runtime === undefined moduleLookupKey(sourcePath, specifier),
? undefined );
: context.moduleNodes.get(
moduleLookupKey(sourcePath, reference.module_runtime, specifier),
);
if (module !== undefined) { if (module !== undefined) {
const file = context.filesByPath.get(sourcePath); const file = context.filesByPath.get(sourcePath);
return file === undefined return file === undefined
@@ -426,11 +407,10 @@ const resolveBundlerChunkReference = (
if (file === undefined) return null; if (file === undefined) return null;
const registrations = context.analysis.files const registrations = context.analysis.files
.find(({ file: analyzedFile }) => analyzedFile.path === sourcePath) .find(({ file: analyzedFile }) => analyzedFile.path === sourcePath)
?.javascript?.bundler_registrations.filter(({ runtime, modules }) => ?.javascript?.bundler_registrations.filter(({ modules }) =>
reference.module_runtime === undefined reference.module_key === null
? reference.module_key === null ? true
: runtime === reference.module_runtime && : modules.some(
modules.some(
({ module_key: moduleKey }) => moduleKey === reference.module_key, ({ module_key: moduleKey }) => moduleKey === reference.module_key,
), ),
); );
@@ -2,9 +2,6 @@ import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySn
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js"; import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
import { completeApplicationCoverage } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js"; import { completeApplicationCoverage } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
import type { JavaScriptModuleArtifactAnalysis } from "./JavaScriptArtifactAnalysisTypes.js"; import type { JavaScriptModuleArtifactAnalysis } from "./JavaScriptArtifactAnalysisTypes.js";
import { jsonValueSchema } from "../../domain/jsonValue.js";
import { freezeJsonSnapshot } from "../../domain/immutableJson.js";
import type { JavaScriptSourceRange } from "../../domain/javascript/javascriptStaticAnalysisTypes.js";
import type { import type {
JavaScriptArtifactContainer, JavaScriptArtifactContainer,
JavaScriptArtifactFile, JavaScriptArtifactFile,
@@ -15,14 +12,12 @@ import {
addUnavailableStaticParseScope, addUnavailableStaticParseScope,
artifactFileNodeKind, artifactFileNodeKind,
artifactLocalIdentity, artifactLocalIdentity,
javascriptAnalysisCoverage,
createElectronRoleNode, createElectronRoleNode,
linkElectronRoleToAsset, linkElectronRoleToAsset,
type JavaScriptArtifactGraphContext, type JavaScriptArtifactGraphContext,
} from "./JavaScriptArtifactGraphContext.js"; } from "./JavaScriptArtifactGraphContext.js";
import { import {
artifactObservationEvidence, artifactObservationEvidence,
astObservationEvidence,
staticInferenceEvidence, staticInferenceEvidence,
} from "./JavaScriptArtifactGraphEvidence.js"; } from "./JavaScriptArtifactGraphEvidence.js";
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js"; import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
@@ -120,52 +115,9 @@ export const addJavaScriptArtifactContainers = (
export const addJavaScriptArtifactFiles = ( export const addJavaScriptArtifactFiles = (
context: JavaScriptArtifactGraphContext, context: JavaScriptArtifactGraphContext,
): void => { ): void => {
const packagesByPath = indexFirstByPath(context.analysis.packages);
const jsonModulesByPath = indexFirstByPath(context.analysis.json_modules);
const sourceMapsByPath = indexFirstByPath(context.analysis.source_maps);
for (const analyzed of context.analysis.files) { for (const analyzed of context.analysis.files) {
const { file } = analyzed; const { file } = analyzed;
const jsonValue = jsonModulesByPath.get(file.path); const target = createFileTarget(context, file, analyzed.javascript);
const target = createFileTarget(
context,
file,
analyzed.javascript,
jsonValue,
);
if (
analyzed.application_projection_failure !== undefined &&
analyzed.javascript !== null
)
context.accumulator.addNode({
kind: target.kind,
identity: target.identity,
observations: [
{
label: file.path,
properties: {
static_analysis: freezeJsonSnapshot(
jsonValueSchema.parse(analyzed.javascript),
),
semantic_module_analysis:
analyzed.semantic === null
? null
: freezeJsonSnapshot(
jsonValueSchema.parse(analyzed.semantic.ir),
),
application_projection_failure:
analyzed.application_projection_failure.error,
},
evidence: astObservationEvidence({
sha256: file.sha256,
path: file.path,
operation: "retain-unexpanded-static-findings",
range: sourceFileRange(file),
coverage: javascriptAnalysisCoverage(analyzed.javascript),
limitations: [analyzed.application_projection_failure.reason],
}),
},
],
});
context.fileNodes.set(file.path, target); context.fileNodes.set(file.path, target);
if (file.kind === "javascript") context.assetNodes.set(file.path, target); if (file.kind === "javascript") context.assetNodes.set(file.path, target);
const entry = createAsarEntry(context, file); const entry = createAsarEntry(context, file);
@@ -198,18 +150,7 @@ export const addJavaScriptArtifactFiles = (
}), }),
}); });
} }
if (analyzed.analysis_failure !== undefined) if (
addUnavailableStaticParseScope(context, {
file,
asset: target,
operation: "analyze-javascript-source",
limitation: analyzed.analysis_failure.reason,
failure: analyzed.analysis_failure.error,
limits: javaScriptAnalysisCoverageLimits(
analyzed.analysis_failure.limits,
),
});
else if (
file.kind === "javascript" && file.kind === "javascript" &&
(analyzed.javascript === null || (analyzed.javascript === null ||
analyzed.javascript.parse_status === "failed") analyzed.javascript.parse_status === "failed")
@@ -222,18 +163,9 @@ export const addJavaScriptArtifactFiles = (
? "JavaScript syntax could not be parsed." ? "JavaScript syntax could not be parsed."
: `JavaScript text was unavailable: ${file.text.reason}.`, : `JavaScript text was unavailable: ${file.text.reason}.`,
}); });
if (analyzed.application_projection_failure !== undefined) const packageValue = context.analysis.packages.find(
addUnavailableStaticParseScope(context, { ({ path }) => path === file.path,
file, );
asset: target,
operation: "project-javascript-application",
limitation: analyzed.application_projection_failure.reason,
failure: analyzed.application_projection_failure.error,
limits: javaScriptAnalysisCoverageLimits(
analyzed.application_projection_failure.limits,
),
});
const packageValue = packagesByPath.get(file.path);
if (packageValue !== undefined && packageValue.status !== "included") if (packageValue !== undefined && packageValue.status !== "included")
addUnavailableStaticParseScope(context, { addUnavailableStaticParseScope(context, {
file, file,
@@ -241,6 +173,9 @@ export const addJavaScriptArtifactFiles = (
operation: "parse-package-json", operation: "parse-package-json",
limitation: packageValue.limitation, limitation: packageValue.limitation,
}); });
const jsonValue = context.analysis.json_modules.find(
({ path }) => path === file.path,
);
if (jsonValue !== undefined && jsonValue.status !== "included") if (jsonValue !== undefined && jsonValue.status !== "included")
addUnavailableStaticParseScope(context, { addUnavailableStaticParseScope(context, {
file, file,
@@ -248,7 +183,9 @@ export const addJavaScriptArtifactFiles = (
operation: "parse-json-module", operation: "parse-json-module",
limitation: jsonValue.limitation, limitation: jsonValue.limitation,
}); });
const sourceMap = sourceMapsByPath.get(file.path); const sourceMap = context.analysis.source_maps.find(
({ path }) => path === file.path,
);
if (sourceMap !== undefined && sourceMap.status === "invalid") if (sourceMap !== undefined && sourceMap.status === "invalid")
addUnavailableStaticParseScope(context, { addUnavailableStaticParseScope(context, {
file, file,
@@ -259,23 +196,6 @@ export const addJavaScriptArtifactFiles = (
} }
}; };
const sourceFileRange = (
file: JavaScriptArtifactFile,
): JavaScriptSourceRange => {
if (!file.text.included)
throw new TypeError("Retained static facts require their source text");
let line = 1;
let lastLineStart = 0;
for (const match of file.text.value.matchAll(/\r\n|[\n\r\u2028\u2029]/gu)) {
line += 1;
lastLineStart = match.index + match[0].length;
}
return {
start: { line: 1, column: 0 },
end: { line, column: file.text.value.length - lastLineStart },
};
};
/** Project package metadata and its declared Electron roles. */ /** Project package metadata and its declared Electron roles. */
export const addJavaScriptPackageNodes = ( export const addJavaScriptPackageNodes = (
context: JavaScriptArtifactGraphContext, context: JavaScriptArtifactGraphContext,
@@ -350,9 +270,11 @@ const createFileTarget = (
context: JavaScriptArtifactGraphContext, context: JavaScriptArtifactGraphContext,
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
javascript: JavaScriptModuleArtifactAnalysis["files"][number]["javascript"], javascript: JavaScriptModuleArtifactAnalysis["files"][number]["javascript"],
json: JavaScriptModuleArtifactAnalysis["json_modules"][number] | undefined,
): ApplicationNode => { ): ApplicationNode => {
const kind = artifactFileNodeKind(file.kind); const kind = artifactFileNodeKind(file.kind);
const json = context.analysis.json_modules.find(
({ path }) => path === file.path,
);
return context.accumulator.addNode({ return context.accumulator.addNode({
kind, kind,
identity: { identity: {
@@ -387,16 +309,6 @@ const createFileTarget = (
}); });
}; };
const indexFirstByPath = <Value extends { readonly path: string }>(
values: readonly Value[],
): ReadonlyMap<string, Value> => {
const index = new Map<string, Value>();
for (const value of values) {
if (!index.has(value.path)) index.set(value.path, value);
}
return index;
};
const createAsarEntry = ( const createAsarEntry = (
context: JavaScriptArtifactGraphContext, context: JavaScriptArtifactGraphContext,
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
@@ -485,4 +397,3 @@ const containerProperties = (container: JavaScriptArtifactContainer) => ({
bytes: container.bytes, bytes: container.bytes,
inventory_artifact_id: container.inventory_artifact_id, inventory_artifact_id: container.inventory_artifact_id,
}); });
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
@@ -234,8 +234,7 @@ const resolvePackageSpecifier = (
if (rawExports === undefined || rawExports === null) { if (rawExports === undefined || rawExports === null) {
if (subpath === ".") if (subpath === ".")
return resolveDirectory(input, directory, metadata, new Set()); return resolveDirectory(input, directory, metadata, new Set());
// Preserve the trailing slash for a package directory request. const candidate = confineCandidate(posix.join(directory, subpath.slice(2)));
const candidate = confineCandidate(`${directory}/${subpath.slice(2)}`);
return typeof candidate === "string" return typeof candidate === "string"
? resolveCandidate(input, candidate, new Set()) ? resolveCandidate(input, candidate, new Set())
: candidate; : candidate;
@@ -11,22 +11,15 @@ import { createJavaScriptArtifactReader as createReader } from "../../artifacts/
import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js"; import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js";
import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js"; import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js";
import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { parseOwnedJavaScriptApplicationAnalysisSteps } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js"; import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import type { ArtifactInventoryPartialObservation } from "../../domain/artifactPartialObservation.js"; import { analyzeAndProjectJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js";
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
import { analyzeAndAdoptJavaScriptArtifactFiles } from "./JavaScriptArtifactAnalysis.js";
import { createJavaScriptSourceAnalysis } from "../../artifacts/javascript/JavaScriptSourceAnalysis.js";
import { JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
import { createOwnedJavaScriptApplicationEvidenceCooperatively } from "./JavaScriptApplicationEvidence.js";
import { readJavaScriptArtifactFiles } from "../../artifacts/javascript/JavaScriptArtifactFiles.js"; import { readJavaScriptArtifactFiles } from "../../artifacts/javascript/JavaScriptArtifactFiles.js";
import { buildImmutableJavaScriptArtifactGraphSteps } from "./JavaScriptArtifactGraphBuilder.js"; import { buildImmutableJavaScriptArtifactGraphSteps } from "./JavaScriptArtifactGraphBuilder.js";
import { import {
javascriptArtifactReconstructionInputSchema, javascriptArtifactReconstructionInputSchema,
type JavaScriptArtifactReconstructionInput, type JavaScriptArtifactReconstructionInput,
} from "./JavaScriptArtifactReconstructionInput.js"; } from "./JavaScriptArtifactReconstructionInput.js";
import { scanCanonicalArtifactInventoryInScope } from "../../artifacts/inventory/scanCanonical.js"; import { scanCanonicalArtifactInventory } from "../../artifacts/inventory/scanCanonical.js";
import { summarizeElectronBoundaries } from "./ElectronBoundaryAnalysis.js"; import { summarizeElectronBoundaries } from "./ElectronBoundaryAnalysis.js";
import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js"; import { createJavaScriptSemanticGraphProjection } from "./JavaScriptSemanticGraphBuilder.js";
import type { ProgressReporter } from "../ProgressReporter.js"; import type { ProgressReporter } from "../ProgressReporter.js";
@@ -63,241 +56,118 @@ export interface JavaScriptArtifactReconstructionResult {
/** Reconstruct one local ASAR or extracted directory without executing code. */ /** Reconstruct one local ASAR or extracted directory without executing code. */
export const reconstructJavaScriptArtifact = async ( export const reconstructJavaScriptArtifact = async (
rawInput: unknown, rawInput: unknown,
scope: ArtifactResourceScope,
signal?: AbortSignal, signal?: AbortSignal,
progress?: ProgressReporter, progress?: ProgressReporter,
): Promise<JavaScriptArtifactReconstructionResult> => ): Promise<JavaScriptArtifactReconstructionResult> => {
scope.run(async () => { const reportPhase = async (phase: string, message: string): Promise<void> => {
let effectiveSignal = signal; await progress?.report({ phase, completed: 0, total: 1, message });
let effectiveProgress = progress; await checkpointJavaScriptAnalysis(signal);
const reportPhase = async ( };
phase: string, const input = javascriptArtifactReconstructionInputSchema.parse(rawInput);
message: string, abortIfNeeded(signal);
): Promise<void> => { const path = await resolveSelectedInput(input.input_path);
await effectiveProgress?.report({ const format = await resolveFormat(path, input);
phase, const snapshot = await scanCanonicalArtifactInventory(path, {
completed: 0, signal,
total: 1, integrity: { mode: input.integrity_policy },
message,
});
await checkpointJavaScriptAnalysis(effectiveSignal);
};
const input = javascriptArtifactReconstructionInputSchema.parse(rawInput);
abortIfNeeded(signal);
const path = await resolveSelectedInput(input.input_path);
const format = await resolveFormat(path, input);
const snapshot = await scanCanonicalArtifactInventoryInScope(path, {
resourceScope: scope,
signal,
integrity: { mode: input.integrity_policy },
});
if (snapshot.manifest.root_format !== format)
throw new ArtifactReaderFailure(
"format",
`Artifact inventory classified ${path} as ${snapshot.manifest.root_format}, not ${format}`,
);
const reader = createReader(path, format);
const readerOwner = {
kind: "reader" as const,
reader,
resource: `JavaScript artifact reader for ${path}`,
};
let readerOpen = true;
let reconstructionOutcome:
| {
readonly kind: "completed";
readonly result: JavaScriptArtifactReconstructionResult;
}
| { readonly kind: "failed"; readonly cause: unknown };
try {
await reportPhase(
"read_javascript_artifacts",
"Reading inventoried JavaScript application sources",
);
const files = await readJavaScriptArtifactFiles(
reader,
snapshot,
scope,
signal,
);
// Release acquisition before analysis consumes these owned file facts.
const earlyCleanup = await scope.release(readerOwner);
readerOpen = false;
if (earlyCleanup.kind === "failed")
throw ArtifactReaderFailure.withCleanup(
earlyCleanup.cause,
ArtifactReaderFailure.cleanupObservation(
earlyCleanup.cause,
readerOwner.resource,
),
{ kind: "artifact-inventory", inventory: snapshot },
);
await reportPhase(
"parse_javascript_sources",
`Parsing and projecting ${String(files.files.length)} application source files`,
);
const semanticProjection = createJavaScriptSemanticGraphProjection();
const worker = createJavaScriptSourceAnalysis(
{
heapMb:
input.max_heap_mb ?? JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS.heapMb,
timeoutMs:
input.analysis_timeout_ms ??
JAVASCRIPT_ANALYSIS_RESOURCE_DEFAULTS.timeoutMs,
},
files.files.filter(
(file) => file.kind === "javascript" && file.text.included,
).length,
);
let outcome;
let cleanupFailure: AnalysisError | undefined;
try {
outcome = await analyzeAndAdoptJavaScriptArtifactFiles(
files,
worker,
semanticProjection,
async (file, completed, total) => {
abortIfNeeded(signal);
await progress?.report({
phase: "parse_javascript_source",
completed: 0,
total: 1,
message: `Parsing and projecting ${file.path} (${String(completed + 1)}/${String(total)})`,
});
abortIfNeeded(signal);
},
signal,
);
} finally {
try {
const cleanup = await scope.release({
kind: "javascript-source-analysis",
analysis: worker,
resource: `JavaScript source analyzer for ${path}`,
});
if (cleanup.kind === "failed") throw cleanup.cause;
} catch (cause: unknown) {
if (!(cause instanceof AnalysisError)) throw cause;
cleanupFailure = cause;
}
}
const analysis = outcome.analysis;
const interruption = cleanupFailure ?? outcome.interruption;
if (interruption !== null && interruption !== undefined) {
effectiveSignal = undefined;
effectiveProgress = undefined;
}
await checkpointJavaScriptAnalysis(effectiveSignal);
abortIfNeeded(effectiveSignal);
await reportPhase(
"build_javascript_application_graph",
"Constructing application and Electron boundary relationships",
);
const applicationGraphSteps = buildImmutableJavaScriptArtifactGraphSteps(
snapshot,
files,
analysis,
);
await reportPhase(
"seal_javascript_application_graph",
"Sealing the validated application graph",
);
const graph = await completeJavaScriptAnalysisSteps(
applicationGraphSteps,
effectiveSignal,
);
await reportPhase(
"build_javascript_semantic_graph",
"Binding and validating static semantic relationships",
);
const semanticGraphSteps = semanticProjection.finishImmutableSteps(
snapshot.manifest.root_sha256,
graph,
);
await reportPhase(
"seal_javascript_semantic_graph",
"Validating and sealing the semantic graph",
);
const semanticGraph = await completeJavaScriptAnalysisSteps(
semanticGraphSteps,
effectiveSignal,
);
await checkpointJavaScriptAnalysis(effectiveSignal);
const completed = {
input_path: path,
format,
root_artifact_sha256: snapshot.manifest.root_sha256,
inventory_manifest_id: snapshot.manifest.manifest_id,
inventory_graph_sha256: snapshot.manifest.graph_sha256,
integrity_contradictions: snapshot.integrity_contradictions,
graph,
semantic_graph: semanticGraph,
electron_summary: summarizeElectronBoundaries(analysis),
statistics: {
relevant_files: files.files.length,
nested_asar_containers: files.containers.length,
text_bytes_read: files.text_bytes_read,
invalid_utf8_files: files.invalid_utf8_files,
parsed_javascript_files: analysis.files.filter(
({ javascript }) => javascript !== null,
).length,
visited_ast_nodes: analysis.visited_ast_nodes,
findings: analysis.findings,
modules: analysis.modules,
parse_failures: analysis.parse_failures,
},
limitations: analysis.limitations,
};
if (interruption !== null && interruption !== undefined) {
const { electron_summary: summary, ...application } = completed;
const partial = await completeJavaScriptAnalysisSteps(
parseOwnedJavaScriptApplicationAnalysisSteps({
...application,
summary,
limitations: completed.graph.limitations,
}),
);
const { max_heap_mb, analysis_timeout_ms, ...coreInput } = input;
const parameters = {
...coreInput,
...(max_heap_mb === undefined ? {} : { max_heap_mb }),
...(analysis_timeout_ms === undefined ? {} : { analysis_timeout_ms }),
};
const evidence =
await createOwnedJavaScriptApplicationEvidenceCooperatively(
parameters,
partial,
);
throw interruption.retainPartialObservation(evidence);
}
reconstructionOutcome = { kind: "completed", result: completed };
} catch (cause: unknown) {
reconstructionOutcome = { kind: "failed", cause };
}
if (readerOpen) {
const cleanupAttempt = await scope.release(readerOwner);
if (cleanupAttempt.kind === "failed") {
const cleanup = ArtifactReaderFailure.cleanupObservation(
cleanupAttempt.cause,
readerOwner.resource,
);
const partialObservation: ArtifactInventoryPartialObservation = {
kind: "artifact-inventory",
inventory: snapshot,
};
throw ArtifactReaderFailure.withCleanup(
reconstructionOutcome.kind === "failed"
? reconstructionOutcome.cause
: cleanupAttempt.cause,
cleanup,
partialObservation,
);
}
}
if (reconstructionOutcome.kind === "failed")
throw reconstructionOutcome.cause;
return reconstructionOutcome.result;
}); });
if (snapshot.manifest.root_format !== format)
throw new ArtifactReaderFailure(
"format",
`Artifact inventory classified ${path} as ${snapshot.manifest.root_format}, not ${format}`,
);
const reader = createReader(path, format);
try {
await reportPhase(
"read_javascript_artifacts",
"Reading inventoried JavaScript application sources",
);
const files = await readJavaScriptArtifactFiles(reader, snapshot, signal);
await reportPhase(
"parse_javascript_sources",
`Parsing and projecting ${String(files.files.length)} application source files`,
);
const semanticProjection = createJavaScriptSemanticGraphProjection();
const analysis = await analyzeAndProjectJavaScriptArtifactFiles(
files,
semanticProjection.projectFileSteps,
async (file, completed, total) => {
abortIfNeeded(signal);
await progress?.report({
phase: "parse_javascript_source",
completed: 0,
total: 1,
message: `Parsing and projecting ${file.path} (${String(completed + 1)}/${String(total)})`,
});
abortIfNeeded(signal);
},
signal,
);
await checkpointJavaScriptAnalysis(signal);
abortIfNeeded(signal);
await reportPhase(
"build_javascript_application_graph",
"Constructing application and Electron boundary relationships",
);
const applicationGraphSteps = buildImmutableJavaScriptArtifactGraphSteps(
snapshot,
files,
analysis,
);
await reportPhase(
"seal_javascript_application_graph",
"Sealing the validated application graph",
);
const graph = await completeJavaScriptAnalysisSteps(
applicationGraphSteps,
signal,
);
await reportPhase(
"build_javascript_semantic_graph",
"Binding and validating static semantic relationships",
);
const semanticGraphSteps = semanticProjection.finishImmutableSteps(
snapshot.manifest.root_sha256,
graph,
);
await reportPhase(
"seal_javascript_semantic_graph",
"Validating and sealing the semantic graph",
);
const semanticGraph = await completeJavaScriptAnalysisSteps(
semanticGraphSteps,
signal,
);
await checkpointJavaScriptAnalysis(signal);
return {
input_path: path,
format,
root_artifact_sha256: snapshot.manifest.root_sha256,
inventory_manifest_id: snapshot.manifest.manifest_id,
inventory_graph_sha256: snapshot.manifest.graph_sha256,
integrity_contradictions: snapshot.integrity_contradictions,
graph,
semantic_graph: semanticGraph,
electron_summary: summarizeElectronBoundaries(analysis),
statistics: {
relevant_files: files.files.length,
nested_asar_containers: files.containers.length,
text_bytes_read: files.text_bytes_read,
invalid_utf8_files: files.invalid_utf8_files,
parsed_javascript_files: analysis.files.filter(
({ javascript }) => javascript !== null,
).length,
visited_ast_nodes: analysis.visited_ast_nodes,
findings: analysis.findings,
modules: analysis.modules,
parse_failures: analysis.parse_failures,
},
limitations: analysis.limitations,
};
} finally {
await reader.close();
}
};
const OPERATION = "analyze_javascript_application"; const OPERATION = "analyze_javascript_application";
@@ -1,13 +1,11 @@
import { z } from "zod"; import { z } from "zod";
import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js"; import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js";
import { javaScriptAnalysisResourceControlsSchema } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
/** Local ASAR/directory reconstruction request. */ /** Local ASAR/directory reconstruction request. */
export const javascriptArtifactReconstructionInputSchema = z.strictObject({ export const javascriptArtifactReconstructionInputSchema = z.strictObject({
input_path: z.string().min(1), input_path: z.string().min(1),
format: z.enum(["auto", "asar", "directory"]).default("auto"), format: z.enum(["auto", "asar", "directory"]).default("auto"),
integrity_policy: artifactIntegrityPolicySchema, integrity_policy: artifactIntegrityPolicySchema,
...javaScriptAnalysisResourceControlsSchema.shape,
}); });
/** Parsed local reconstruction request. */ /** Parsed local reconstruction request. */
@@ -8,7 +8,7 @@ import { describe, expect, it } from "vitest";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { staticPath } from "../../domain/javascript/javascriptStaticAnalysisHelpers.js"; import { staticPath } from "../../domain/javascript/javascriptStaticAnalysisHelpers.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
const urlCases = [ const urlCases = [
["./preload%20name.cjs", "preload name.cjs", "main.mjs"], ["./preload%20name.cjs", "preload name.cjs", "main.mjs"],
@@ -7,7 +7,7 @@ import { promisify } from "node:util";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js"; import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js"; import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
const filesFor = (paths: readonly string[]) => const filesFor = (paths: readonly string[]) =>
@@ -5,7 +5,7 @@ import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js"; import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js"; import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js"; import { analyzeJavaScriptApplication } from "./JavaScriptApplicationService.js";
it("classifies overloaded open calls from lexical receiver facts", async () => { it("classifies overloaded open calls from lexical receiver facts", async () => {
const inputPath = await createTestTempDirectory( const inputPath = await createTestTempDirectory(
@@ -355,7 +355,7 @@ const semanticNodeFor = (roleKey: string) => {
unpacked: false, unpacked: false,
text: { included: true, value: "" }, text: { included: true, value: "" },
}; };
const state = createSemanticGraphProjectionState(); const state = createSemanticGraphProjectionState({ nodes: [] });
return constructSemanticGraphNode( return constructSemanticGraphNode(
file, file,
{ {
@@ -565,7 +565,7 @@ it("does not mark a file truncated when it exactly fills its budget", () => {
it("only records node loss when the budget rejects a new identity", () => { it("only records node loss when the budget rejects a new identity", () => {
// The truncation signal must record blocked node creation, not a zero // The truncation signal must record blocked node creation, not a zero
// remaining budget: a file using exactly its share drops nothing. // remaining budget: a file using exactly its share drops nothing.
const state = createSemanticGraphProjectionState(); const state = createSemanticGraphProjectionState({ nodes: [] });
state.fileNodeBudget = 2; state.fileNodeBudget = 2;
state.fileNodesDropped = false; state.fileNodesDropped = false;
const first = addSemanticGraphNode(state, semanticNodeFor("first")); const first = addSemanticGraphNode(state, semanticNodeFor("first"));
@@ -1,6 +1,5 @@
import { import {
createJavaScriptSemanticGraph, createJavaScriptSemanticGraph,
createJavaScriptSemanticGraphUnknown,
sealTransferredJavaScriptSemanticGraphSteps, sealTransferredJavaScriptSemanticGraphSteps,
type JavaScriptSemanticGraph, type JavaScriptSemanticGraph,
} from "../../domain/javascript/javascriptSemanticGraph.js"; } from "../../domain/javascript/javascriptSemanticGraph.js";
@@ -11,7 +10,6 @@ import type {
import { import {
JAVASCRIPT_SEMANTIC_RELATION_FAMILIES, JAVASCRIPT_SEMANTIC_RELATION_FAMILIES,
JAVASCRIPT_SEMANTIC_RELATION_FAMILY, JAVASCRIPT_SEMANTIC_RELATION_FAMILY,
JAVASCRIPT_SEMANTIC_RELATIONS,
} from "../../domain/javascript/javascriptSemanticGraphSchemas.js"; } from "../../domain/javascript/javascriptSemanticGraphSchemas.js";
import type { import type {
JavaScriptSemanticCallArgument, JavaScriptSemanticCallArgument,
@@ -35,13 +33,6 @@ import { projectSemanticResources } from "./JavaScriptSemanticGraphResourceProje
import { projectSemanticObjects } from "./JavaScriptSemanticGraphObjectProjection.js"; import { projectSemanticObjects } from "./JavaScriptSemanticGraphObjectProjection.js";
import { projectSemanticValues } from "./JavaScriptSemanticGraphValueProjection.js"; import { projectSemanticValues } from "./JavaScriptSemanticGraphValueProjection.js";
import { inferredSemanticEvidenceAt } from "./JavaScriptSemanticGraphEvidence.js"; import { inferredSemanticEvidenceAt } from "./JavaScriptSemanticGraphEvidence.js";
import { observedSemanticEvidence } from "./JavaScriptSemanticGraphEvidence.js";
import {
partialApplicationCoverage,
type ApplicationCoverage,
} from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
import type { JsonValue } from "../../domain/jsonValue.js";
import { javaScriptAnalysisCoverageLimits } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
import { projectSemanticFunctionFingerprints } from "./JavaScriptSemanticGraphFingerprintProjection.js"; import { projectSemanticFunctionFingerprints } from "./JavaScriptSemanticGraphFingerprintProjection.js";
import { import {
projectSemanticClosureCaptures, projectSemanticClosureCaptures,
@@ -64,7 +55,6 @@ import {
bindSemanticGraphApplicationNodes, bindSemanticGraphApplicationNodes,
type SemanticGraphProjectionState as BuilderState, type SemanticGraphProjectionState as BuilderState,
} from "./JavaScriptSemanticGraphConstruction.js"; } from "./JavaScriptSemanticGraphConstruction.js";
import type { JavaScriptSemanticFileProjection } from "../../domain/javascript/javascriptSemanticFileProjection.js";
interface BuilderInput { interface BuilderInput {
readonly rootArtifactSha256: string; readonly rootArtifactSha256: string;
@@ -136,19 +126,6 @@ export const buildJavaScriptSemanticGraph = ({
/** File-local semantic projection that does not retain consumed source IR. */ /** File-local semantic projection that does not retain consumed source IR. */
export interface JavaScriptSemanticGraphProjection { export interface JavaScriptSemanticGraphProjection {
/** Node capacity offered to the next independently analyzed source file. */
readonly remainingFileNodeBudget: () => number;
/** Adopt decoded file-local facts; final validation and proof remain in this realm. */
readonly adoptFileProjection: (
projection: JavaScriptSemanticFileProjection,
) => void;
/** Preserve source-bound unknowns for an execution or result-retention failure. */
readonly recordUnavailableFile: (
file: JavaScriptArtifactFile,
reason: string,
limits: Readonly<Record<string, JsonValue>> | null,
unknownReason: "resource-limit" | "incomplete-module",
) => void;
readonly projectFile: ( readonly projectFile: (
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
ir: JavaScriptSemanticIr, ir: JavaScriptSemanticIr,
@@ -171,15 +148,9 @@ export interface JavaScriptSemanticGraphProjection {
/** Accumulate semantic nodes while each source IR is still file-local. */ /** Accumulate semantic nodes while each source IR is still file-local. */
export const createJavaScriptSemanticGraphProjection = export const createJavaScriptSemanticGraphProjection =
(): JavaScriptSemanticGraphProjection => { (): JavaScriptSemanticGraphProjection => {
const state = emptyState(); const state = emptyState({ nodes: [] });
const fingerprints: JavaScriptSemanticFingerprint[] = []; const fingerprints: JavaScriptSemanticFingerprint[] = [];
let truncatedFiles = 0; let truncatedFiles = 0;
let unavailableFiles = 0;
let resourceTruncatedFiles = 0;
const executionLimits = new Map<
string,
ApplicationCoverage["limits"][number]
>();
const projectSourceSteps = function* ( const projectSourceSteps = function* (
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
ir: JavaScriptSemanticIr, ir: JavaScriptSemanticIr,
@@ -236,15 +207,12 @@ export const createJavaScriptSemanticGraphProjection =
fingerprints: [...fingerprints], fingerprints: [...fingerprints],
unknowns, unknowns,
coverage: { coverage: {
status: status: truncatedFiles > 0 ? "partial" : "unknown",
truncatedFiles > 0 || unavailableFiles > 0 ? "partial" : "unknown", truncated: truncatedFiles > 0,
truncated: truncatedFiles > 0 || resourceTruncatedFiles > 0, omitted_nodes: truncatedFiles > 0 ? null : 0,
omitted_nodes: truncatedFiles > 0 || unavailableFiles > 0 ? null : 0, omitted_relations: truncatedFiles > 0 ? null : 0,
omitted_relations: limits:
truncatedFiles > 0 || unavailableFiles > 0 ? null : 0, truncatedFiles > 0
limits: [
...executionLimits.values(),
...(truncatedFiles > 0
? [ ? [
{ {
name: "semantic_graph_node_ceiling", name: "semantic_graph_node_ceiling",
@@ -252,14 +220,12 @@ export const createJavaScriptSemanticGraphProjection =
unit: "items" as const, unit: "items" as const,
}, },
] ]
: []), : [],
],
families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({ families: JAVASCRIPT_SEMANTIC_RELATION_FAMILIES.map((family) => ({
family, family,
status: semanticFamilyStatus(family), status: semanticFamilyStatus(family),
retained_relations: retainedByFamily.get(family) ?? 0, retained_relations: retainedByFamily.get(family) ?? 0,
omitted_relations: omitted_relations: truncatedFiles > 0 ? null : 0,
truncatedFiles > 0 || unavailableFiles > 0 ? null : 0,
unknown_ids: unknownIdsByFamily.get(family) ?? [], unknown_ids: unknownIdsByFamily.get(family) ?? [],
})), })),
}, },
@@ -286,87 +252,6 @@ export const createJavaScriptSemanticGraphProjection =
return graph; return graph;
}; };
return { return {
remainingFileNodeBudget: () =>
Math.min(
SEMANTIC_GRAPH_FILE_NODE_CEILING,
Math.max(0, SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size),
),
adoptFileProjection: (projection) => {
const newNodes = projection.nodes.filter(
({ node_id }) => !state.nodes.has(node_id),
);
if (
newNodes.length >
Math.min(
SEMANTIC_GRAPH_FILE_NODE_CEILING,
SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size,
)
)
throw new RangeError(
"Transferred semantic nodes exceed the offered tree capacity",
);
for (const context of projection.evidenceContexts)
state.evidenceContexts.adopt(context);
for (const node of newNodes) state.nodes.set(node.node_id, node);
for (const relation of projection.relations)
state.relations.set(relation.relation_id, relation);
for (const unknown of projection.unknowns)
state.unknowns.set(unknown.unknown_id, unknown);
for (const root of projection.roots) state.roots.add(root);
fingerprints.push(...projection.fingerprints);
if (projection.truncated) truncatedFiles += 1;
},
recordUnavailableFile: (file, reason, limits, unknownReason) => {
const retainedLimits = javaScriptAnalysisCoverageLimits(limits);
for (const limit of retainedLimits) {
executionLimits.set(
JSON.stringify([limit.name, limit.value, limit.unit]),
limit,
);
}
state.fileNodeBudget = Math.min(
SEMANTIC_GRAPH_FILE_NODE_CEILING,
Math.max(0, SEMANTIC_GRAPH_NODE_CEILING - state.nodes.size),
);
const module = retainNode(state, file, {
kind: "module",
roleKey: "module",
location: null,
label: file.path,
functionNodeId: null,
});
state.fileNodeBudget = null;
if (module !== null) state.roots.add(module.node_id);
const evidence = {
...observedSemanticEvidence(file, null),
authority: "unknown" as const,
state: "unknown" as const,
confidence: "unknown" as const,
coverage: partialApplicationCoverage(retainedLimits, null),
limitations: [reason],
};
for (const family of JAVASCRIPT_SEMANTIC_RELATION_FAMILIES) {
const unknown = createJavaScriptSemanticGraphUnknown(
{
node_id: module?.node_id ?? null,
family,
relation_kinds: JAVASCRIPT_SEMANTIC_RELATIONS.filter(
(relation) =>
JAVASCRIPT_SEMANTIC_RELATION_FAMILY[relation] === family,
),
reason: unknownReason,
detail: reason,
candidate_node_ids: [],
evidence,
},
state.evidenceContexts,
);
state.unknowns.set(unknown.unknown_id, unknown);
}
unavailableFiles += 1;
if (unknownReason === "resource-limit" && retainedLimits.length > 0)
resourceTruncatedFiles += 1;
},
projectFile: (file, ir) => { projectFile: (file, ir) => {
const steps = projectSourceSteps(file, ir); const steps = projectSourceSteps(file, ir);
while (steps.next().done !== true); while (steps.next().done !== true);
@@ -387,35 +272,6 @@ export const createJavaScriptSemanticGraphProjection =
}; };
}; };
/** Project one source with the parent's remaining node capacity, retaining no full IR. */
export function* projectJavaScriptSemanticFileSteps(
file: JavaScriptArtifactFile,
ir: JavaScriptSemanticIr,
nodeBudget: number,
): Generator<void, JavaScriptSemanticFileProjection> {
if (
!Number.isSafeInteger(nodeBudget) ||
nodeBudget < 0 ||
nodeBudget > SEMANTIC_GRAPH_FILE_NODE_CEILING
)
throw new RangeError(
"File semantic projection requires a valid offered node capacity",
);
const state = emptyState();
state.fileNodeBudget = nodeBudget;
state.fileNodesDropped = false;
const fingerprints = yield* projectFileSteps(file, ir, state);
return {
roots: [...state.roots],
evidenceContexts: state.evidenceContexts.contexts,
nodes: [...state.nodes.values()],
relations: [...state.relations.values()],
unknowns: [...state.unknowns.values()],
fingerprints,
truncated: Boolean(state.fileNodesDropped),
};
}
function* projectFileSteps( function* projectFileSteps(
file: JavaScriptArtifactFile, file: JavaScriptArtifactFile,
ir: JavaScriptSemanticIr, ir: JavaScriptSemanticIr,
@@ -28,6 +28,7 @@ export interface SemanticGraphProjectionState {
readonly unknowns: Map<string, JavaScriptSemanticGraphUnknown>; readonly unknowns: Map<string, JavaScriptSemanticGraphUnknown>;
readonly evidenceContexts: JavaScriptSemanticEvidenceContextRegistry; readonly evidenceContexts: JavaScriptSemanticEvidenceContextRegistry;
readonly roots: Set<string>; readonly roots: Set<string>;
readonly applicationNodeIdsByLocation: ReadonlyMap<string, readonly string[]>;
/** /**
* Remaining node budget for the file currently being projected. * Remaining node budget for the file currently being projected.
* *
@@ -68,29 +69,32 @@ export interface SemanticRelationConstructionInput {
readonly properties?: Readonly<Record<string, JsonValue>>; readonly properties?: Readonly<Record<string, JsonValue>>;
} }
/** Create empty projection state; application-node links are bound at finish. */ /** Create empty projection state with structural application-node mappings. */
export const createSemanticGraphProjectionState = export const createSemanticGraphProjectionState = (
(): SemanticGraphProjectionState => ({ applicationGraph: Pick<JavaScriptApplicationGraph, "nodes">,
): SemanticGraphProjectionState => {
const index = indexApplicationNodes(applicationGraph.nodes);
return {
nodes: new Map(), nodes: new Map(),
relations: new Map(), relations: new Map(),
unknowns: new Map(), unknowns: new Map(),
evidenceContexts: new JavaScriptSemanticEvidenceContextRegistry(), evidenceContexts: new JavaScriptSemanticEvidenceContextRegistry(),
roots: new Set(), roots: new Set(),
}); applicationNodeIdsByLocation: index.identifiers,
};
};
/** Bind retained semantic identities after all application observations exist. */ /** Bind retained semantic identities after all application observations exist. */
export const bindSemanticGraphApplicationNodes = ( export const bindSemanticGraphApplicationNodes = (
state: SemanticGraphProjectionState, state: SemanticGraphProjectionState,
applicationGraph: Pick<JavaScriptApplicationGraph, "nodes">, applicationGraph: Pick<JavaScriptApplicationGraph, "nodes">,
): void => { ): void => {
const applicationNodeIdsByLocation = indexApplicationNodes( const index = indexApplicationNodes(applicationGraph.nodes).identifiers;
applicationGraph.nodes,
);
for (const [identifier, node] of state.nodes) { for (const [identifier, node] of state.nodes) {
state.nodes.set(identifier, { state.nodes.set(identifier, {
...node, ...node,
application_node_ids: [ application_node_ids: [
...(applicationNodeIdsByLocation.get( ...(index.get(
applicationLocationKey( applicationLocationKey(
node.identity.artifact_sha256, node.identity.artifact_sha256,
node.identity.module_path, node.identity.module_path,
@@ -118,7 +122,7 @@ export const constructSemanticGraphNode = (
role_key: input.roleKey, role_key: input.roleKey,
}, },
function_node_id: input.functionNodeId, function_node_id: input.functionNodeId,
application_node_ids: [], application_node_ids: matchingApplicationNodeIds(file, input, state),
label: input.label === null ? null : javascriptDisplayText(input.label), label: input.label === null ? null : javascriptDisplayText(input.label),
properties: input.properties ?? {}, properties: input.properties ?? {},
evidence: observedSemanticEvidence(file, input.location), evidence: observedSemanticEvidence(file, input.location),
@@ -243,7 +247,9 @@ export const addSemanticFallbackRoot = (
const indexApplicationNodes = ( const indexApplicationNodes = (
nodes: JavaScriptApplicationGraph["nodes"], nodes: JavaScriptApplicationGraph["nodes"],
): ReadonlyMap<string, readonly string[]> => { ): {
readonly identifiers: ReadonlyMap<string, readonly string[]>;
} => {
const identifiers = new Map<string, Set<string>>(); const identifiers = new Map<string, Set<string>>();
for (const node of nodes) { for (const node of nodes) {
for (const observation of node.observations) { for (const observation of node.observations) {
@@ -274,9 +280,19 @@ const indexApplicationNodes = (
return [key, sorted]; return [key, sorted];
}), }),
); );
return indexed; return { identifiers: indexed };
}; };
const matchingApplicationNodeIds = (
file: JavaScriptArtifactFile,
input: SemanticNodeConstructionInput,
state: SemanticGraphProjectionState,
): string[] => [
...(state.applicationNodeIdsByLocation.get(
applicationLocationKey(file.sha256, file.path, input.location),
) ?? []),
];
const addApplicationNodeIdentifier = ( const addApplicationNodeIdentifier = (
identifiers: Map<string, Set<string>>, identifiers: Map<string, Set<string>>,
key: string, key: string,
@@ -325,7 +325,8 @@ const resolveUiAction = (
(node) => (node) =>
["control", "view", "view_controller", "scene"].includes(node.kind) && ["control", "view", "view_controller", "scene"].includes(node.kind) &&
(node.id === query || (node.id === query ||
node.attributes.interface_builder_object_id === query), node.attributes.interface_builder_object_id === query ||
node.id.endsWith(`:object:${query}`)),
); );
const actionIds = new Set( const actionIds = new Set(
graph.edges.flatMap((edge) => graph.edges.flatMap((edge) =>
@@ -7,7 +7,6 @@ import {
import { ghidraFunctionDossier } from "../../domain/ghidraValues.fixture.js"; import { ghidraFunctionDossier } from "../../domain/ghidraValues.fixture.js";
import { functionDossierSchema } from "../../domain/hopperValues.js"; import { functionDossierSchema } from "../../domain/hopperValues.js";
import { jsonValueSchema } from "../../domain/jsonValue.js"; import { jsonValueSchema } from "../../domain/jsonValue.js";
import { nativeValueTraceSchema } from "../../domain/native/nativeValueTrace.js";
import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js"; import { AnalysisCancelledError } from "../../domain/analysisErrorCore.js";
import { ok, err } from "../../domain/result.js"; import { ok, err } from "../../domain/result.js";
@@ -103,132 +102,6 @@ const analysis: AnalysisOperationPort = {
); );
}, },
}; };
describe("native value parameter-use node membership", () => {
it("retains only parameter uses whose parameter nodes fit the node budget", async () => {
const operationCount = 3_000;
const parameterUseCount = 12_000;
const base = functionDossierSchema.parse(ghidraFunctionDossier());
const procedureAddress = base.procedure.address;
const largeFlow = {
available: true as const,
provenance: "ghidra-high-pcode" as const,
operations: Array.from({ length: operationCount }, (_, index) => ({
...node(`operation-${String(index)}`, procedureAddress, "MULTIEQUAL"),
inputs: [scalar, scalar, scalar, scalar],
})),
def_use: [],
effects: [],
parameters: [{ ordinal: 0, name: "value", data_type: "/int" }],
parameter_uses: Array.from({ length: parameterUseCount }, (_, index) => ({
ordinal: 0,
use: `operation-${String(index % operationCount)}`,
input_index: Math.floor(index / operationCount),
})),
truncated: false,
omitted_operations_lower_bound: 0,
known_omitted_inputs: 0,
known_omitted_edges: 0,
limitations: [],
};
const largeDossier = {
...base,
native_value_flow: largeFlow,
};
const largeAnalysis: AnalysisOperationPort = {
execute: async () =>
ok(createAnalysisExecution(largeDossier, provider, { subject })),
};
const includedParameter = await traceNativeValues(largeAnalysis, {
procedure: procedureAddress,
max_nodes: 20_000,
max_edges: 40_000,
offset: operationCount,
limit: 1,
});
if (!includedParameter.ok) throw includedParameter.error;
const includedResult = nativeValueTraceSchema.parse(
includedParameter.value,
);
expect(includedResult).toMatchObject({
total_nodes: operationCount + 1,
total_edges: parameterUseCount,
nodes: [
expect.objectContaining({
id: `${procedureAddress}/parameter:0`,
kind: "parameter",
}),
],
truncated: true,
});
expect(includedResult.edges).toHaveLength(parameterUseCount);
const omittedParameter = await traceNativeValues(largeAnalysis, {
procedure: procedureAddress,
max_nodes: operationCount,
max_edges: 40_000,
offset: operationCount - 1,
limit: 1,
});
if (!omittedParameter.ok) throw omittedParameter.error;
const omittedResult = nativeValueTraceSchema.parse(omittedParameter.value);
expect(omittedResult).toMatchObject({
total_nodes: operationCount,
total_edges: 0,
nodes: [expect.objectContaining({ kind: "operation" })],
edges: [],
truncated: true,
});
});
it("preserves operation IDs that collide with generated parameter IDs", async () => {
const base = functionDossierSchema.parse(ghidraFunctionDossier());
const address = base.procedure.address;
const execution = createAnalysisExecution(
{
...base,
native_value_flow: {
available: true,
provenance: "ghidra-high-pcode",
operations: [
node("parameter:0", address, "COPY"),
node("use", address, "COPY"),
],
def_use: [],
effects: [],
parameters: [{ ordinal: 0, name: "value", data_type: "/int" }],
parameter_uses: [{ ordinal: 0, use: "use", input_index: 1 }],
truncated: false,
omitted_operations_lower_bound: 0,
known_omitted_inputs: 0,
known_omitted_edges: 0,
limitations: [],
},
},
provider,
{ subject },
);
const result = await traceNativeValues(
{ execute: async () => ok(execution) },
{ procedure: address, max_nodes: 2 },
);
if (!result.ok) throw result.error;
const parsedResult = nativeValueTraceSchema.parse(result.value);
expect(parsedResult).toMatchObject({
total_nodes: 2,
total_edges: 1,
edges: [
{
source: `${address}/parameter:0`,
target: `${address}/use`,
kind: "parameter-use",
},
],
});
});
});
describe("bounded native value dependency composition", () => { describe("bounded native value dependency composition", () => {
it("reports work truncation and stable pagination without traversing ambiguous calls", async () => { it("reports work truncation and stable pagination without traversing ambiguous calls", async () => {
const bounded = await traceNativeValues(analysis, { const bounded = await traceNativeValues(analysis, {
+2 -6
View File
@@ -21,7 +21,6 @@ export const traceNativeValues = async (
): EnhancedResult => { ): EnhancedResult => {
const input = nativeValueTraceInputSchema.parse(parameters); const input = nativeValueTraceInputSchema.parse(parameters);
const nodes: typeof nativeValueTraceSchema._output.nodes = []; const nodes: typeof nativeValueTraceSchema._output.nodes = [];
const nodeIds = new Set<string>();
const edges: typeof nativeValueTraceSchema._output.edges = []; const edges: typeof nativeValueTraceSchema._output.edges = [];
const procedures: typeof nativeValueTraceSchema._output.procedures = []; const procedures: typeof nativeValueTraceSchema._output.procedures = [];
const unknowns: typeof nativeValueTraceSchema._output.unknowns = []; const unknowns: typeof nativeValueTraceSchema._output.unknowns = [];
@@ -155,28 +154,25 @@ export const traceNativeValues = async (
bytes += size; bytes += size;
ids.add(operation.id); ids.add(operation.id);
nodes.push(node); nodes.push(node);
nodeIds.add(node.id);
} }
for (const parameter of flow.parameters) { for (const parameter of flow.parameters) {
if (nodes.length >= input.max_nodes) { if (nodes.length >= input.max_nodes) {
truncated = true; truncated = true;
break; break;
} }
const id = `${address}/parameter:${parameter.ordinal}`;
nodes.push({ nodes.push({
id, id: `${address}/parameter:${parameter.ordinal}`,
procedure: address, procedure: address,
evidence_id: evidenceId, evidence_id: evidenceId,
kind: "parameter", kind: "parameter",
operation: null, operation: null,
parameter, parameter,
}); });
nodeIds.add(id);
} }
for (const use of flow.parameter_uses) { for (const use of flow.parameter_uses) {
if ( if (
!ids.has(use.use) || !ids.has(use.use) ||
!nodeIds.has(`${address}/parameter:${use.ordinal}`) !nodes.some((node) => node.id === `${address}/parameter:${use.ordinal}`)
) )
continue; continue;
if (edges.length >= input.max_edges) { if (edges.length >= input.max_edges) {
-33
View File
@@ -60,30 +60,6 @@ describe("English Unicode case collisions", () => {
]); ]);
}); });
it("retains every spelling in a high-cardinality folded sibling group", () => {
const count = 8_192;
const variants = caseVariants("abcdefghijklm", count);
const { limitation, byPath } = annotate(variants);
const ordered = [...variants].sort();
const first = ordered[0];
const second = ordered[1];
const original = variants[0];
if (first === undefined || second === undefined || original === undefined) {
throw new Error("Expected at least two case variants");
}
expect(limitation).toBe(ENGLISH_UNICODE_CASE_INVENTORY_LIMITATION);
expect(byPath.size).toBe(count);
expect(byPath.get(first)).toEqual([
`Logical path ${first} collides under English (en-US) Unicode case folding with ${second} and ${String(count - 2)} other spellings; ${NOTE}`,
]);
expect(byPath.get(original)?.[0]).toContain(
`with ${first} and ${String(count - 2)} other spellings;`,
);
expect(
[...byPath.values()].every((limitations) => limitations.length === 1),
).toBe(true);
});
it("stays linear for large case-variant subtrees", () => { it("stays linear for large case-variant subtrees", () => {
const count = 5_000; const count = 5_000;
const paths = Array.from({ length: count }, (_, index) => [ const paths = Array.from({ length: count }, (_, index) => [
@@ -105,12 +81,3 @@ describe("English Unicode case collisions", () => {
expect(byPath.get(deep)).toEqual([]); expect(byPath.get(deep)).toEqual([]);
}); });
}); });
const caseVariants = (lowercase: string, count: number): string[] =>
Array.from({ length: count }, (_, mask) =>
[...lowercase]
.map((character, index) =>
(mask & (1 << index)) === 0 ? character : character.toUpperCase(),
)
.join(""),
);
+4 -4
View File
@@ -177,12 +177,12 @@ const englishUnicodeCaseCollisions = (
if (path.length === 0 || path === ".") continue; if (path.length === 0 || path === ".") continue;
let node = root; let node = root;
for (const part of path.split("/")) { for (const part of path.split("/")) {
const folded = englishCaseFold(part);
const spellings = node.spellingsByFold.get(folded);
if (spellings === undefined) node.spellingsByFold.set(folded, [part]);
else if (!spellings.includes(part)) spellings.push(part);
let child = node.bySpelling.get(part); let child = node.bySpelling.get(part);
if (child === undefined) { if (child === undefined) {
const folded = englishCaseFold(part);
const spellings = node.spellingsByFold.get(folded);
if (spellings === undefined) node.spellingsByFold.set(folded, [part]);
else spellings.push(part);
child = emptyCaseNode(); child = emptyCaseNode();
node.bySpelling.set(part, child); node.bySpelling.set(part, child);
} }
+1 -25
View File
@@ -33,12 +33,10 @@ import type { JsonValue } from "../domain/jsonValue.js";
import { interfaceBuilderLimitsSchema } from "../domain/apple/interfaceBuilderGraph.js"; import { interfaceBuilderLimitsSchema } from "../domain/apple/interfaceBuilderGraph.js";
import { err, ok, type Result } from "../domain/result.js"; import { err, ok, type Result } from "../domain/result.js";
import { ArtifactReaderFailure } from "./ArtifactReader.js"; import { ArtifactReaderFailure } from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import { artifactCapabilities } from "./ArtifactProviderMetadata.js"; import { artifactCapabilities } from "./ArtifactProviderMetadata.js";
import { ARTIFACT_GRAPH_PROVIDER } from "../application/InvestigationProviders.js"; import { ARTIFACT_GRAPH_PROVIDER } from "../application/InvestigationProviders.js";
import { createEvidence } from "../domain/evidence.js"; import { createEvidence } from "../domain/evidence.js";
import { createArtifactInspection } from "../domain/artifactInspection.js"; import { createArtifactInspection } from "../domain/artifactInspection.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
/** Read-only inventory and exclusively owned extraction provider. */ /** Read-only inventory and exclusively owned extraction provider. */
export class ArtifactProvider implements AnalysisProvider { export class ArtifactProvider implements AnalysisProvider {
@@ -69,8 +67,6 @@ export class ArtifactProvider implements AnalysisProvider {
} }
class ArtifactClient implements AnalysisClient { class ArtifactClient implements AnalysisClient {
readonly #resourceScope = new ArtifactResourceScope();
constructor( constructor(
private readonly target: BinaryTarget, private readonly target: BinaryTarget,
private readonly environment: Readonly<NodeJS.ProcessEnv>, private readonly environment: Readonly<NodeJS.ProcessEnv>,
@@ -188,7 +184,6 @@ class ArtifactClient implements AnalysisClient {
outputRoot: parsed.output_root, outputRoot: parsed.output_root,
environment: this.environment, environment: this.environment,
integrityPolicy: parsed.integrity_policy, integrityPolicy: parsed.integrity_policy,
resourceScope: this.#resourceScope,
}, },
options?.signal, options?.signal,
); );
@@ -225,25 +220,7 @@ class ArtifactClient implements AnalysisClient {
} }
close(): Promise<Result<null, AnalysisError>> { close(): Promise<Result<null, AnalysisError>> {
return this.#resourceScope.close().then( return Promise.resolve(ok(null));
() => ok(null),
(cause: unknown) => {
const failure =
cause instanceof ArtifactReaderFailure ? cause : undefined;
return err(
new ProviderCleanupError(
ARTIFACT_GRAPH_PROVIDER.id,
failure?.cleanup?.resources ?? ["artifact resources"],
{
reason:
failure?.cleanup?.reason ??
(cause instanceof Error ? cause.message : String(cause)),
},
{ cause, operation: "close_binary" },
),
);
},
);
} }
/** The active target's kind is outside this operation's supported targets. */ /** The active target's kind is outside this operation's supported targets. */
@@ -376,7 +353,6 @@ class ArtifactClient implements AnalysisClient {
options?: ExecutionOptions, options?: ExecutionOptions,
) { ) {
return inventoryArtifact(this.target.sourcePath ?? this.target.path, { return inventoryArtifact(this.target.sourcePath ?? this.target.path, {
resourceScope: this.#resourceScope,
environment: this.environment, environment: this.environment,
...(options?.signal === undefined ? {} : { signal: options.signal }), ...(options?.signal === undefined ? {} : { signal: options.signal }),
integrity: { mode: parsed.integrity_policy }, integrity: { mode: parsed.integrity_policy },
-31
View File
@@ -31,37 +31,6 @@ export interface ArtifactEntry {
}; };
} }
/** Snapshot mutable entry metadata before the producer later validates open(). */
export const copyArtifactEntry = (entry: ArtifactEntry): ArtifactEntry => ({
...entry,
limitations: [...entry.limitations],
...(entry.sourceIdentity === undefined
? {}
: { sourceIdentity: { ...entry.sourceIdentity } }),
});
/** Compare every portable and adapter-owned fact supplied with an entry. */
export const sameArtifactEntry = (
entry: ArtifactEntry,
produced: ArtifactEntry,
): boolean =>
entry.path === produced.path &&
entry.kind === produced.kind &&
entry.declaredSize === produced.declaredSize &&
entry.compressedSize === produced.compressedSize &&
entry.executable === produced.executable &&
entry.encrypted === produced.encrypted &&
entry.byteOffset === produced.byteOffset &&
entry.declaredSha256 === produced.declaredSha256 &&
entry.unpacked === produced.unpacked &&
entry.adapterKey === produced.adapterKey &&
entry.limitations.length === produced.limitations.length &&
entry.limitations.every(
(limitation, index) => limitation === produced.limitations[index],
) &&
entry.sourceIdentity?.device === produced.sourceIdentity?.device &&
entry.sourceIdentity?.inode === produced.sourceIdentity?.inode;
/** Read-only adapter over one directory, archive, or virtual container. */ /** Read-only adapter over one directory, archive, or virtual container. */
export interface ArtifactReader { export interface ArtifactReader {
readonly format: "directory" | ZipPackageFormat | "asar" | "file"; readonly format: "directory" | ZipPackageFormat | "asar" | "file";
-114
View File
@@ -1,114 +0,0 @@
import { expect, it } from "vitest";
import { err } from "../domain/result.js";
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
import type { ArtifactReader } from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
const deferred = <T>() => {
let resolve: (value: T) => void = () => {};
const promise = new Promise<T>((resolvePromise) => {
resolve = resolvePromise;
});
return { promise, resolve };
};
it("waits for admitted operations and shares concurrent close", async () => {
const scope = new ArtifactResourceScope();
const started = deferred<void>();
const finish = deferred<void>();
const operation = scope.run(async () => {
started.resolve();
await finish.promise;
});
await started.promise;
const close = scope.close();
expect(scope.close()).toBe(close);
await expect(scope.run(async () => undefined)).rejects.toMatchObject({
reason: "unavailable",
message: "Artifact resource scope is closed",
});
let closed = false;
void close.then(() => {
closed = true;
});
await Promise.resolve();
expect(closed).toBe(false);
finish.resolve();
await operation;
await close;
expect(closed).toBe(true);
});
it("retains the same failed reader owner and retries before admitting work", async () => {
const scope = new ArtifactResourceScope();
let closeAttempts = 0;
const reader = {
async close() {
closeAttempts += 1;
if (closeAttempts < 4) throw new Error(`close attempt ${closeAttempts}`);
},
} as unknown as ArtifactReader;
const owner = { kind: "reader" as const, resource: "fixture reader", reader };
expect(await scope.release(owner)).toMatchObject({ kind: "failed" });
expect(closeAttempts).toBe(1);
await expect(scope.run(async () => "must not run")).rejects.toMatchObject({
reason: "unavailable",
message: "Artifact cleanup remains incomplete: fixture reader",
});
expect(closeAttempts).toBe(2);
await expect(scope.close()).rejects.toMatchObject({
reason: "unavailable",
});
expect(closeAttempts).toBe(3);
await scope.close();
expect(closeAttempts).toBe(4);
});
it("retains a failed source analyzer until shutdown can verify its release", async () => {
const scope = new ArtifactResourceScope();
let releaseAllowed = false;
const failure = new AnalysisOutputError(
"analyze_javascript_application",
"Owned worker is still running",
);
const owner = {
kind: "javascript-source-analysis" as const,
resource: "fixture source analyzer",
analysis: {
async analyze() {
return err({
error: failure,
javascript: null,
module: null,
projection: null,
});
},
async close() {
if (!releaseAllowed) throw failure;
},
},
};
expect(await scope.release(owner)).toEqual({
kind: "failed",
cause: failure,
});
let admitted = false;
await expect(
scope.run(async () => {
admitted = true;
}),
).rejects.toMatchObject({
cleanup: { resources: ["fixture source analyzer"] },
});
expect(admitted).toBe(false);
await expect(scope.close()).rejects.toMatchObject({
cleanup: { resources: ["fixture source analyzer"] },
});
releaseAllowed = true;
await scope.close();
});
-156
View File
@@ -1,156 +0,0 @@
import type { JavaScriptSourceAnalysisPort } from "../domain/javascript/javascriptSourceAnalysis.js";
import type { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
import type { OwnedDirectoryHandle } from "../filesystem/OwnedDirectoryHandle.js";
import type { ArtifactReader } from "./ArtifactReader.js";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
import type { SafeOutputTree } from "./SafeOutputTree.js";
/** One artifact resource and its format-specific cleanup owner. */
export type ArtifactResourceOwner =
| {
readonly kind: "javascript-source-analysis";
readonly resource: string;
readonly analysis: JavaScriptSourceAnalysisPort;
}
| {
readonly kind: "directory-handle";
readonly resource: string;
readonly handle: OwnedDirectoryHandle;
}
| {
readonly kind: "reader";
readonly resource: string;
readonly reader: ArtifactReader;
}
| {
readonly kind: "file-handle";
readonly resource: string;
readonly handle: OwnedFileHandle;
}
| {
readonly kind: "output-tree";
readonly resource: string;
readonly tree: SafeOutputTree;
};
export type ArtifactCleanupAttempt =
| { readonly kind: "released" }
| { readonly kind: "failed"; readonly cause: unknown };
/** Caller-owned scope that retains failed artifact cleanup for a later close. */
export class ArtifactResourceScope {
readonly #pending = new Set<ArtifactResourceOwner>();
readonly #active = new Set<Promise<unknown>>();
#draining: Promise<void> | undefined;
#closed = false;
/** Admit a resource operation while the scope remains open. */
async run<T>(operation: () => Promise<T>): Promise<T> {
while (true) {
if (this.#closed)
throw new ArtifactReaderFailure(
"unavailable",
"Artifact resource scope is closed",
);
if (this.#draining !== undefined) await this.#draining;
if (this.#closed)
throw new ArtifactReaderFailure(
"unavailable",
"Artifact resource scope is closed",
);
if (this.#pending.size > 0) {
await this.#retryCleanup();
continue;
}
const running = Promise.resolve().then(operation);
this.#active.add(running);
try {
return await running;
} finally {
this.#active.delete(running);
}
}
}
/** Attempt cleanup now and retain the same owner if it fails. */
async release(owner: ArtifactResourceOwner): Promise<ArtifactCleanupAttempt> {
try {
await cleanup(owner);
this.#pending.delete(owner);
return { kind: "released" };
} catch (cause: unknown) {
this.#pending.add(owner);
return { kind: "failed", cause };
}
}
/** Retry retained owners without closing resource admission. */
#retryCleanup(): Promise<void> {
if (this.#draining !== undefined) return this.#draining;
this.#draining = this.#drainAndRetry().finally(() => {
this.#draining = undefined;
});
return this.#draining;
}
/** Stop admission, wait for active operations, and retry every retained owner. */
close(): Promise<void> {
this.#closed = true;
return this.#retryCleanup();
}
async #drainAndRetry(): Promise<void> {
while (this.#active.size > 0) await Promise.allSettled(this.#active);
const failures: {
readonly owner: ArtifactResourceOwner;
readonly cause: unknown;
}[] = [];
for (const owner of this.#pending) {
try {
await cleanup(owner);
this.#pending.delete(owner);
} catch (cause: unknown) {
failures.push({ owner, cause });
}
}
if (failures.length === 0) return;
const observations = failures.map(({ owner, cause }) =>
ArtifactReaderFailure.cleanupObservation(cause, owner.resource),
);
const resources = [
...new Set(observations.flatMap(({ resources }) => resources)),
];
throw new ArtifactReaderFailure(
"unavailable",
`Artifact cleanup remains incomplete: ${resources.join(", ")}`,
{
cause: new AggregateError(
failures.map(({ cause }) => cause),
"One or more artifact resources remain owned",
),
cleanup: {
reason: observations.map(({ reason }) => reason).join("; "),
resources,
},
},
);
}
}
const cleanup = async (owner: ArtifactResourceOwner): Promise<void> => {
switch (owner.kind) {
case "javascript-source-analysis":
await owner.analysis.close();
return;
case "reader":
await owner.reader.close();
return;
case "file-handle":
case "directory-handle":
await owner.handle.close();
return;
case "output-tree":
await owner.tree.rollback();
}
};
+89 -110
View File
@@ -19,19 +19,15 @@ import { isAbsolute, join, relative, sep } from "node:path";
import { Readable, Transform } from "node:stream"; import { Readable, Transform } from "node:stream";
import { pipeline } from "node:stream/promises"; import { pipeline } from "node:stream/promises";
import { listPackage, statFile, uncache } from "@electron/asar"; import { getRawHeader, listPackage, statFile, uncache } from "@electron/asar";
import { import {
ArtifactReaderFailure, ArtifactReaderFailure,
copyArtifactEntry,
sameArtifactEntry,
type ArtifactEntry, type ArtifactEntry,
type ArtifactReader, type ArtifactReader,
} from "./ArtifactReader.js"; } from "./ArtifactReader.js";
import { closeAsarHandle, readValidatedAsarEntry } from "./AsarEntryStream.js"; import { closeAsarHandle, readValidatedAsarEntry } from "./AsarEntryStream.js";
import { admitAsarHeader } from "./AsarHeader.js"; import { admitAsarHeader } from "./AsarHeader.js";
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
/** /**
* Official Electron ASAR adapter with range-streamed member reads. * Official Electron ASAR adapter with range-streamed member reads.
@@ -45,13 +41,6 @@ export class AsarArtifactReader implements ArtifactReader {
#archiveSize: number | undefined; #archiveSize: number | undefined;
#headerSize: number | undefined; #headerSize: number | undefined;
readonly #entries = new Map<string, AsarEntryState>(); readonly #entries = new Map<string, AsarEntryState>();
readonly #unpackedHandles = new Map<
UnpackedFileHandle,
{
readonly path: string;
readonly owner: OwnedFileHandle<UnpackedFileHandle>;
}
>();
#snapshot: { readonly path: string; readonly sha256: string } | undefined; #snapshot: { readonly path: string; readonly sha256: string } | undefined;
#snapshotRoot: string | undefined; #snapshotRoot: string | undefined;
@@ -76,7 +65,6 @@ export class AsarArtifactReader implements ArtifactReader {
try { try {
const admitted = await admitAsarHeader(headerHandle); const admitted = await admitAsarHeader(headerHandle);
this.#archiveSize = admitted.archiveBytes; this.#archiveSize = admitted.archiveBytes;
this.#headerSize = admitted.headerBytes;
} finally { } finally {
await headerHandle.close(); await headerHandle.close();
} }
@@ -84,6 +72,7 @@ export class AsarArtifactReader implements ArtifactReader {
paths = listPackage(snapshotPath, { isPack: false }).sort((left, right) => paths = listPackage(snapshotPath, { isPack: false }).sort((left, right) =>
left.localeCompare(right, "en"), left.localeCompare(right, "en"),
); );
this.#headerSize = getRawHeader(snapshotPath).headerSize;
const archiveMetadata = await lstat(snapshotPath); const archiveMetadata = await lstat(snapshotPath);
if (!archiveMetadata.isFile() || archiveMetadata.isSymbolicLink()) if (!archiveMetadata.isFile() || archiveMetadata.isSymbolicLink())
throw new ArtifactReaderFailure( throw new ArtifactReaderFailure(
@@ -167,8 +156,32 @@ export class AsarArtifactReader implements ArtifactReader {
); );
const { metadata } = state; const { metadata } = state;
const producedEntry = state.entry; const producedEntry = state.entry;
if (producedEntry.unpacked) if (producedEntry.unpacked) {
return this.#openUnpackedStream(producedEntry, signal); const handle = await this.#openUnpackedEntry(producedEntry);
try {
const observed = await handle.stat();
abortIfNeeded(signal);
if (!observed.isFile())
throw new ArtifactReaderFailure(
"path",
`ASAR unpacked entry is not a regular file: ${producedEntry.path}`,
);
} catch (cause: unknown) {
await handle.close().catch(() => undefined);
throw asarFailure(this.path, `read ${producedEntry.path}`, cause);
}
const source = handle.createReadStream({
start: 0,
autoClose: true,
});
return readValidatedAsarEntry(
source,
undefined,
producedEntry.path,
this.path,
signal,
);
}
const archiveSize = this.#archiveSize; const archiveSize = this.#archiveSize;
const headerSize = this.#headerSize; const headerSize = this.#headerSize;
const offset = const offset =
@@ -302,39 +315,22 @@ export class AsarArtifactReader implements ArtifactReader {
} }
async close(): Promise<void> { async close(): Promise<void> {
let cleanupFailure: ArtifactReaderFailure | undefined;
for (const [handle, { path }] of this.#unpackedHandles) {
try {
await this.#closeUnpackedHandle(handle);
} catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup(
cleanupFailure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, path),
);
}
}
const root = this.#snapshotRoot; const root = this.#snapshotRoot;
if (root !== undefined) { if (root !== undefined) {
try { try {
await this.removeSnapshot(root); await this.removeSnapshot(root);
if (this.#snapshotRoot === root) this.#snapshotRoot = undefined;
} catch (cause: unknown) { } catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup( throw ArtifactReaderFailure.withCleanup(cause, {
cleanupFailure ?? cause, reason: `Could not remove ASAR snapshot: ${root}`,
{ resources: [root],
reason: `Could not remove ASAR snapshot: ${root}`, });
resources: [root],
},
);
} }
this.#snapshotRoot = undefined;
} }
if (root === undefined || this.#snapshotRoot === undefined) { if (this.#snapshot !== undefined) uncache(this.#snapshot.path);
if (this.#snapshot !== undefined) uncache(this.#snapshot.path); this.#snapshot = undefined;
this.#snapshot = undefined; this.#resetArchiveState();
this.#resetArchiveState(); this.#entries.clear();
this.#entries.clear();
}
if (cleanupFailure !== undefined) throw cleanupFailure;
} }
#snapshotPath(): string { #snapshotPath(): string {
@@ -350,12 +346,8 @@ export class AsarArtifactReader implements ArtifactReader {
return []; return [];
} }
async #openUnpackedStream( async #openUnpackedEntry(entry: ArtifactEntry): Promise<UnpackedFileHandle> {
entry: ArtifactEntry,
signal?: AbortSignal,
): Promise<Readable> {
const unpackedRoot = `${this.path}.unpacked`; const unpackedRoot = `${this.path}.unpacked`;
let handle: UnpackedFileHandle | undefined;
try { try {
const rootMetadata = await lstat(unpackedRoot); const rootMetadata = await lstat(unpackedRoot);
if (!rootMetadata.isDirectory() || rootMetadata.isSymbolicLink()) if (!rootMetadata.isDirectory() || rootMetadata.isSymbolicLink())
@@ -382,21 +374,22 @@ export class AsarArtifactReader implements ArtifactReader {
"path", "path",
`ASAR unpacked entry is not a regular file: ${entry.path}`, `ASAR unpacked entry is not a regular file: ${entry.path}`,
); );
handle = await this.openUnpackedFile( const handle = await this.openUnpackedFile(
canonical, canonical,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
); );
const openedHandle = handle; const openedMetadata = await handle
this.#unpackedHandles.set(openedHandle, { .stat()
path: canonical, .catch(async (cause: unknown) => {
owner: new OwnedFileHandle(openedHandle), await handle.close().catch(() => undefined);
}); throw cause;
const openedMetadata = await openedHandle.stat(); });
if ( if (
!openedMetadata.isFile() || !openedMetadata.isFile() ||
openedMetadata.dev !== pathMetadata.dev || openedMetadata.dev !== pathMetadata.dev ||
openedMetadata.ino !== pathMetadata.ino openedMetadata.ino !== pathMetadata.ino
) { ) {
await handle.close();
throw new ArtifactReaderFailure( throw new ArtifactReaderFailure(
"integrity", "integrity",
`ASAR unpacked entry changed before read: ${entry.path}`, `ASAR unpacked entry changed before read: ${entry.path}`,
@@ -409,46 +402,25 @@ export class AsarArtifactReader implements ArtifactReader {
}, },
); );
} }
abortIfNeeded(signal); return handle;
const source = readFileHandleChunks(openedHandle, {
start: 0,
});
source.once("close", () => {
void this.#closeUnpackedHandle(openedHandle).catch(() => undefined);
});
return readValidatedAsarEntry(
source,
undefined,
entry.path,
this.path,
signal,
);
} catch (cause: unknown) { } catch (cause: unknown) {
let failure = asarUnpackedEntryFailure(this.path, entry, cause); if (cause instanceof ArtifactReaderFailure) throw cause;
if (handle !== undefined && this.#unpackedHandles.has(handle)) { if (entry.unpacked && isMissingFile(cause))
try { throw new ArtifactReaderFailure(
await this.#closeUnpackedHandle(handle); "unavailable",
} catch (cleanupCause: unknown) { `ASAR unpacked entry bytes are unavailable: ${entry.path}`,
failure = ArtifactReaderFailure.withCleanup( { cause },
failure, {
ArtifactReaderFailure.cleanupObservation( logicalPath: entry.path,
cleanupCause, declaredSha256: entry.declaredSha256,
this.#unpackedHandles.get(handle)?.path ?? entry.path, calculatedSha256: null,
), unpacked: true,
); },
} );
} throw asarFailure(this.path, `read ${entry.path}`, cause);
throw failure;
} }
} }
async #closeUnpackedHandle(handle: UnpackedFileHandle): Promise<void> {
const owned = this.#unpackedHandles.get(handle);
if (owned === undefined) return;
await owned.owner.close();
this.#unpackedHandles.delete(handle);
}
#resetArchiveState(): void { #resetArchiveState(): void {
this.#archiveSize = undefined; this.#archiveSize = undefined;
this.#headerSize = undefined; this.#headerSize = undefined;
@@ -461,36 +433,43 @@ type AsarEntryState = {
readonly entry: ArtifactEntry; readonly entry: ArtifactEntry;
}; };
type UnpackedFileHandle = { type UnpackedFileHandle = {
readonly fd: number;
stat(): Promise<Stats>; stat(): Promise<Stats>;
close(): Promise<void>; close(): Promise<void>;
read: FileHandle["read"]; createReadStream: FileHandle["createReadStream"];
}; };
type OpenAsarUnpackedFile = ( type OpenAsarUnpackedFile = (
path: string, path: string,
flags: number, flags: number,
) => Promise<UnpackedFileHandle>; ) => Promise<UnpackedFileHandle>;
const asarUnpackedEntryFailure = ( const copyArtifactEntry = (entry: ArtifactEntry): ArtifactEntry => ({
archivePath: string, ...entry,
limitations: [...entry.limitations],
...(entry.sourceIdentity === undefined
? {}
: { sourceIdentity: { ...entry.sourceIdentity } }),
});
const sameArtifactEntry = (
entry: ArtifactEntry, entry: ArtifactEntry,
cause: unknown, produced: ArtifactEntry,
): ArtifactReaderFailure => { ): boolean =>
if (cause instanceof ArtifactReaderFailure) return cause; entry.path === produced.path &&
if (entry.unpacked && isMissingFile(cause)) entry.kind === produced.kind &&
return new ArtifactReaderFailure( entry.declaredSize === produced.declaredSize &&
"unavailable", entry.compressedSize === produced.compressedSize &&
`ASAR unpacked entry bytes are unavailable: ${entry.path}`, entry.executable === produced.executable &&
{ cause }, entry.encrypted === produced.encrypted &&
{ entry.byteOffset === produced.byteOffset &&
logicalPath: entry.path, entry.declaredSha256 === produced.declaredSha256 &&
declaredSha256: entry.declaredSha256, entry.unpacked === produced.unpacked &&
calculatedSha256: null, entry.adapterKey === produced.adapterKey &&
unpacked: true, entry.limitations.length === produced.limitations.length &&
}, entry.limitations.every(
); (limitation, index) => limitation === produced.limitations[index],
return asarFailure(archivePath, `read ${entry.path}`, cause); ) &&
}; entry.sourceIdentity?.device === produced.sourceIdentity?.device &&
entry.sourceIdentity?.inode === produced.sourceIdentity?.inode;
const isAsarFileMetadata = ( const isAsarFileMetadata = (
metadata: ReturnType<typeof statFile>, metadata: ReturnType<typeof statFile>,
+69 -238
View File
@@ -1,33 +1,23 @@
import { constants, type Stats } from "node:fs"; import { constants } from "node:fs";
import { import { lstat, open, opendir, realpath } from "node:fs/promises";
lstat,
open,
opendir,
realpath,
type FileHandle,
} from "node:fs/promises";
import { isAbsolute, join, relative, sep } from "node:path"; import { isAbsolute, join, relative, sep } from "node:path";
import { Readable } from "node:stream"; import type { Readable } from "node:stream";
import { import {
ArtifactReaderFailure, ArtifactReaderFailure,
type ArtifactEntry, type ArtifactEntry,
type ArtifactReader, type ArtifactReader,
} from "./ArtifactReader.js"; } from "./ArtifactReader.js";
import { OwnedDirectoryHandle } from "../filesystem/OwnedDirectoryHandle.js";
import { OwnedFileHandle } from "../filesystem/OwnedFileHandle.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
/** Non-symlink-following directory reader. */ /** Non-symlink-following directory reader. */
export class DirectoryArtifactReader implements ArtifactReader { export class DirectoryArtifactReader implements ArtifactReader {
readonly format = "directory" as const; readonly format = "directory" as const;
#rootPromise: Promise<string> | undefined; readonly #rootPromise: Promise<string>;
readonly #rootPath: string; readonly #rootPath: string;
readonly #directories = new Set<OwnedDirectoryHandle>();
readonly #files = new Map<OwnedFileHandle, string>();
constructor(root: string) { constructor(root: string) {
this.#rootPath = root; this.#rootPath = root;
this.#rootPromise = realpath(root);
} }
/** Enumerate entries, optionally restricting descent while retaining directories. */ /** Enumerate entries, optionally restricting descent while retaining directories. */
@@ -35,7 +25,9 @@ export class DirectoryArtifactReader implements ArtifactReader {
signal?: AbortSignal, signal?: AbortSignal,
descendInto?: (path: string) => boolean, descendInto?: (path: string) => boolean,
): AsyncIterable<ArtifactEntry> { ): AsyncIterable<ArtifactEntry> {
const root = await this.#resolveRoot(); const root = await this.#rootPromise.catch((cause: unknown) => {
throw directoryIoFailure("resolve artifact root", this.#rootPath, cause);
});
const pending = [root]; const pending = [root];
while (pending.length > 0) { while (pending.length > 0) {
abortIfNeeded(signal); abortIfNeeded(signal);
@@ -44,100 +36,62 @@ export class DirectoryArtifactReader implements ArtifactReader {
await withDirectoryIoContext("inspect directory", directory, () => await withDirectoryIoContext("inspect directory", directory, () =>
assertContainedDirectory(root, directory), assertContainedDirectory(root, directory),
); );
for await (const entry of this.#readDirectory( const handle = await opendir(directory).catch((cause: unknown) => {
directory, throw directoryIoFailure("open directory", directory, cause);
pending, });
signal, const directories: string[] = [];
descendInto, try {
)) for await (const child of handle) {
yield entry; abortIfNeeded(signal);
const absolute = join(directory, child.name);
const metadata = await lstat(absolute).catch((cause: unknown) => {
throw directoryIoFailure("inspect entry", absolute, cause);
});
const path = relative(root, absolute).split(sep).join("/");
const kind = metadata.isSymbolicLink()
? "symlink"
: metadata.isDirectory()
? "directory"
: metadata.isFile()
? "file"
: undefined;
if (kind === undefined) continue;
yield {
path,
kind,
declaredSize: kind === "file" ? metadata.size : null,
compressedSize: null,
executable: (metadata.mode & 0o111) !== 0,
encrypted: false,
byteOffset: null,
declaredSha256: null,
unpacked: false,
limitations:
kind === "symlink"
? ["Symlink target was not followed or disclosed."]
: [],
adapterKey: absolute,
...(kind === "file"
? {
sourceIdentity: { device: metadata.dev, inode: metadata.ino },
}
: {}),
};
if (kind === "directory" && (descendInto?.(path) ?? true))
directories.push(absolute);
}
} catch (cause: unknown) {
if (cause instanceof ArtifactReaderFailure) throw cause;
throw directoryIoFailure("read directory", directory, cause);
} finally {
// best-effort cleanup: directory-handle close must not mask traversal.
await handle.close().catch(() => undefined);
}
await withDirectoryIoContext("inspect directory", directory, () => await withDirectoryIoContext("inspect directory", directory, () =>
assertContainedDirectory(root, directory), assertContainedDirectory(root, directory),
); );
} directories.reverse();
} pending.push(...directories);
async #resolveRoot(): Promise<string> {
return (this.#rootPromise ??= realpath(this.#rootPath)).catch(
(cause: unknown) => {
throw directoryIoFailure(
"resolve artifact root",
this.#rootPath,
cause,
);
},
);
}
async *#readDirectory(
directory: string,
pending: string[],
signal?: AbortSignal,
descendInto?: (path: string) => boolean,
): AsyncIterable<ArtifactEntry> {
const root = await this.#resolveRoot();
const handle = new OwnedDirectoryHandle(
await opendir(directory).catch((cause: unknown) => {
throw directoryIoFailure("open directory", directory, cause);
}),
);
this.#directories.add(handle);
const childDirectories: string[] = [];
let traversal:
| { readonly kind: "complete" }
| { readonly kind: "failed"; readonly cause: unknown } = {
kind: "complete",
};
try {
for (;;) {
const child = await handle.handle.read();
if (child === null) break;
abortIfNeeded(signal);
const absolute = join(directory, child.name);
const metadata = await lstat(absolute).catch((cause: unknown) => {
throw directoryIoFailure("inspect entry", absolute, cause);
});
const path = relative(root, absolute).split(sep).join("/");
const entry = artifactEntry(path, absolute, metadata);
if (entry === undefined) continue;
yield entry;
if (entry.kind === "directory" && (descendInto?.(path) ?? true))
childDirectories.push(absolute);
}
} catch (cause: unknown) {
traversal = {
kind: "failed",
cause:
cause instanceof ArtifactReaderFailure
? cause
: directoryIoFailure("read directory", directory, cause),
};
} finally {
await this.#finishDirectory(handle, directory, traversal);
}
if (traversal.kind === "failed") throw traversal.cause;
childDirectories.reverse();
pending.push(...childDirectories);
}
async #finishDirectory(
handle: OwnedDirectoryHandle,
path: string,
traversal:
| { readonly kind: "complete" }
| { readonly kind: "failed"; readonly cause: unknown },
): Promise<void> {
try {
await handle.close();
this.#directories.delete(handle);
} catch (cause: unknown) {
const closeFailure = directoryIoFailure("close directory", path, cause);
if (traversal.kind === "failed")
throw ArtifactReaderFailure.withCleanup(
traversal.cause,
ArtifactReaderFailure.cleanupObservation(closeFailure, path),
);
throw closeFailure;
} }
} }
@@ -148,16 +102,14 @@ export class DirectoryArtifactReader implements ArtifactReader {
"format", "format",
"Only regular files can be opened", "Only regular files can be opened",
); );
const fileHandle = await open( const handle = await open(
entry.adapterKey, entry.adapterKey,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
).catch((cause: unknown) => { ).catch((cause: unknown) => {
throw directoryIoFailure("open entry", entry.adapterKey, cause); throw directoryIoFailure("open entry", entry.adapterKey, cause);
}); });
const owner = new OwnedFileHandle(fileHandle);
this.#files.set(owner, entry.adapterKey);
try { try {
const observed = await owner.handle.stat().catch((cause: unknown) => { const observed = await handle.stat().catch((cause: unknown) => {
throw directoryIoFailure( throw directoryIoFailure(
"inspect opened entry", "inspect opened entry",
entry.adapterKey, entry.adapterKey,
@@ -175,76 +127,16 @@ export class DirectoryArtifactReader implements ArtifactReader {
`Directory entry changed before open: ${entry.path}`, `Directory entry changed before open: ${entry.path}`,
); );
} }
abortIfNeeded(signal); return handle.createReadStream({ autoClose: true });
const source = Readable.from(
mapFileReadCancellation(owner.handle, signal),
{ objectMode: false },
);
source.once("close", () => {
void this.#closeFile(owner).catch(() => undefined);
});
return source;
} catch (cause: unknown) { } catch (cause: unknown) {
try { // best-effort cleanup: file-handle close must not mask the open failure.
await this.#closeFile(owner); await handle.close().catch(() => undefined);
} catch (closeCause: unknown) {
throw ArtifactReaderFailure.withCleanup(
cause,
ArtifactReaderFailure.cleanupObservation(
directoryIoFailure(
"close opened entry",
entry.adapterKey,
closeCause,
),
entry.adapterKey,
),
);
}
throw cause; throw cause;
} }
} }
async close(): Promise<void> { close(): Promise<void> {
const failures: unknown[] = []; return Promise.resolve();
for (const [owner, path] of this.#files) {
try {
await this.#closeFile(owner);
} catch (cause: unknown) {
failures.push(directoryIoFailure("close opened entry", path, cause));
}
}
for (const owner of this.#directories) {
try {
await owner.close();
this.#directories.delete(owner);
} catch (cause: unknown) {
failures.push(
directoryIoFailure("close directory", owner.handle.path, cause),
);
}
}
if (failures.length > 0) {
const reason = failures.map(failureMessage).join("; ");
throw new ArtifactReaderFailure(
"io",
`Directory artifact cleanup failed: ${reason}`,
{
cause: new AggregateError(failures),
cleanup: {
reason,
resources: [
...this.#files.values(),
...[...this.#directories].map((owner) => owner.handle.path),
],
},
},
);
}
}
async #closeFile(owner: OwnedFileHandle<FileHandle>): Promise<void> {
await owner.close();
this.#files.delete(owner);
} }
provenance(): readonly [] { provenance(): readonly [] {
@@ -252,42 +144,6 @@ export class DirectoryArtifactReader implements ArtifactReader {
} }
} }
const artifactEntry = (
path: string,
absolute: string,
metadata: Stats,
): ArtifactEntry | undefined => {
const kind = metadata.isSymbolicLink()
? "symlink"
: metadata.isDirectory()
? "directory"
: metadata.isFile()
? "file"
: undefined;
if (kind === undefined) return undefined;
return {
path,
kind,
declaredSize: kind === "file" ? metadata.size : null,
compressedSize: null,
executable: (metadata.mode & 0o111) !== 0,
encrypted: false,
byteOffset: null,
declaredSha256: null,
unpacked: false,
limitations:
kind === "symlink"
? ["Symlink target was not followed or disclosed."]
: [],
adapterKey: absolute,
...(kind === "file"
? {
sourceIdentity: { device: metadata.dev, inode: metadata.ino },
}
: {}),
};
};
const directoryIoFailure = ( const directoryIoFailure = (
operation: string, operation: string,
path: string, path: string,
@@ -301,38 +157,13 @@ const directoryIoFailure = (
typeof cause.code === "string" typeof cause.code === "string"
? ` (${cause.code})` ? ` (${cause.code})`
: ""; : "";
const detail = failureMessage(cause);
return new ArtifactReaderFailure( return new ArtifactReaderFailure(
"io", "io",
`Could not ${operation} at ${path}${code}: ${detail}`, `Could not ${operation} at ${path}${code}`,
{ cause }, { cause },
); );
}; };
const failureMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const mapFileReadCancellation = async function* (
handle: FileHandle,
signal?: AbortSignal,
): AsyncGenerator<Buffer> {
try {
for await (const chunk of readFileHandleChunks(handle, { signal })) {
abortIfNeeded(signal);
yield chunk;
}
abortIfNeeded(signal);
} catch (cause: unknown) {
if (signal?.aborted === true)
throw new ArtifactReaderFailure(
"cancelled",
"Directory artifact read cancelled",
{ cause },
);
throw cause;
}
};
const withDirectoryIoContext = async <T>( const withDirectoryIoContext = async <T>(
operation: string, operation: string,
path: string, path: string,
+35 -78
View File
@@ -1,84 +1,41 @@
import { writeFile } from "node:fs/promises"; import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { describe, expect, it, onTestFinished } from "vitest"; import { describe, expect, it } from "vitest";
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
import { CPU, fatImage, machoImage } from "./apple/MachoImage.fixture.js";
import { MachOSliceArtifactReader } from "./MachOSliceArtifactReader.js"; import { MachOSliceArtifactReader } from "./MachOSliceArtifactReader.js";
import { ok } from "../domain/result.js";
import { openRegularFile } from "../filesystem/RegularFile.js";
import type { NativeCommandRunner } from "../native/CommandRunner.js";
describe("MachOSliceArtifactReader", () => { /**
it("keeps a borrowed source usable after a cancelled slice read", async () => { * Read a 32-bit FAT table directly. The host's /bin/ls changes between macOS
const root = await createTestTempDirectory("rea-slices-cancelled-"); * releases, so the expected slice layout comes from the file itself.
const binary = join(root, "fat"); */
const x86 = machoImage({ cpu: CPU.x86_64 }); const fatSlices = async (
const arm = Buffer.concat([ path: string,
Buffer.from(machoImage({ cpu: CPU.arm64 })), ): Promise<readonly (readonly [number, number])[]> => {
Buffer.alloc(160_000, 0x5a), const bytes = await readFile(path);
]); expect(bytes.readUInt32BE(0)).toBe(0xca_fe_ba_be);
await writeFile( return Array.from({ length: bytes.readUInt32BE(4) }, (_, index) => {
binary, const entry = 8 + index * 20;
fatImage([ return [bytes.readUInt32BE(entry + 8), bytes.readUInt32BE(entry + 12)];
{ cpu: CPU.x86_64, bytes: x86 },
{ cpu: CPU.arm64, bytes: arm },
]),
);
const handle = await openRegularFile(binary, { symlinks: "reject" });
const source = { handle, initial: await handle.stat() };
const runner: NativeCommandRunner = {
run: () =>
Promise.resolve(
ok({
tool: "lipo",
executable: "/usr/bin/lipo",
executableSha256: "1".repeat(64),
toolVersion: null,
versionReason: "fixture",
arguments: ["-detailed_info", binary],
stdout: `architecture x86_64\n cputype 16777223\n cpusubtype 3\n offset 4096\n size ${String(x86.length)}\n align 2^12 (4096)\narchitecture arm64\n cputype 16777228\n cpusubtype 0\n offset 8192\n size ${String(arm.length)}\n align 2^12 (4096)\n`,
stderr: "",
stdoutBytes: 1,
stderrBytes: 0,
exitCode: 0,
signal: null,
}),
),
};
const reader = new MachOSliceArtifactReader(binary, {}, runner, source);
onTestFinished(async () => {
await reader.close();
await handle.close();
});
const entries = [];
for await (const entry of reader.entries()) entries.push(entry);
const armEntry = entries.find((entry) => entry.path === "slices/arm64");
if (armEntry === undefined) throw new Error("Expected an arm64 slice");
const controller = new AbortController();
const cancelledRead = await reader.open(armEntry, controller.signal);
const iterator = cancelledRead[Symbol.asyncIterator]();
const firstChunk = await iterator.next();
expect(firstChunk.value).toHaveLength(64 * 1024);
controller.abort();
await expect(iterator.next()).rejects.toMatchObject({
reason: "cancelled",
});
const verifiedEntries = [];
for await (const entry of reader.entries()) verifiedEntries.push(entry);
const verifiedArmEntry = verifiedEntries.find(
(entry) => entry.path === "slices/arm64",
);
if (verifiedArmEntry === undefined)
throw new Error("Expected arm64 after re-verifying the borrowed source");
const verifiedChunks: Buffer[] = [];
const verifiedRead = await reader.open(verifiedArmEntry);
for await (const chunk of verifiedRead)
verifiedChunks.push(Buffer.from(chunk));
expect(Buffer.concat(verifiedChunks)).toEqual(arm);
}); });
};
describe("Mach-O slice reader with system lipo", () => {
it.skipIf(process.platform !== "darwin")(
"checks real /bin/ls lipo CPU symbols and ptrauth capabilities against the FAT table",
async () => {
const reader = new MachOSliceArtifactReader("/bin/ls", {});
const entries = [];
for await (const entry of reader.entries()) entries.push(entry);
expect(entries.map(({ path }) => path)).toEqual([
"slices/x86_64",
"slices/arm64e",
]);
expect(
entries.map(({ byteOffset, declaredSize }) => [
byteOffset,
declaredSize,
]),
).toEqual(await fatSlices("/bin/ls"));
},
);
}); });

Some files were not shown because too many files have changed in this diff Show More