Compare commits

..
Author SHA1 Message Date
Kaoru0822andClaude Opus 5.5 92fddf3176 fix(adb): share host configuration refusals with apktool
The ADB provider, which the apktool provider mirrors, had the same two
defects: a relative REA_ADB_PATH escaped as UNKNOWN because selection
ran before the try that maps configuration failures, and a missing adb
binary projected generic target advice instead of its reason.

Move the configuration-failure-to-refusal mapping into one
providerConfigurationRefusal helper used by both providers, resolve the
ADB selection inside the mapped try, and share the CLI refusal assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 20:05:33 +00:00
Kaoru0822andClaude Opus 5.5 9655845bbc fix(apktool): name host requirements in configuration refusals
A missing apktool launcher projected the generic "unavailable for the
current target" message and remediation, leaving the actual reason only
in details.reason. A relative REA_APKTOOL_COMMAND escaped as UNKNOWN
because command selection ran before the try that maps configuration
failures.

Resolve the selection inside that try for both execution and
availability, and pass the refusal reason as userMessage, as JADX and
the adapters fixed in #1224 do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 19:35:57 +00:00
212 changed files with 2017 additions and 10708 deletions
@@ -9,19 +9,7 @@ provenance; inputs and recovered code are never executed. Recovery requires a
caller-supplied Wakaru on Linux x64; see [the recovery guide](https://github.com/morluto/rea/blob/main/docs/javascript-recovery.md).
Use `analyze_javascript_application` directly on the operator-supplied ASAR or
extracted tree. When the question concerns known source paths and the connected
schema advertises `source_paths`, select exact inventoried paths such as
`["package.json", "main.js", "preload.js"]`. Include supporting manifests or
source maps explicitly; dependencies are not implicitly analyzed. REA retains
the complete artifact inventory and identity but reads source text and analyzes
only selected files. The result and summary record `source_selection`; files
outside scope and their relationships remain unknown, so partial application
coverage does not mean the selected source failed. Use repeated `--source-path`
on either static application CLI route. Omit the selection when the question
requires complete application analysis. `detail: "summary"` reduces delivery
and does not by itself reduce analysis work.
Exact `#alias` specifiers resolve through the importer's
extracted tree. Exact `#alias` specifiers resolve through the importer's
enclosing package `imports` map to package-local files; nested `node_modules`
packages use their own map. Wildcard, recursive, and external-package import
targets stay unresolved with that limitation. When size is unknown or large and the connected schema advertises
+1 -1
View File
@@ -493,7 +493,7 @@ jobs:
tests/acceptance/webSourceLocationCli.test.ts
tests/boundary/cli/referenceSourceImportStatus.test.ts
tests/boundary/filesystem/asarInventoryFreshness.test.ts
tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts
tests/composition/javascript-applications/javascriptArtifactReconstruction.test.ts
tests/conformance/providers/ghidra/windowsCapabilities.test.ts
src/ghidra/GhidraInstallation.test.ts
tests/boundary/providers/ghidra/ghidraTransport.test.ts
+2 -2
View File
@@ -1,6 +1,6 @@
{
"_comment": "Test files remain included so the gate measures duplication across both src/ and tests/. The 3.0% limit leaves headroom for ordinary changes while keeping duplication visible.",
"threshold": 3.0,
"_comment": "Test files are intentionally NOT excluded: the gate used to report zero clones because it skipped **/*.test.ts, hiding all test-tree duplication. Current actual is ~2.47% lines across src+ and tests/, down from 3.4% when tests were included. The 2.6% limit leaves modest headroom for ordinary changes; lower it as duplication is removed.",
"threshold": 2.6,
"reporters": ["consoleFull"],
"ignore": [
"node_modules/**",
-2
View File
@@ -18,8 +18,6 @@ Give each interpretation and resource a clear owner. Preserve identity, omission
Assess brittleness in touched code during implementation, debugging, review, and maintenance. Fix the producing representation or ownership rule before adding downstream repairs. Normalize once at the owning boundary using format-aware parsers. Remove superseded representations, aliases, wrappers, and redundant tests after verifying affected consumers. Introduce abstractions only when they remove observed complexity.
When fixing a bug, inspect the producing boundary and affected consumers for duplicated interpretation or unclear ownership. Complete any directly related refactor in the same change and remove superseded code. Passing regression tests alone does not establish that the brittleness is resolved. If related cleanup remains, explain what remains and why.
Parse unknown boundary values and model expected failures with the tagged error algebra and `Result`. Preserve partial facts and meaningful failure reasons. Keep identity and lookup values separate from display formatting; preserve source values and explicit unknowns when normalization loses information. Absolute paths, file URLs, and HTTP paths have different semantics.
Validate advertised JSON Schemas after SDK conversion against their declared dialect and actual clients. Bind handlers to named contracts with exact types; catalog ordering must never select an operation or schema.
@@ -1,38 +0,0 @@
import java.util.Map;
import ghidra.app.plugin.core.analysis.AutoAnalysisManager;
import ghidra.app.util.headless.HeadlessScript;
/** Disable the named auto-analyzers before default auto-analysis. */
public final class ReaGhidraAnalysisOptions extends HeadlessScript {
@Override
public void run() throws Exception {
String[] arguments = getScriptArgs();
if (arguments.length != 1 || arguments[0].isBlank()) {
throw new IllegalArgumentException(
"ReaGhidraAnalysisOptions requires one comma-separated analyzer-name argument");
}
String[] names = arguments[0].split(",");
AutoAnalysisManager analysis = AutoAnalysisManager.getAnalysisManager(currentProgram);
Map<String, String> options = getCurrentAnalysisOptionsAndValues(currentProgram);
for (String requestedName : names) {
monitor.checkCancelled();
String name = requestedName.trim();
if (analysis.getAnalyzer(name) == null || !options.containsKey(name)) {
throw new IllegalArgumentException("Unknown Ghidra analyzer for this program: " + name);
}
}
for (String requestedName : names) {
monitor.checkCancelled();
setAnalysisOption(currentProgram, requestedName.trim(), "false");
}
options = getCurrentAnalysisOptionsAndValues(currentProgram);
for (String requestedName : names) {
monitor.checkCancelled();
String name = requestedName.trim();
if (!"false".equals(options.get(name))) {
throw new IllegalStateException("Ghidra analyzer remained enabled: " + name);
}
}
}
}
-5
View File
@@ -450,11 +450,6 @@ public final class ReaGhidraBridge extends HeadlessScript {
String seeds = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
.getString("REA analysis seeds", null);
if (seeds != null) result.add("analysis_seeds", JsonParser.parseString(seeds));
// Written by ReaGhidraMipsSoftFloat after analysis; REA requires it
// exactly when the committed profile declares soft-float storage.
String softFloat = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
.getString("REA MIPS soft-float storage", null);
if (softFloat != null) result.add("mips_soft_float_storage", JsonParser.parseString(softFloat));
return result;
}
-176
View File
@@ -1,176 +0,0 @@
import ghidra.app.script.GhidraScript;
import ghidra.program.database.SpecExtension;
import ghidra.program.model.data.AbstractFloatDataType;
import ghidra.program.model.data.DataType;
import ghidra.program.model.data.DoubleDataType;
import ghidra.program.model.data.FloatDataType;
import ghidra.program.model.data.FunctionDefinition;
import ghidra.program.model.data.IntegerDataType;
import ghidra.program.model.data.ParameterDefinition;
import ghidra.program.model.data.TypeDef;
import ghidra.program.model.lang.CompilerSpec;
import ghidra.program.model.lang.PrototypeModel;
import ghidra.program.model.lang.Register;
import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.Parameter;
import ghidra.program.model.listing.Program;
import ghidra.program.model.listing.VariableStorage;
import ghidra.program.model.pcode.XmlEncode;
import java.io.StringReader;
import java.io.StringWriter;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.regex.Pattern;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
/**
* Soft-float o32 passes and returns float/double values exactly where same-size integers go.
* Ghidra's MIPS32 compiler specifications model only hard-float o32, so every typed float
* prototype (functions, externals, thunks via their targets, and function-pointer definitions)
* would otherwise read $f12/$f14/$f0. Derive a program-specific prototype model from the
* program's own default o32 model with its floating-point entries and rules removed, install it
* as a specification extension, and assign it to every callable whose formal types use floats.
*/
public final class ReaGhidraMipsSoftFloat extends GhidraScript {
static final String REPORT_KEY = "REA MIPS soft-float storage";
static final String POLICY = "o32-soft-float-prototype-v1";
static final String PROTOTYPE = "__rea_soft_float";
private static final Pattern FPU_REGISTER = Pattern.compile("f\\d+(_\\d+)?");
@Override
public void run() throws Exception {
String processor = currentProgram.getLanguage().getProcessor().toString();
if (!processor.equals("MIPS") || currentProgram.getDefaultPointerSize() != 4)
throw new IllegalStateException(
"REA soft-float storage requires a 32-bit MIPS program, not " + processor);
CompilerSpec spec = currentProgram.getCompilerSpec();
PrototypeModel base = spec.getDefaultCallingConvention();
new SpecExtension(currentProgram)
.addReplaceCompilerSpecExtension(softFloatPrototype(base, spec), monitor);
PrototypeModel soft = currentProgram.getCompilerSpec().getCallingConvention(PROTOTYPE);
if (soft == null)
throw new IllegalStateException("REA soft-float prototype was not installed");
requireIntegerStorage(
soft.getStorageLocations(
currentProgram,
new DataType[] {
FloatDataType.dataType, IntegerDataType.dataType, DoubleDataType.dataType,
FloatDataType.dataType, DoubleDataType.dataType
},
false),
"soft-float prototype self-check");
int functions = 0;
List<Function> callables = new ArrayList<>();
currentProgram.getFunctionManager().getFunctions(true).forEach(callables::add);
currentProgram.getFunctionManager().getExternalFunctions().forEach(callables::add);
for (Function function : callables) {
// A thunk's signature is its target's; the target is in this list.
if (function.isThunk() || !usesFloat(function)) continue;
if (function.hasCustomVariableStorage()) function.setCustomVariableStorage(false);
function.setCallingConvention(PROTOTYPE);
List<VariableStorage> storage = new ArrayList<>();
storage.add(function.getReturn().getVariableStorage());
for (Parameter parameter : function.getParameters()) storage.add(parameter.getVariableStorage());
requireIntegerStorage(storage.toArray(VariableStorage[]::new), function.getName(true));
functions += 1;
}
int definitions = 0;
Iterator<DataType> types = currentProgram.getDataTypeManager().getAllDataTypes();
while (types.hasNext()) {
if (!(types.next() instanceof FunctionDefinition definition) || !usesFloat(definition))
continue;
definition.setCallingConvention(PROTOTYPE);
definitions += 1;
}
// Written last: any failure above leaves no report, and REA refuses the session.
currentProgram.getOptions(Program.PROGRAM_INFO).setString(
REPORT_KEY,
"{\"policy\":\"" + POLICY + "\",\"functions\":" + functions +
",\"definitions\":" + definitions + "}");
println("REA MIPS soft-float prototype: functions=" + functions + " definitions=" + definitions);
}
/** The default o32 model without floating-point registers or float-consuming rules. */
private static String softFloatPrototype(PrototypeModel base, CompilerSpec spec) throws Exception {
XmlEncode encoder = new XmlEncode();
base.encode(encoder, spec.getPcodeInjectLibrary());
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
Document document =
factory.newDocumentBuilder().parse(new InputSource(new StringReader(encoder.toString())));
Element root = document.getDocumentElement();
if (!root.getTagName().equals("prototype"))
throw new IllegalStateException("Unexpected default prototype encoding " + root.getTagName());
root.setAttribute("name", PROTOTYPE);
List<Node> removed = new ArrayList<>();
collect(root.getElementsByTagName("pentry"), "metatype", "float", null, removed);
collect(root.getElementsByTagName("consume"), "storage", "float", "rule", removed);
if (removed.isEmpty())
throw new IllegalStateException("Default prototype has no floating-point entries to remove");
for (Node node : removed) node.getParentNode().removeChild(node);
StringWriter xml = new StringWriter();
var transformer = TransformerFactory.newInstance().newTransformer();
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
transformer.transform(new DOMSource(document), new StreamResult(xml));
return xml.toString();
}
/** Select matching elements, or their nearest `ancestor` element when one is named. */
private static void collect(
NodeList nodes, String attribute, String value, String ancestor, List<Node> selected) {
for (int index = 0; index < nodes.getLength(); index += 1) {
Element element = (Element) nodes.item(index);
if (!value.equals(element.getAttribute(attribute))) continue;
Node target = element;
while (ancestor != null && target != null && !ancestor.equals(target.getNodeName()))
target = target.getParentNode();
if (target != null && !selected.contains(target)) selected.add(target);
}
}
private static void requireIntegerStorage(VariableStorage[] storage, String owner) {
for (VariableStorage item : storage) {
if (item.isBadStorage() || item.isUnassignedStorage())
throw new IllegalStateException("REA soft-float storage unresolved for " + owner);
List<Register> registers = item.getRegisters();
if (registers == null) continue; // stack-only storage
for (Register register : registers) {
if (FPU_REGISTER.matcher(register.getName()).matches())
throw new IllegalStateException(
"REA soft-float storage still uses " + register.getName() + " for " + owner);
}
}
}
private static boolean usesFloat(Function function) {
if (isFloat(function.getReturn().getFormalDataType())) return true;
for (Parameter parameter : function.getParameters()) {
if (isFloat(parameter.getFormalDataType())) return true;
}
return false;
}
private static boolean usesFloat(FunctionDefinition definition) {
if (isFloat(definition.getReturnType())) return true;
for (ParameterDefinition argument : definition.getArguments()) {
if (isFloat(argument.getDataType())) return true;
}
return false;
}
private static boolean isFloat(DataType type) {
DataType current = type;
while (current instanceof TypeDef typeDef) current = typeDef.getBaseDataType();
return current instanceof AbstractFloatDataType;
}
}
+1 -7
View File
@@ -108,13 +108,7 @@ existing analysis families directly, for example
`ro.kernel.qemu` value, `null` when absent.
- **Device paths are not host paths.** `base_apk_device_path` values live on
the device filesystem. Only pulls and captures produce host-local paths,
each paired with byte size and SHA-256 digest. A pulled file's host name is
the device path's last segment, accepted only when every host can store it:
a `pull_adb_file` whose device path ends in a separator, `.`/`..`, a
control or reserved character, a Windows reserved device name, a trailing
dot or space, or a `:` alternate-data-stream marker is refused as
`invalid_request` before any `adb pull` runs, and REA never renames a
device-reported file.
each paired with byte size and SHA-256 digest.
- **Splits are part of the answer.** `pm path` enumerates the base APK and
every installed split; pulls cover the whole set. System packages whose
APKs keep other names report the `unknown` role rather than guessing.
+15 -99
View File
@@ -13,13 +13,13 @@ retains class, byte order, `e_type`, raw `e_flags`, and inspected ABI declaratio
Recognition does not mean every provider or MIPS variant is supported. Existing
PE, managed and Mach-O architecture sets are unchanged.
The Ghidra lane admits ELF32 `ET_EXEC` and `ET_REL`, little or big endian, with
explicit o32 and `EF_MIPS_ARCH_32R2` flags and no machine-specific or ASE
encoding flags. These are header declarations, not proof that every instruction
conforms to that ISA. ELF64, n32/n64, unspecified ABI, MIPS16/microMIPS,
machine-specific variants, shared objects and PSP PRX are not covered by this
lane. Their family may be identified, but provider selection returns the
specific unsupported constraint rather than guessing a compatible profile.
The first Ghidra lane admits ELF32 `ET_EXEC`, little or big endian, with explicit
o32 and `EF_MIPS_ARCH_32R2` flags and no machine-specific or ASE encoding flags.
These are header declarations, not proof that every instruction conforms to
that ISA. ELF64, n32/n64, unspecified ABI, MIPS16/microMIPS, machine-specific
variants, shared/relocatable objects and PSP PRX are not covered by this lane.
Their family may be identified, but provider selection returns the specific
unsupported constraint rather than guessing a compatible profile.
Explicit `EF_MIPS_FP64` and `EF_MIPS_NAN2008` declarations are refused. Their
absence is insufficient: Clang can emit identical `e_flags` for FP32, FP64 and
@@ -39,14 +39,12 @@ absent or supply a default ABI.
The field layout follows
[`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h).
Provider admission requires record version 0, MIPS32r2, 32-bit GPRs, no CPR2,
no ISA extension, ASE or unknown general flags, and one of two floating-point
declarations: 32-bit CPR1 with a double-precision or FPXX ABI, or no CPR1 with
the soft-float ABI. Other combinations, including single-float, FP64 and
unspecified ABIs, are refused. Missing records remain unknown rather than
defaulting to a compatible ABI, and this lane refuses them. Unknown record
values are retained for diagnosis but do not gain provider support. These checks
bind the declared interpretation; the fixtures do not establish full
Provider admission requires record version 0, MIPS32r2, 32-bit GPR/CPR1, no CPR2,
and a double-precision or FPXX ABI declaration with no ISA extension, ASE or
unknown general flags. Missing records remain unknown rather than defaulting
to a compatible ABI, and this initial lane refuses them. Unknown record values
are retained for diagnosis but do not gain provider support. These checks bind
the declared interpretation; the integer-only fixture does not establish full
floating-point behavioral conformance.
MIPS header and ABI record facts are committed in the analysis profile so
@@ -55,51 +53,6 @@ identity is versioned independently of file identity. Evidence, lifecycle
outputs and saved snapshots preserve the `mips` family. Hopper and IDA adapters
do not implicitly gain MIPS support. Windows Ghidra P0 remains PE x86/x86-64 only.
### Soft-float prototype
Ghidra's MIPS32 compiler specifications model hard-float o32 only: a typed
`float` or `double` parameter is bound to `$f12`/`$f14` and returned in `$f0`.
Soft-float o32 passes those values in integer registers and stack slots, so a
typed prototype (for example from DWARF) would make the decompiler read FPU
registers the code never writes and drop the real dataflow. That applies to
functions, external functions and indirect calls through typed function
pointers alike.
For soft-float targets REA commits
`mips_soft_float_storage: o32-soft-float-prototype-v1` in the analysis profile
and runs `ReaGhidraMipsSoftFloat.java` after auto-analysis. The script encodes
the program's own default o32 prototype, removes its floating-point parameter
and return entries and the rules that consume floating-point storage, and
installs the result as the program-specific `__rea_soft_float` prototype through
a Ghidra specification extension. Deriving it keeps the byte-order-specific
aggregate rules and the hidden struct-return pointer. The script then assigns
that prototype, with dynamic storage, to every non-thunk function and external
function, and to every function-pointer definition, whose declared types use
`float` or `double`. Thunks follow their targets. Types are kept.
The script fails, and writes no report, if the derived prototype still places a
float in an FPU register or any reassigned function does. It records the
function and definition counts last; REA requires that report exactly when the
profile commits the policy and refuses the session otherwise, before serving any
operation. Session Evidence carries the policy and counts as limitations.
Callables without typed float signatures keep Ghidra's default prototype, under
which Ghidra already infers integer-register parameters. Hard-float profiles do
not gain the policy, so their digests are unchanged.
### Relocatable objects
`ET_REL` objects, such as Linux kernel modules (`.ko`), are imported by Ghidra's
ELF loader, which assigns each allocated section an analysis address and applies
the object's REL relocations there. Undefined symbols are placed in Ghidra's
`EXTERNAL` block: a thunk for code, a label for data. These addresses are
analysis coordinates, not the addresses a kernel or linker would choose, and
session Evidence says so. `inspect_native_load_image` reports each relocation
with Ghidra's status; HI16 entries report `PARTIAL` because Ghidra applies them
together with their paired LO16. Relocatable targets commit
`mips_support_lane: elf32-rel-o32-arch32r2-standard-v1`, separate from the
executable lane, whose identity executable targets keep. Kernel modules are
usually soft-float and therefore also use the soft-float storage policy.
## Use
With a caller-installed Ghidra and compatible JDK on a supported Linux/macOS host:
@@ -117,10 +70,7 @@ REA's ELF parser.
## Verification
### Executables
The source-owned freestanding fixture is `tests/conformance/c/mips.c`; the
soft-float variant also links `tests/conformance/c/mips-soft-float.c`. The
The source-owned freestanding fixture is `tests/conformance/c/mips.c`. The
optional cross-target lane requires a caller-supplied Clang with MIPS targets,
LLD, GNU `readelf`, Ghidra and its compatible JDK; ordinary host-native checks
do not acquire these tools. `REA_MIPS_CLANG` selects Clang and
@@ -131,8 +81,7 @@ These are test-time prerequisites, not REA runtime providers.
GHIDRA_INSTALL_DIR=/absolute/path/to/ghidra npm run verify:ghidra:mips
```
The lane compiles both byte orders with hard-float and soft-float ABIs and never
executes the targets. GNU readelf
The lane compiles both byte orders and never executes the targets. GNU readelf
independently supplies header/ABI facts and symbol addresses, compared with the
production REA target resolver before provider startup. Its version, bounded
command output and selected tool path are retained in the verification report.
@@ -155,42 +104,9 @@ all program-header payloads. It checks the actual Ghidra language, entry-address
decompilation, distinct artifact digest and missing-section limitations through
CLI Evidence and an MCP-produced snapshot. Caller-selected Ghidra heap limits
are passed to these child workflows.
For soft-float fixtures the lane also checks, through `inspect_native_api`, the
exact parameter and return types and storage of float, double, mixed (pair
alignment and stack spill) and struct-returning functions; that a caller's
decompilation receives every source float constant without FPU inputs,
including an indirect call through a typed function pointer; and that the
snapshot profile and Evidence retain the policy and outcome counts.
Pseudocode presence is a liveness check, not proof of semantic equivalence.
Instruction decoding and reader agreement likewise do not establish runtime,
delay-slot execution or complete floating-point semantics. Assertion regressions
reject incorrect byte order, constants, destinations and loaded identities.
Parser/profile tests are narrower checks, not substitutes for this real-provider
run. Report the actual lanes run and leave unexecuted verification pending.
### Relocatable objects
This lane has its own command and fixture:
```bash
GHIDRA_INSTALL_DIR=/absolute/path/to/ghidra npm run verify:ghidra:mips-relocatable
```
It compiles `tests/conformance/c/mips-relocatable.c` with `-c -G0` in both byte
orders, plus a little-endian soft-float `-mlong-calls` variant matching Linux
module options. GNU readelf independently supplies sections, symbols and REL
entries. For every relocation in an allocated section, the lane reads the
loaded word through `read_bytes` and requires it to encode `S + A`: `S` from
Ghidra's own section placement (`list_segments`) or `EXTERNAL` placement, `A`
from the original file bytes. HI16/LO16 are checked as the effective address
each LO16 forms with each HI16 it resolves, including several HI16s before one
LO16, so a missing signed-low carry fails and a HI16 without a following LO16
fails. The lane also requires matching Ghidra relocation entries and statuses,
section file offsets and sizes, coverage of a shared HI16, a LO16 carry,
external code and data and `R_MIPS_32` words, direct calls from the entry
function, the relocatable support lane in the snapshot profile, the
relocatable limitation in Evidence and an unchanged source object. The fixture
also calls `powf`, an external whose float prototype comes from Ghidra's C
library archive; every variant must decompile `powf(1.5,2.5)` without FPU
inputs, and the soft-float variant must report that the soft-float prototype
covered it.
+1 -1
View File
@@ -98,7 +98,7 @@ For guided registration, set `REA_IDA_MCP_CONFIG` before `rea setup --client <cl
## Coverage and evidence
Both profiles support procedure/string inventory and literal or regex search, procedure resolution, pseudocode, assembly, raw function instructions, resolved callees, incoming address references, and a function dossier. Inventories and modern pseudocode/assembly follow upstream pagination. Upstream limits on modern xrefs and callees are retained as explicit truncation limitations. In the modern headless profile, targets that are not proven x86-only use processor-neutral `callgraph` edges instead of the x86 instruction checks in upstream `callees` (mrexodia/ida-pro-mcp#549). REA requests depth zero: the pinned upstream emits the root's outgoing edges without scanning child functions, then REA batches function-name lookup for those addresses. The graph covers recognized function destinations only; imports and other non-function targets remain unknown. A server without `callgraph` returns an empty relationship list with an explicit coverage limitation. Graph and path workflows retain those limitations alongside observed edges. A failed or inconsistent graph reply remains an error. A transport-level truncated preview is rejected rather than reported as complete evidence; the error preserves upstream retrieval guidance.
Both profiles support procedure/string inventory and literal or regex search, procedure resolution, pseudocode, assembly, raw function instructions, resolved callees, incoming address references, and a function dossier. Inventories and modern pseudocode/assembly follow upstream pagination. Upstream limits on modern xrefs and callees are retained as explicit truncation limitations. A transport-level truncated preview is rejected rather than reported as complete evidence; the error preserves upstream retrieval guidance.
The legacy profile additionally supports direct callers. Its producer may report call-site or interior addresses; REA resolves the reported function symbol to its canonical entry. The modern profile does not prove direct callers from its code/data xref labels, so that capability is unavailable and the dossier marks it unknown.
-21
View File
@@ -800,27 +800,6 @@ Each line is `address<TAB>kind[<TAB>name]`, where kind is `function`, `code` or
Label seeds can name mapped uninitialized storage such as `.bss`; function and code seeds require initialized bytes.
To skip a named Ghidra auto-analyzer (for example, when default auto-analysis
stalls in one analyzer on a Swift/Objective-C arm64 Mach-O):
```bash
export REA_GHIDRA_DISABLED_ANALYZERS="Objective-C Message Analyzer"
```
The value is a comma-separated list of exact Ghidra analyzer names. A pre-script
disables each name before auto-analysis, the analysis profile records the
override in `analyzer_preset` (for example
`ghidra-default+disabled:Objective-C Message Analyzer`), and a session fails to
open when its committed profile and the setting differ. A name starts with a
letter or digit and contains only letters, digits, spaces, periods, underscores,
and hyphens; at most 32 names are admitted, each at most 120 characters.
Names must identify analyzers registered for the selected program. Unknown names
or options that remain enabled abort the headless import before auto-analysis
and the bridge, so an unsuccessful override cannot produce a session claiming
that the analyzer was disabled. For Windows PE resource analysis, the exact name
is `WindowsResourceReference`. Disabling an analyzer skips the facts it would
have produced.
### Ghidra heap and CPU controls
Set resource controls in the environment that launches the CLI or MCP server,
+2 -54
View File
@@ -66,45 +66,6 @@ code-point order. Dependency manifests remain package observations but do not
declare Electron main or renderer entries. If only dependency manifests are
present, the inventory artifact remains the graph root.
## Select sources before analysis
For a question about known entry sources, supply exact artifact-relative file
paths in `source_paths`, or repeat `--source-path` on either CLI route:
```bash
rea analyze /absolute/path/to/app.asar \
--source-path package.json --source-path main.js --json
```
```json
{
"input_path": "/absolute/path/to/app.asar",
"source_paths": ["package.json", "main.js"],
"detail": "summary"
}
```
REA inventories and verifies the complete artifact, including unselected bytes,
but allocates source text and performs static analysis only for selected files.
Include supporting manifests, HTML or source maps explicitly when needed;
imports do not expand the selection. Paths must match relevant inventoried
files exactly, including nested ASAR prefixes. Unknown paths, aliases and globs
return an `invalid_request` correction. Selected files are analyzed in request
order before unselected scopes are projected. Repeated paths preserve the
caller's request and first-occurrence priority, analyzing the file once.
Omit the selection for the existing complete
application workflow.
The result and summary retain `source_selection.requested_paths` and
`source_selection.unselected_files`. Every relevant file keeps its original
inventory identity; unselected files report `text_status:
"outside-analysis-scope"`. Their relationships remain unknown, and application
coverage is partial while any file remains outside scope. Selection does not
claim whole-application completeness or change integrity policy. Summary detail
only reduces delivery; source selection reduces text acquisition and analysis.
Explicit native provider or snapshot routing rejects `--source-path` rather
than ignoring it.
## ASAR integrity
ASAR inventory checks Electron integrity metadata for embedded archive entries
@@ -508,12 +469,6 @@ from unmatched nested targets rather than reporting array indices.
For imports and requires, the active Node conditions include `module-sync`;
earlier active conditions retain precedence over later ones.
Package self-references use the importer's enclosing package scope before
installed dependencies. ESM requires the parsed package name to equal the
metadata name; a scope-only name such as `@scope` cannot capture `@scope/pkg`.
CommonJS retains Node's raw-request name/prefix matching. Both paths preserve
the original specifier and select exports under their own loader conditions.
Exact package-private `#alias` specifiers resolve against the importer's
enclosing `package.json` `imports` map — the nearest same-container package
scope, stopping at a `node_modules` directory so a nested package cannot inherit
@@ -529,15 +484,8 @@ URL targets before selecting a later package-local file; an unsupported bare
package or recursive alias keeps its unresolved outcome rather than selecting
a later fallback.
ESM relative module paths, legacy package subpaths without exports, and selected
package exports targets use URL suffix handling and one percent-decoding pass.
Bare specifiers retain their complete query/fragment text for exports key matching.
For dependency lookup, ESM constructs a package metadata URL instead of treating
the parsed package name as a literal directory. A root request such as `legacy?x`
does not load either `node_modules/legacy?x` or `node_modules/legacy`; an exact
self-reference with that metadata name can still select its exports first.
Without an exports map, ESM subpaths select exact files; directory imports and
extension fallback are refused. CommonJS relative paths and legacy package main
ESM relative module paths and selected package exports targets use URL suffix handling
and one percent-decoding pass. CommonJS relative paths and legacy package main
fields retain literal filename punctuation. Exports targets must start with
`./` and exclude `.`, `..`, and `node_modules` path segments, including their
encoded forms, before URL normalization. Arrays skip invalid exports targets
+2 -19
View File
@@ -310,13 +310,6 @@ directly to `compare_functions`, and `inspect_artifact` Evidence to
`compare_artifacts`. Use `get_evidence_bundle` when the task needs broader
retained session history or an explicit bundle for transfer.
`get_call_graph` keeps provider-limited relationship reads on their node as
`limitations`, alongside any observed `calls`. An empty call list with such a
limitation is incomplete coverage, not proof that the function has no calls.
`trace_call_path` reports each limited expansion in `residual_unknowns` and
sets `truncated: true`; any path already supported by observed edges remains
visible. Tool errors remain in their existing error or failure fields.
REA prepares complete MCP results within the pinned stdio client's 10 MiB
receive-buffer budget, including both text and structured representations and
room for the JSON-RPC envelope. Tool errors carry only their text
@@ -328,18 +321,8 @@ session. Its exact reference is reported in
`details.reported_limits.evidence_reference`; call `inspect_analysis_view` with
that reference for a summary, one section/module, or a stable page, or call
`export_evidence_bundle` with a destination path. Complete
bundle exports stream canonical JSON into an atomically published file.
For oversized native function Evidence, `details.reported_limits.continuation`
contains a ready-to-call `inspect_analysis_view` request for the retained
pseudocode's first page. Use its `name` and `arguments` directly, then follow
`normalized_result.coverage.next_offset` for subsequent pages. The suggested
limit is the view contract's default, not a guarantee that arbitrary source or
metadata fits; reduce the limit or export when needed. Continuations are
offered for completed `analyze_function` results only after session recording is
acknowledged; the complete result and its retention reference remain intact.
A broad follow-up or `get_evidence_bundle` can also exceed the response budget; exporting
bundle exports stream canonical JSON into an atomically published file. A broad
follow-up or `get_evidence_bundle` can also exceed the response budget; exporting
preserves the complete session without sending it through a single MCP frame.
To discover existing records without their large payloads, call
`get_evidence_bundle` with `detail: "summary"`. Its `result.records` contain
+7 -11
View File
@@ -35,18 +35,14 @@ Omitted `format` keeps header-based detection; architecture and entry without
`format=raw-binary` are rejected.
Ghidra imports a private snapshot with `BinaryLoader` and these little-endian
generic languages and compiler specifications:
generic languages:
| architecture | language_id | compiler_spec_id |
| ------------ | ------------------- | ---------------- |
| x86 | `x86:LE:32:default` | `gcc` |
| x86_64 | `x86:LE:64:default` | `gcc` |
| arm | `ARM:LE:32:v7` | `default` |
| arm64 | `AARCH64:LE:64:v8A` | `default` |
Compiler specifications select analysis conventions; they do not identify the
original compiler or ABI of a headerless target. Ghidra's x86 languages do not
provide a compiler specification named `default`.
| architecture | language_id |
| ------------ | ------------------- |
| x86 | `x86:LE:32:default` |
| x86_64 | `x86:LE:64:default` |
| arm | `ARM:LE:32:v7` |
| arm64 | `AARCH64:LE:64:v8A` |
Those settings, the caller-supplied entry, and `language_selection:
caller-supplied-v1` are committed in the analysis profile. A different entry
+4 -7
View File
@@ -463,7 +463,7 @@ prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
| Ghidra lane | Supported runner/target | Additional local tools |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:seeds` | POSIX; raw x86 fixture; seed ordering, BSS labels, analyzer readback and unknown-name rejection | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:seeds` | POSIX host; source-owned 32-bit x86 raw fixture; function order and BSS labels | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
@@ -803,10 +803,8 @@ through the compiled CLI and stdio MCP server. With the same Ghidra/JDK
prerequisites, it checks preparation and bridge startup, verified load mappings,
instruction and byte readback, canonical-entry session reuse, replacement when
the entry changes without `provider_id`, unchanged source bytes, and owned
cleanup. Six-byte x86 and x86_64 controls also verify supported compiler
specifications, independent mappings, unchanged bytes, and decompilation of a
known return value. AArch64 remains covered by deterministic tests rather than
this real-provider lane. See [raw binary analysis](raw-binary.md).
cleanup. Other raw CPU families remain covered by deterministic tests rather
than this real-provider lane. See [raw binary analysis](raw-binary.md).
## Apple Interface Builder archives
@@ -1386,8 +1384,7 @@ generation. It installs nothing. Without configuration, the default invocation
reports a named skip; `--require-tools` makes missing configuration fail.
The verifier creates and validates real modules with custom sections, multiple
import/export kinds, escaped names, an empty module, and a bulk-memory
DataCount section. It compares exact WAT,
import/export kinds, escaped names and empty modules. It compares exact WAT,
section output and selected/tool digests through the compiled CLI and real MCP
SDK, validates advertised schemas, checks identical CLI/MCP Evidence IDs and
session bundle readback, retains invalid/truncated module diagnostics, preserves
-3
View File
@@ -55,9 +55,7 @@
"bridge/ghidra/ReaGhidraExtensions.java",
"bridge/ghidra/ReaGhidraPrepareCom.java",
"bridge/ghidra/ReaGhidraApplySeeds.java",
"bridge/ghidra/ReaGhidraAnalysisOptions.java",
"bridge/ghidra/ReaGhidraNoReturnFix.java",
"bridge/ghidra/ReaGhidraMipsSoftFloat.java",
"bridge/ghidra/ReaGhidraPrepareRaw.java",
"bridge/android/ReaJadxBridge.java",
"bridge/mitmproxy/capture.py",
@@ -165,7 +163,6 @@
"verify:ghidra": "npm run build:cached && node scripts/verify-real-ghidra.mjs",
"verify:ghidra:seeds": "npm run build:cached && node scripts/verify/ghidra/seeds.mjs",
"verify:ghidra:mips": "npm run build:cached && node scripts/verify-real-ghidra-mips.mjs",
"verify:ghidra:mips-relocatable": "npm run build:cached && node scripts/verify-real-ghidra-mips-relocatable.mjs",
"verify:ghidra:swift": "npm run build:cached && node scripts/verify-real-ghidra-swift.mjs",
"verify:ida": "npm run build:cached && node scripts/verify-real-ida.mjs",
"verify:ghidra:switch": "npm run build:cached && node scripts/verify-real-ghidra-switch.mjs",
+2 -4
View File
@@ -172,13 +172,11 @@ const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
const runtimeParent = join(workspace, "runtime with spaces");
await mkdir(runtimeParent);
const callerDirectory = join(workspace, "caller cwd with spaces");
// Keep sorted for comparison with the final caller-directory listing.
const scriptCollisions = [
"ReaGhidraAnalysisOptions.java",
"ReaGhidraApplySeeds.java",
"ReaGhidraBridge.java",
"ReaGhidraNoReturnFix.java",
"ReaGhidraPrepareCom.java",
"ReaGhidraApplySeeds.java",
"ReaGhidraNoReturnFix.java",
"ReaGhidraPrepareRaw.java",
];
await mkdir(callerDirectory);
@@ -1,333 +0,0 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdtemp, mkdir, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseBinaryTarget } from "../dist/application/BinaryTargetResolver.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
import { functionDossierSchema } from "../dist/domain/hopperValues.js";
import { inspectGhidraInstallation } from "../dist/ghidra/GhidraInstallation.js";
import { buildMipsRelocatableFixture } from "../tests/conformance/ghidra/mips-fixture.mjs";
import {
assertMipsRelocations,
inspectMipsRelocatableReadelf,
} from "../tests/conformance/ghidra/mips-relocatable-checks.mjs";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
// Optional cross-target lane for ET_REL objects. No tool acquisition, no
// linking and no target execution.
if (process.argv.length > 2)
throw new Error(
"Usage: node scripts/verify-real-ghidra-mips-relocatable.mjs",
);
if (!["linux", "darwin"].includes(process.platform))
throw new Error(
"MIPS verification requires a supported Linux/macOS Ghidra host",
);
if (process.env.GHIDRA_INSTALL_DIR === undefined)
throw new Error("Missing prerequisite: caller-supplied GHIDRA_INSTALL_DIR");
const installation = inspectGhidraInstallation({
environment: process.env,
installDir: process.env.GHIDRA_INSTALL_DIR,
...(process.env.JAVA_HOME === undefined
? {}
: { javaHome: process.env.JAVA_HOME }),
});
if (installation.status !== "available")
throw new Error(
`Ghidra prerequisite unavailable: ${JSON.stringify(installation)}`,
);
const execute = promisify(execFile);
const entrypoint = fileURLToPath(new URL("./rea.mjs", import.meta.url));
const run = createVerifierRun();
const workspace = await mkdtemp(join(tmpdir(), "rea-mips-rel-proof-"));
const runtime = join(workspace, "runtime");
await mkdir(runtime);
const env = {
PATH: process.env.PATH ?? "/usr/bin:/bin",
TMPDIR: runtime,
REA_LOG_LEVEL: "silent",
REA_ANALYSIS_PROVIDER: "ghidra",
REA_PROCESS_RUN_ID: run.run_id,
GHIDRA_INSTALL_DIR: installation.installDir,
JAVA_HOME: installation.javaHome,
HOPPER_LAUNCHER_PATH: "/rea-unconfigured-deep-provider/hopper",
};
for (const setting of ["GHIDRA_HEADLESS_MAXMEM", "GHIDRA_MAXMEM"]) {
if (process.env[setting] !== undefined) env[setting] = process.env[setting];
}
/** Load every relocated word through the public read_bytes operation. */
async function loadedWords(call, independent, segments, byteOrder) {
const words = new Map();
for (const relocation of independent.relocations) {
const section = [...independent.sections.values()].find(
(item) => item.name === relocation.section,
);
if (!section?.allocated) continue;
const segment = segments.find((item) => item.name === section.name);
assert.ok(segment, `Ghidra segment ${section.name}`);
const address = BigInt(segment.start) + BigInt(relocation.offset);
const read = await call("read_bytes", {
address: `0x${address.toString(16)}`,
length: 4,
});
assert.equal(read.returned_bytes, 4);
const bytes = Buffer.from(read.bytes_hex, "hex");
words.set(
address,
byteOrder === "little" ? bytes.readUInt32LE(0) : bytes.readUInt32BE(0),
);
}
return words;
}
async function verifyFixture(byteOrder, kernelModule) {
const fixture = await buildMipsRelocatableFixture(
workspace,
byteOrder,
kernelModule,
);
const independent = await inspectMipsRelocatableReadelf(
fixture.path,
byteOrder,
fixture.float_abi,
);
const resolved = await parseBinaryTarget(fixture.path);
if (!resolved.ok) throw resolved.error;
assert.deepEqual(
resolved.value.mips,
independent.mips,
"REA ELF/ABI interpretation disagrees with GNU readelf",
);
assert.equal(resolved.value.sha256, fixture.sha256);
const label = `${byteOrder}${kernelModule ? "-module" : ""}`;
const snapshotPath = join(workspace, `${label}.snapshot.json`);
const transport = new StdioClientTransport({
command: process.execPath,
args: [entrypoint, "mcp"],
env,
stderr: "pipe",
});
const client = new Client({ name: "rea-real-mips-rel-proof", version: "1" });
let stderr = "";
transport.stderr?.on("data", (chunk) => {
stderr = (stderr + chunk.toString()).slice(-65536);
});
const call = async (name, args = {}) =>
requireMcpOperationResult(
await client.callTool({ name, arguments: args }, undefined, {
timeout: 180000,
}),
name,
);
try {
await client.connect(transport);
const target = await call("open_binary", {
path: fixture.path,
provider_id: "ghidra",
});
assert.equal(target.architecture, "mips");
assert.equal(target.sha256, fixture.sha256);
const image = (await call("inspect_native_load_image")).observations;
assert.equal(
image.language_id,
`MIPS:${byteOrder === "little" ? "LE" : "BE"}:32:default`,
);
assert.equal(image.source_files.length, 1);
assert.equal(image.source_files[0].original_sha256, fixture.sha256);
assert.notEqual(
image.source_files[0].modified_sha256,
fixture.sha256,
"Relocations must change the loaded bytes",
);
const names = await call("list_names");
const segments = await call("list_segments");
const coverage = assertMipsRelocations({
facts: independent,
fileBytes: await readFile(fixture.path),
segments,
mappings: image.mappings,
ghidraRelocations: image.relocations,
words: await loadedWords(call, independent, segments, byteOrder),
names,
byteOrder,
});
assert.equal(coverage.unplaced, 0, "Fixture references an empty section");
assert.ok(coverage.shared_hi16, "No HI16 shared by several LO16s");
assert.ok(coverage.lo16_carry, "No LO16 >= 0x8000 forced a HI16 carry");
assert.ok(coverage.external_code, "No external code relocation");
assert.ok(coverage.external_data, "No external data relocation");
assert.ok(coverage.types.R_MIPS_32 > 0, "No R_MIPS_32 relocation");
assert.equal(
coverage.types.R_MIPS_26 === undefined,
kernelModule,
"Long calls must replace every R_MIPS_26 with HI16/LO16",
);
const procedures = await call("list_procedures");
const named = (name) => {
const found = procedures.filter((item) => item.value === name);
assert.equal(found.length, 1, `Procedure ${name}`);
return found[0].address;
};
const entry = named("rea_mips_rel_entry");
const callees = await call("procedure_callees", { procedure: entry });
for (const callee of [
named("rea_mips_rel_leaf"),
named("rea_mips_rel_far"),
])
assert.ok(callees.includes(callee), `Missing direct call to ${callee}`);
// Ghidra lists the external function and its EXTERNAL-block thunk.
assert.ok(
procedures.some(
(item) =>
item.value === "<EXTERNAL>::rea_mips_rel_external" &&
callees.includes(item.address),
),
"Missing call to the external function",
);
const dossier = functionDossierSchema.parse(
await call("analyze_function", { procedure: entry }),
);
for (const name of [
"rea_mips_rel_leaf",
"rea_mips_rel_far",
"rea_mips_rel_external",
])
assert.ok(dossier.pseudocode.includes(name), `Pseudocode lost ${name}`);
// powf's float prototype comes from Ghidra's C library archive; soft-float
// builds pass 1.5f/2.5f in $a0/$a1, hard-float builds in $f12/$f14.
const scaled = functionDossierSchema.parse(
await call("analyze_function", {
procedure: named("rea_mips_rel_scaled"),
}),
);
assert.match(scaled.pseudocode, /\bpowf\(1\.5,2\.5\)/u);
assert.doesNotMatch(scaled.pseudocode, /\bin_f\d+\b/u);
await call("close_binary", { snapshot_path: snapshotPath });
const snapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(snapshotPath, "utf8")),
);
const parameters = snapshot.binding.analysis_profile.parameters;
assert.equal(parameters.mips_elf.type, 1);
assert.equal(
parameters.mips_support_lane,
"elf32-rel-o32-arch32r2-standard-v1",
);
assert.equal(
parameters.mips_soft_float_storage,
kernelModule ? "o32-soft-float-prototype-v1" : undefined,
);
assert.ok(
snapshot.evidence_bundle.records.some((record) =>
parseEvidence(record).limitations.some((item) =>
item.includes("Relocatable MIPS ELF"),
),
),
"Relocatable limitation missing from retained Evidence",
);
assert.equal(
snapshot.evidence_bundle.records.some((record) =>
parseEvidence(record).limitations.some((item) =>
/Soft-float prototype outcome: [1-9]\d* functions/u.test(item),
),
),
kernelModule,
"Soft-float prototype must cover the archive-typed external in module builds",
);
await call("open_binary", {
path: fixture.path,
provider_id: "ghidra",
snapshot_path: snapshotPath,
});
assert.deepEqual(await call("list_procedures"), procedures);
await call("close_binary");
// Same public operation through CLI, not a private injected provider.
const cli = await execute(
process.execPath,
[
entrypoint,
"function",
fixture.path,
"rea_mips_rel_entry",
"--provider",
"ghidra",
"--json",
],
{ env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 },
);
const evidence = parseEvidence(JSON.parse(cli.stdout));
assert.equal(evidence.subject?.architecture, "mips");
assert.equal(evidence.subject?.digest.sha256, fixture.sha256);
assert.equal(
functionDossierSchema.parse(evidence.normalized_result).procedure.address,
dossier.procedure.address,
);
assert.equal(
createHash("sha256")
.update(await readFile(fixture.path))
.digest("hex"),
fixture.sha256,
"Analysis modified the source object",
);
return {
...fixture,
independent_reader: {
command: independent.command,
version: independent.version,
},
effective_language: image.language_id,
relocation_coverage: coverage,
procedures: procedures.length,
status: "passed",
};
} catch (cause) {
throw new Error(
`MIPS ${label} relocatable verification failed; MCP stderr: ${stderr}`,
{ cause },
);
} finally {
try {
await client.close();
} finally {
await transport.close();
}
}
}
const reports = [];
let primaryFailure;
try {
for (const [byteOrder, kernelModule] of [
["little", false],
["big", false],
["little", true],
])
reports.push(await verifyFixture(byteOrder, kernelModule));
} catch (cause) {
primaryFailure = cause;
}
const completed = await completeVerifierRun(run);
if (
completed.process_lineage.status !== "verified" ||
completed.process_lineage.descendants.length !== 0
)
throw new Error(
`Cleanup unverified; retained diagnostic workspace: ${workspace}`,
{ cause: primaryFailure },
);
if (primaryFailure !== undefined)
throw new Error(
`MIPS relocatable verification failed; retained workspace: ${workspace}`,
{ cause: primaryFailure },
);
await rm(workspace, { recursive: true, force: true });
console.log(JSON.stringify({ ...completed, fixtures: reports }, null, 2));
+7 -83
View File
@@ -15,7 +15,6 @@ import {
assertMipsLoadedImage,
assertMipsProbe,
assertMipsGlobal,
assertMipsSoftFloatStorage,
} from "../tests/conformance/ghidra/mips-checks.mjs";
import { parseEvidence } from "../dist/domain/evidence.js";
import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
@@ -78,64 +77,16 @@ const env = {
for (const setting of ["GHIDRA_HEADLESS_MAXMEM", "GHIDRA_MAXMEM"]) {
if (process.env[setting] !== undefined) env[setting] = process.env[setting];
}
/**
* Typed float signatures must use soft-float o32 storage, and callers must see
* the source constants through it rather than FPU inputs.
*/
async function verifySoftFloat(call, symbols, byteOrder) {
const boundaries = {};
for (const name of [
"rea_mips_soft_pick",
"rea_mips_soft_wide",
"rea_mips_soft_mixed",
"rea_mips_soft_struct",
]) {
const api = await call("inspect_native_api", { procedure: symbols[name] });
boundaries[name] = api.boundary;
}
assertMipsSoftFloatStorage(boundaries, byteOrder);
const caller = functionDossierSchema.parse(
await call("analyze_function", {
procedure: symbols.rea_mips_soft_caller,
}),
);
// Includes the struct call and the indirect call through a typed pointer.
assert.doesNotMatch(caller.pseudocode, /\bin_f\d+\b/u);
for (const literal of [
"1.5",
"2.5",
"3.25",
"4.75",
"5.5",
"6.125",
"7.5",
"8.5",
"9.5",
])
assert.ok(
caller.pseudocode.includes(literal),
`Soft-float caller lost argument ${literal}`,
);
return { storage: boundaries };
}
const reports = [];
let primaryFailure;
try {
for (const [byteOrder, floatAbi] of psp
? [["little", "hard"]]
: [
["little", "hard"],
["big", "hard"],
["little", "soft"],
["big", "soft"],
]) {
for (const byteOrder of psp ? ["little"] : ["little", "big"]) {
const fixture = psp
? await buildPspFixture(workspace)
: await buildMipsFixture(workspace, byteOrder, floatAbi);
: await buildMipsFixture(workspace, byteOrder);
const independent = psp
? fixture
: await inspectMipsReadelf(fixture.path, byteOrder, floatAbi);
: await inspectMipsReadelf(fixture.path, byteOrder);
const resolved = await parseBinaryTarget(fixture.path);
if (!resolved.ok) throw resolved.error;
if (psp) {
@@ -153,10 +104,7 @@ try {
);
}
assert.equal(resolved.value.sha256, fixture.sha256);
const snapshotPath = join(
workspace,
`${byteOrder}-${floatAbi}.snapshot.json`,
);
const snapshotPath = join(workspace, `${byteOrder}.snapshot.json`);
const transport = new StdioClientTransport({
command: process.execPath,
args: [entrypoint, "mcp"],
@@ -290,10 +238,6 @@ try {
callees.includes(leaf.address),
"Expected direct call not recovered",
);
const softFloat =
floatAbi === "soft"
? await verifySoftFloat(call, independent.symbols, byteOrder)
: undefined;
const assembly = await call("procedure_assembly", {
procedure: entry.address,
});
@@ -349,19 +293,6 @@ try {
},
"Snapshot profile did not retain the independently inspected ELF/ABI declaration",
);
assert.equal(
snapshot.binding.analysis_profile.parameters.mips_soft_float_storage,
floatAbi === "soft" ? "o32-soft-float-prototype-v1" : undefined,
);
assert.equal(
snapshot.evidence_bundle.records.some((record) =>
/Soft-float prototype outcome: 4 functions and [1-9]\d* function-pointer definitions use __rea_soft_float\./u.test(
JSON.stringify(record),
),
),
floatAbi === "soft",
"Soft-float prototype outcome missing from, or unexpected in, retained Evidence",
);
assert.ok(snapshot.evidence_bundle.records.length > 0);
if (psp) {
assert.equal(
@@ -488,10 +419,7 @@ try {
assert.ok(read32(offset + 4) + read32(offset + 16) <= cut);
}
const carvedBytes = sourceBytes.subarray(0, cut);
const carvedPath = join(
workspace,
`${byteOrder}-${floatAbi}-carved.elf`,
);
const carvedPath = join(workspace, `${byteOrder}-carved.elf`);
const carvedSha256 = createHash("sha256")
.update(carvedBytes)
.digest("hex");
@@ -532,7 +460,7 @@ try {
assert.ok(carvedDossier.pseudocode.trim().length > 0);
const carvedSnapshotPath = join(
workspace,
`${byteOrder}-${floatAbi}-carved.snapshot.json`,
`${byteOrder}-carved.snapshot.json`,
);
await call("close_binary", { snapshot_path: carvedSnapshotPath });
const carvedSnapshot = parseAnalysisSnapshot(
@@ -595,11 +523,7 @@ try {
: "conditional branch destination",
"global word and marker bytes",
"CLI/MCP and snapshot identity",
...(floatAbi === "soft"
? ["soft-float o32 parameter/return storage and caller constants"]
: []),
],
...(softFloat === undefined ? {} : { soft_float: softFloat }),
procedures: procedures.length,
carved: carved ?? {
status: "not_run",
@@ -610,7 +534,7 @@ try {
});
} catch (cause) {
throw new Error(
`MIPS ${byteOrder} ${floatAbi}-float verification failed; MCP stderr: ${stderr}`,
`MIPS ${byteOrder} verification failed; MCP stderr: ${stderr}`,
{ cause },
);
} finally {
+2 -54
View File
@@ -53,15 +53,6 @@ const env = {
...(process.env.JAVA_HOME === undefined
? {}
: { JAVA_HOME: process.env.JAVA_HOME }),
...(process.env.GHIDRA_HEADLESS_MAXMEM === undefined
? {}
: { GHIDRA_HEADLESS_MAXMEM: process.env.GHIDRA_HEADLESS_MAXMEM }),
...(process.env.REA_GHIDRA_DISABLED_ANALYZERS === undefined
? {}
: {
REA_GHIDRA_DISABLED_ANALYZERS:
process.env.REA_GHIDRA_DISABLED_ANALYZERS,
}),
};
const transport = new StdioClientTransport({
command: process.execPath,
@@ -169,48 +160,6 @@ try {
assert.equal(cli.normalized_result.base_address, "0x8000000");
assert.equal(cli.analysis_profile.digest, profiles[0]);
assert.deepEqual(await readFile(path), bytes);
const x86Targets = [];
for (const architecture of ["x86", "x86_64"]) {
const x86Bytes = Buffer.from("b82a000000c3", "hex"); // mov eax, 42; ret.
const x86Sha256 = createHash("sha256").update(x86Bytes).digest("hex");
const x86Path = join(workspace, `${architecture}.bin`);
await writeFile(x86Path, x86Bytes);
await call("open_binary", {
path: x86Path,
format: "raw-binary",
architecture,
entry_point: "0x1000",
});
const image = await call("inspect_native_load_image", {}, x86Sha256);
assert.equal(
image.normalized_result.status,
"verified",
JSON.stringify(image),
);
assert.equal(image.normalized_result.base_address, "0x1000");
assert.equal(image.normalized_result.observations.compiler_spec_id, "gcc");
assert.equal(image.analysis_profile.parameters.compiler_spec_id, "gcc");
const memory = await call(
"read_bytes",
{ address: "0x1000", length: x86Bytes.length },
x86Sha256,
);
assert.equal(memory.normalized_result.bytes_hex, x86Bytes.toString("hex"));
const pseudo = await call(
"procedure_pseudo_code",
{ procedure: "0x1000" },
x86Sha256,
);
assert.match(pseudo.normalized_result, /\b(?:42|0x2a)\b/iu);
await call("close_binary");
assert.deepEqual(await readFile(x86Path), x86Bytes);
x86Targets.push({
architecture,
sha256: x86Sha256,
compiler_spec_id: "gcc",
return_42: true,
});
}
report = {
ok: true,
mocked: false,
@@ -230,7 +179,6 @@ try {
invalid_entry_preserves_active_session: true,
load_images_verified: true,
original_unchanged: true,
x86_targets: x86Targets,
};
} catch (error) {
if (stderr.trim()) process.stderr.write(stderr);
@@ -251,7 +199,7 @@ process.stdout.write(
}) + "\n",
);
async function call(name, args = {}, targetSha256 = sha256) {
async function call(name, args = {}) {
const result = await client.callTool(
{ name, arguments: args },
{ timeout: 240000 },
@@ -265,7 +213,7 @@ async function call(name, args = {}, targetSha256 = sha256) {
)
return projected;
const evidence = parseEvidence(result.structuredContent);
assert.equal(evidence.subject.digest.sha256, targetSha256);
assert.equal(evidence.subject.digest.sha256, sha256);
assert.equal(evidence.subject.format, "raw-binary");
return evidence;
}
-25
View File
@@ -122,15 +122,6 @@ try {
new URL("../tests/fixtures/wasm/custom-names.wat", import.meta.url),
"utf8",
),
`(module
(memory 1)
(data "x")
(func (export "init")
i32.const 0
i32.const 0
i32.const 1
memory.init 0
data.drop 0))\n`,
];
for (const [index, source] of sources.entries()) {
const fixtureRoot = join(root.path, `fixture-${index}`);
@@ -229,22 +220,6 @@ try {
'"unicode-é\\\t"',
],
);
if (index === 4) {
assert.match(
headers,
/^DataCount start=0x[\da-f]+ end=0x[\da-f]+ \(size=0x[\da-f]+\) count: 1$/m,
);
assert.deepEqual(
report.sections.find(({ kind }) => kind === "DataCount"),
{
kind: "DataCount",
start: 0x23,
end: 0x24,
bytes: 1,
description: "count: 1",
},
);
}
const bundleReply = await client.callTool({
name: "get_evidence_bundle",
arguments: {},
+16 -62
View File
@@ -42,66 +42,11 @@ try {
"",
].join("\n"),
);
const output = await runHeadless(target, "seed-regression", [
"-postScript",
"ReaSeedRegressionProbe.java",
seeds,
]);
// Headless can exit zero after a script fails, so require the probe's marker.
assert.ok(output.includes("REA_SEED_REGRESSION_OK "), output);
const analyzer = "Decompiler Switch Analysis";
const disabled = await runHeadless(target, "analyzer-regression-valid", [
"-preScript",
"ReaGhidraAnalysisOptions.java",
analyzer,
"-postScript",
"ReaAnalyzerOptionsRegressionProbe.java",
analyzer,
]);
assert.ok(
disabled.includes(`REA_ANALYZER_OPTIONS_DISABLED_OK ${analyzer}`),
disabled,
);
const unknownAnalyzer = "REA Unknown Analyzer";
const rejected = await runHeadless(target, "analyzer-regression-unknown", [
"-preScript",
"ReaGhidraAnalysisOptions.java",
`${analyzer},${unknownAnalyzer}`,
"-postScript",
"ReaAnalyzerOptionsRegressionProbe.java",
analyzer,
]);
assert.ok(
rejected.includes(
`Unknown Ghidra analyzer for this program: ${unknownAnalyzer}`,
),
rejected,
);
assert.ok(
rejected.includes("Processing aborted as a result of pre-script."),
rejected,
);
assert.ok(!rejected.includes("REA_ANALYZER_OPTIONS_DISABLED_OK"), rejected);
} finally {
if (supervisor !== undefined) {
const stopped = await supervisor.stop();
assert.notEqual(stopped.status, "incomplete", JSON.stringify(stopped));
supervisor.dispose();
}
await rm(workspace, { recursive: true, force: true });
}
console.log(
"Real Ghidra pre-script regression passed: descending function order, out-of-range seed, BSS label, BSS code rejection, analyzer option readback, unknown-analyzer abort before post-scripts, and owned-process cleanup.",
);
async function runHeadless(target, project, scriptArguments) {
const launch = await spawnOwnedProviderProcess({
command: join(installation, "support", "analyzeHeadless"),
arguments: [
workspace,
project,
"seed-regression",
"-import",
target,
"-loader",
@@ -118,7 +63,9 @@ async function runHeadless(target, project, scriptArguments) {
new URL("../../../tests/conformance/ghidra", import.meta.url),
),
].join(";"),
...scriptArguments,
"-postScript",
"ReaSeedRegressionProbe.java",
seeds,
],
runId: run.run_id,
expectedCommand: null,
@@ -135,9 +82,16 @@ async function runHeadless(target, project, scriptArguments) {
const snapshot = supervisor.snapshot();
const output = `${snapshot.stdout.text}\n${snapshot.stderr.text}`;
assert.equal(snapshot.exitCode, 0, output);
const stopped = await supervisor.stop();
assert.notEqual(stopped.status, "incomplete", JSON.stringify(stopped));
supervisor.dispose();
supervisor = undefined;
return output;
// Headless can exit zero after a script fails, so require the probe's marker.
assert.ok(output.includes("REA_SEED_REGRESSION_OK "), output);
} finally {
if (supervisor !== undefined) {
const stopped = await supervisor.stop();
assert.notEqual(stopped.status, "incomplete", JSON.stringify(stopped));
supervisor.dispose();
}
await rm(workspace, { recursive: true, force: true });
}
console.log(
"Real Ghidra seed regression passed: descending function order, out-of-range seed, BSS label, BSS code rejection, and owned-process cleanup.",
);
-41
View File
@@ -1,41 +0,0 @@
import { describe, expect, it } from "vitest";
import { adbHostFileName } from "./AdbHostFileName.js";
describe("adbHostFileName", () => {
it.each([
["/data/local/tmp/notes.txt", "notes.txt"],
["/data/data/com.example/databases/app.db", "app.db"],
["/sdcard/My Notes.txt", "My Notes.txt"],
["/system/etc/hosts", "hosts"],
[
"/data/local/tmp/split_config.arm64_v8a.apk",
"split_config.arm64_v8a.apk",
],
["/data/local/tmp/résumé.xml", "résumé.xml"],
["/data/local/tmp/a-b_c.d", "a-b_c.d"],
["/data/local/tmp/", "tmp"],
])("accepts the portable device path %s", (devicePath, fileName) => {
expect(adbHostFileName(devicePath)).toEqual({ ok: true, fileName });
});
it.each([
["/", "empty_file_name"],
["/data/local/tmp/..", "dot_segment"],
["/data/local/tmp/.", "dot_segment"],
["/data/local/tmp/a\\b", "path_separator"],
["/data/local/tmp/re\u007fport", "control_character"],
["/data/local/tmp/ta\tb", "control_character"],
["/data/local/tmp/CON", "reserved_name"],
["/data/local/tmp/nul.txt", "reserved_name"],
["/data/local/tmp/com1", "reserved_name"],
["/data/local/tmp/LPT9", "reserved_name"],
["/data/local/tmp/report:2024", "alternate_data_stream"],
["/data/local/tmp/report.", "trailing_dot_or_space"],
["/data/local/tmp/report ", "trailing_dot_or_space"],
["/data/local/tmp/a*b.txt", "unsupported_character"],
["/data/local/tmp/a?b.txt", "unsupported_character"],
] as const)("refuses %s as %s", (devicePath, reason) => {
expect(adbHostFileName(devicePath)).toEqual({ ok: false, reason });
});
});
-61
View File
@@ -1,61 +0,0 @@
import { posix } from "node:path";
/** Why a device-reported path cannot be used as a host file name. */
export type AdbHostFileNameReason =
| "empty_file_name"
| "dot_segment"
| "path_separator"
| "control_character"
| "unsupported_character"
| "reserved_name"
| "trailing_dot_or_space"
| "alternate_data_stream";
/** A portable host file name, or the exact reason the device name was refused. */
export type AdbHostFileNameResult =
| { readonly ok: true; readonly fileName: string }
| { readonly ok: false; readonly reason: AdbHostFileNameReason };
/**
* Win32 resolves these basenames to a device regardless of case and of any
* extension (`NUL`, `nul.txt`), so a portable host file name must avoid them on
* every host. `(?:\.|$)` keeps ordinary names such as `console.txt` admissible.
*/
const WINDOWS_RESERVED_BASENAME =
/^(?:con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\.|$)/iu;
/** Characters that are not portable in a host file name on Windows or POSIX. */
const UNSUPPORTED_CHARACTER = /[*?"<>|]/u;
/** Control characters (`Cc`), including C0 controls and DEL. */
const CONTROL_CHARACTER = /\p{Cc}/u;
/**
* Derive a portable host file name from a device-reported path.
*
* The device path is a POSIX path on the device filesystem, so its last
* `/`-separated segment is taken with POSIX semantics rather than the host
* `basename` (which would vary between POSIX and Windows hosts). A name is
* accepted only when every host can store it and address it back unambiguously:
* no separator, control, or reserved-device characters, no trailing dot or
* space, no `:` alternate-data-stream marker, and no `.`/`..` segment.
*/
export const adbHostFileName = (devicePath: string): AdbHostFileNameResult => {
const fileName = posix.basename(devicePath);
if (fileName === "") return { ok: false, reason: "empty_file_name" };
if (fileName === "." || fileName === "..")
return { ok: false, reason: "dot_segment" };
if (fileName.includes("\\") || fileName.includes("/"))
return { ok: false, reason: "path_separator" };
if (CONTROL_CHARACTER.test(fileName))
return { ok: false, reason: "control_character" };
if (fileName.includes(":"))
return { ok: false, reason: "alternate_data_stream" };
if (UNSUPPORTED_CHARACTER.test(fileName))
return { ok: false, reason: "unsupported_character" };
if (WINDOWS_RESERVED_BASENAME.test(fileName))
return { ok: false, reason: "reserved_name" };
if (/[. ]$/u.test(fileName))
return { ok: false, reason: "trailing_dot_or_space" };
return { ok: true, fileName };
};
-68
View File
@@ -336,38 +336,12 @@ it("reports a missing selected binary as a capability gap with recovery", async
if (result.ok) return;
const projected = projectAnalysisError(result.error);
expect(projected).toMatchObject({ code: "capability_unavailable" });
expect(projected.message).toContain("REA_ADB_PATH");
expect(projected.remediation).toMatchObject({
action: expect.stringContaining("REA_ADB_PATH") as unknown,
});
expect(projected.details).toMatchObject({
provider_id: "adb",
reason: expect.stringContaining("not found") as unknown,
});
});
it("reports relative adb configuration as an actionable refusal", async () => {
const provider = new AdbProvider({
environment: { REA_ADB_PATH: "relative/adb" },
});
const availability = await provider.inspectAvailability();
expect(availability).toMatchObject({
status: "unavailable",
reason: expect.stringContaining("REA_ADB_PATH") as unknown,
});
const projected = await refusedBy(provider, {
operation: "list_adb_devices",
input: {},
});
expect(projected).toMatchObject({
code: "capability_unavailable",
message: expect.stringContaining("REA_ADB_PATH") as unknown,
remediation: {
action: expect.stringContaining("REA_ADB_PATH") as unknown,
},
});
});
it("cancels before spawning anything", async () => {
const { binary, root } = await writeAdbStub(
(root) => `: > ${join(root, "spawned")}
@@ -478,48 +452,6 @@ exit 1`,
});
});
it.each([
["/data/local/tmp/..", "dot_segment"],
["/data/local/tmp/a\\b", "path_separator"],
["/data/local/tmp/CON", "reserved_name"],
["/data/local/tmp/report.", "trailing_dot_or_space"],
["/data/local/tmp/report:2024", "alternate_data_stream"],
] as const)(
"refuses the unsafe device path %s before pulling",
async (devicePath, reason) => {
const { binary, root } = await writeAdbStub(
(root) => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
: > "${join(root, "pull-ran")}"
exit 0`,
);
const output = join(root, "pulls");
const result = await providerOver(binary).execute({
operation: "pull_adb_file",
input: {
serial: "emulator-5554",
device_path: devicePath,
output_directory: output,
},
});
expect(result.ok).toBe(false);
if (result.ok) return;
const projected = projectAnalysisError(result.error);
expect(projected).toMatchObject({ code: "invalid_request" });
expect(projected.details).toMatchObject({
issues: [
{
path: ["device_path"],
reason: "invalid_value",
message: expect.stringContaining(reason),
},
],
});
// No host directory and no adb invocation are effects of a refused pull.
await expect(readFile(join(root, "pull-ran"))).rejects.toThrow();
await expect(readFile(output)).rejects.toThrow();
},
);
it("pushes a file and verifies the device-side digest", async () => {
const { binary, root } = await writeAdbStub(
() => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
+4 -15
View File
@@ -2,15 +2,15 @@ import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { constants as fsConstants } from "node:fs";
import { access, lstat, mkdir, rm, stat, writeFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { basename, join, resolve } from "node:path";
import type {
ExecutionOptions,
ProviderAvailability,
} from "../application/AnalysisProvider.js";
import { providerConfigurationRefusal } from "../domain/providerConfigurationRefusal.js";
import {
createAnalysisExecution,
providerConfigurationRefusal,
type AnalysisExecution,
} from "../application/AnalysisProvider.js";
import type { AdbRequest } from "../domain/adb/adbDeviceAnalysis.js";
@@ -47,7 +47,6 @@ import {
parsePackageListOutput,
parsePmPathOutput,
} from "./AdbDeviceOutput.js";
import { adbHostFileName } from "./AdbHostFileName.js";
import {
parseAmStartOutput,
parseBugreportCopiedPath,
@@ -1400,23 +1399,13 @@ export class AdbProvider {
},
signal: AbortSignal | undefined,
): Promise<Result<AnalysisExecution, AnalysisError>> {
const derived = adbHostFileName(input.device_path);
if (!derived.ok)
return err(
new AnalysisInputError(operation, undefined, [
{
path: ["device_path"],
reason: "invalid_value",
message: `The device path does not name a portable host file (${derived.reason}); REA never renames a device-reported file.`,
},
]),
);
const directory = await prepareOutputDirectory(
operation,
input.output_directory,
);
if (!directory.ok) return directory;
const localPath = join(directory.value, derived.fileName);
const fileName = basename(input.device_path) || "pulled";
const localPath = join(directory.value, fileName);
const existing = await refuseExistingTarget(operation, localPath);
if (existing !== undefined) return err(existing);
const pulled = await runAdb(
+2 -3
View File
@@ -20,9 +20,9 @@ import type {
ExecutionOptions,
ProviderAvailability,
} from "../application/AnalysisProvider.js";
import { providerConfigurationRefusal } from "../domain/providerConfigurationRefusal.js";
import {
createAnalysisExecution,
providerConfigurationRefusal,
type AnalysisExecution,
} from "../application/AnalysisProvider.js";
import type { ApktoolRequest } from "../domain/apktool/apktoolResourceAnalysis.js";
@@ -410,7 +410,7 @@ export class ApktoolProvider {
);
} catch (cause) {
if (cause instanceof AnalysisCancelledError) return err(cause);
if (cause instanceof ApktoolConfigurationFailure) {
if (cause instanceof ApktoolConfigurationFailure)
return err(
providerConfigurationRefusal(
APKTOOL_PROVIDER_IDENTITY.id,
@@ -418,7 +418,6 @@ export class ApktoolProvider {
cause,
),
);
}
return err(
new AnalysisProtocolError(
`apktool version probe failed for ${request.operation}`,
+26
View File
@@ -5,6 +5,7 @@ import type { NativeToolName } from "../contracts/native/nativeToolContracts.js"
import type { ArtifactAnalysisOperation } from "../contracts/artifactToolContracts.js";
import type { ManagedToolName } from "../contracts/managed/managedToolContracts.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
import { AnalysisCapabilityUnavailableError } from "../domain/analysisErrorCore.js";
import type { AnalysisProfileCommitment } from "../domain/analysisProfile.js";
import type { JsonValue } from "../domain/jsonValue.js";
import type { Result } from "../domain/result.js";
@@ -210,6 +211,31 @@ export type ProviderAvailability =
readonly diagnostics: Readonly<Record<string, JsonValue>>;
};
/** A host tool configuration failure observed before any target work. */
export interface ProviderConfigurationFailure extends Error {
readonly diagnostics: Readonly<Record<string, JsonValue>>;
}
/**
* Refuse an operation because the host tool is misconfigured. The reason,
* not the generic target advice, tells the caller what to change.
*/
export const providerConfigurationRefusal = (
providerId: string,
operation: string,
failure: ProviderConfigurationFailure,
): AnalysisCapabilityUnavailableError => {
const remediation = failure.diagnostics.remediation;
const reason =
typeof remediation === "string"
? `${failure.message}. ${remediation}.`
: `${failure.message}.`;
return new AnalysisCapabilityUnavailableError(providerId, operation, reason, {
cause: failure,
userMessage: reason,
});
};
/** Provider-owned target support observation before profile resolution. */
export type ProviderTargetSupport =
| {
+3 -21
View File
@@ -1,5 +1,4 @@
import type {
AnalysisExecution,
AnalysisOperation,
ExecutionOptions,
} from "./AnalysisProvider.js";
@@ -18,13 +17,6 @@ type AnalysisCall = (
selectorOrigin?: ExecutionOptions["selectorOrigin"],
) => Promise<Result<JsonValue, AnalysisError>>;
type RelationshipCall = (
name: AnalysisOperation,
arguments_: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
selectorOrigin?: ExecutionOptions["selectorOrigin"],
) => Promise<Result<AnalysisExecution, AnalysisError>>;
export interface CallPathTraceInput {
readonly start: string;
readonly goal?: string | undefined;
@@ -50,7 +42,6 @@ interface TraceState {
readonly error: ReturnType<typeof projectAnalysisError>;
}>;
readonly residual: Set<string>;
readonly limitations: Set<string>;
queueIndex: number;
traversalPath: string[];
}
@@ -72,7 +63,6 @@ interface DiscoveredEdge {
/** Trace direct caller/callee relationships until the graph is exhausted or the goal is reached. */
export const traceCallPath = async (
call: AnalysisCall,
callRelationship: RelationshipCall,
input: CallPathTraceInput,
signal?: AbortSignal,
): EnhancedResult => {
@@ -98,7 +88,7 @@ export const traceCallPath = async (
if (address === undefined) break;
const current = state.visited.get(address);
if (current === undefined) continue;
const found = await expandAddress(callRelationship, {
const found = await expandAddress(call, {
input: resolvedInput,
state,
address,
@@ -116,13 +106,12 @@ const createTraceState = (input: CallPathTraceInput): TraceState => ({
edges: new Map(),
failures: [],
residual: new Set(),
limitations: new Set(),
queueIndex: 0,
traversalPath: input.goal === input.start ? [input.start] : [],
});
const expandAddress = async (
call: RelationshipCall,
call: AnalysisCall,
expansion: AddressExpansion,
): Promise<boolean> => {
const { input, state, address, current, signal } = expansion;
@@ -138,7 +127,7 @@ const expandAddress = async (
state.residual.add(`Call relationships were unavailable for ${address}.`);
return false;
}
const related = parseRelatedAddresses(result.value.result, relation);
const related = parseRelatedAddresses(result.value, relation);
if (!related.ok) {
state.failures.push({
address,
@@ -147,12 +136,6 @@ const expandAddress = async (
state.residual.add(`Call relationships were unreadable for ${address}.`);
return false;
}
for (const limitation of result.value.limitations) {
state.limitations.add(limitation);
state.residual.add(
`Call relationships for ${address} are limited: ${limitation}`,
);
}
for (const relatedAddress of sortedUniqueAddresses(related.value)) {
recordEdge(input, state, {
address,
@@ -232,7 +215,6 @@ const projectTrace = (input: CallPathTraceInput, state: TraceState) => ({
residual_unknowns: [...state.residual],
limitations: [
"Direct provider call relationships may omit unresolved indirect calls.",
...state.limitations,
],
});
@@ -3,8 +3,8 @@ import { dirname, join } from "node:path";
import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { systemDoctorHost } from "../../../src/application/Doctor.js";
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
import { systemDoctorHost } from "./Doctor.js";
it("threads the selected host and environment into system diagnostics", async () => {
const environment = {
@@ -25,9 +25,7 @@ it("threads the selected host and environment into system diagnostics", async ()
calls.push({ command, arguments: arguments_, environment: options?.env });
if (command === "sw_vers") return { stdout: "15.0\n", stderr: "" };
if (command === "ldd") return { stdout: "linux-vdso.so.1", stderr: "" };
if (command === "which")
return { stdout: "/selected/bin/rea\n", stderr: "" };
throw new Error(`Unexpected diagnostic command: ${command}`);
return { stdout: "/selected/bin/rea\n", stderr: "" };
},
});
@@ -116,7 +114,7 @@ it("reports no Homebrew Hopper when the cask has no receipt", async () => {
});
expect(await host.brewHopperPath()).toBeUndefined();
expect(new Set(calls.map((arguments_) => arguments_.join(" ")))).toEqual(
new Set(["--caskroom"]),
);
expect(
calls.every((arguments_) => arguments_.join(" ") === "--caskroom"),
).toBe(true);
});
+3 -24
View File
@@ -1,7 +1,6 @@
import { z } from "zod";
import type {
AnalysisExecution,
AnalysisOperation,
AnalysisOperationPort,
ExecutionOptions,
@@ -167,8 +166,6 @@ export class EnhancedTools {
return traceCallPath(
(name, arguments_, operationSignal, selectorOrigin) =>
this.#call(name, arguments_, operationSignal, selectorOrigin),
(name, arguments_, operationSignal, selectorOrigin) =>
this.#observe(name, arguments_, operationSignal, selectorOrigin),
call.input,
signal,
);
@@ -362,7 +359,7 @@ export class EnhancedTools {
const level = String(current.depth);
graph[level] ??= [];
const result = await this.#observe(
const result = await this.#call(
tool,
{ procedure: current.address },
signal,
@@ -376,7 +373,7 @@ export class EnhancedTools {
});
continue;
}
const related = parseRelatedAddresses(result.value.result, relation);
const related = parseRelatedAddresses(result.value, relation);
if (!related.ok) {
graph[level].push({
address: current.address,
@@ -389,9 +386,6 @@ export class EnhancedTools {
address: current.address,
status: "ok",
calls: [...related.value],
...(result.value.limitations.length === 0
? {}
: { limitations: [...result.value.limitations] }),
});
for (const address of related.value) {
if (!discovered.has(address)) {
@@ -529,28 +523,13 @@ export class EnhancedTools {
signal?: AbortSignal,
selectorOrigin?: ExecutionOptions["selectorOrigin"],
): Promise<Result<JsonValue, AnalysisError>> {
const execution = await this.#observe(
name,
arguments_,
signal,
selectorOrigin,
);
return execution.ok ? ok(execution.value.result) : execution;
}
async #observe(
name: AnalysisOperation,
arguments_: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
selectorOrigin?: ExecutionOptions["selectorOrigin"],
): Promise<Result<AnalysisExecution, AnalysisError>> {
if (isAborted(signal)) return err(new AnalysisCancelledError(name));
const execution = await this.analysis.execute(name, arguments_, {
...(signal === undefined ? {} : { signal }),
...(selectorOrigin === undefined ? {} : { selectorOrigin }),
});
if (isAborted(signal)) return err(new AnalysisCancelledError(name));
return execution;
return execution.ok ? ok(execution.value.result) : execution;
}
}
+2 -17
View File
@@ -8,8 +8,6 @@ import type {
} from "./SetupTypes.js";
import type { SetupClient } from "./SupportedClients.js";
import type { SetupHopperInstallResult } from "./SetupInstallFailure.js";
import { err, ok } from "../domain/result.js";
import { SetupSkillInstallError } from "./SetupSkill.js";
import type { DoctorCheck, DoctorReport, DoctorScope } from "./Doctor.js";
import type { LinuxDistribution } from "./LinuxHopper.js";
import { PRODUCT_IDENTITY, SDK_IDENTITY } from "../identity.js";
@@ -124,22 +122,9 @@ export class FakeSetupHost implements SetupHost {
);
skillNeedsInstall = (): Promise<boolean> =>
Promise.resolve(this.skill !== "unchanged");
installSkill = () => {
installSkill = (): Promise<"installed" | "unchanged" | "failed"> => {
this.skillInstalls += 1;
return Promise.resolve(
this.skill === "failed"
? err(
new SetupSkillInstallError({
path: "/fixture/home/.agents/skills/reverse-engineer-anything/SKILL.md",
stage: "write",
reason: "injected skill write failure",
rollbackFailures: [],
recoveryBackupPaths: [],
cause: new Error("injected skill write failure"),
}),
)
: ok(this.skill),
);
return Promise.resolve(this.skill);
};
doctor = (scope?: DoctorScope): Promise<DoctorReport> => {
this.doctorCalls += 1;
+4 -18
View File
@@ -6,7 +6,6 @@ import { ConfigurationError } from "../domain/configurationErrors.js";
import { SUPPORTED_NODE_VERSION_PROSE } from "../domain/runtimeVersion.js";
import { FakeSetupHost, options } from "./Setup.fixture.js";
import { runSetup } from "./Setup.js";
import type { SetupProgressEvent } from "./SetupTypes.js";
describe("setup workflow", () => {
it("rejects Node 25 before applying setup actions", async () => {
@@ -58,27 +57,14 @@ describe("setup workflow", () => {
const host = new FakeSetupHost();
host.hopper = "/Applications/Hopper";
host.skill = "failed";
const progress: SetupProgressEvent[] = [];
const result = await runSetup(
{
...options(true),
installSkill: true,
clientIds: [],
onProgress: (event) => {
if (event.actionId === "install_skill") progress.push(event);
},
},
{ ...options(true), installSkill: true, clientIds: [] },
host,
);
expect(result.status).toBe("needs_human");
expect(result.code).toBe("skill_install_failed");
expect(result.remediation).toContain("REA skill write failed at");
expect(result.remediation).toContain("injected skill write failure");
expect(result.remediation).toContain("Check permissions and rerun setup.");
expect(progress.at(-1)).toMatchObject({
state: "failed",
detail: expect.stringContaining("injected skill write failure"),
});
expect(result.remediation).toBe(
"REA analysis skill could not be installed or verified. Check permissions for its planned skill directories, then rerun setup.",
);
});
it("delegates remaining unhealthy checks to doctor remediation", async () => {
+14 -18
View File
@@ -26,7 +26,6 @@ import {
import { AnalysisError } from "../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import type { SetupSkillInstallError } from "./SetupSkill.js";
/** Discover, approve, and apply setup actions idempotently. */
export const runSetup = async (
@@ -316,20 +315,18 @@ const runSetupWorkflow = async (
: { onProgress: options.onProgress }),
});
if (clientFailure !== undefined) return fail(clientFailure);
if (planSelection.installSkill) {
const skillFailure = await installSkillAction(
if (
planSelection.installSkill &&
!(await installSkillAction(
host,
options,
appliedActions,
selectedSkillClientIds,
))
)
return fail(
"REA analysis skill could not be installed or verified. Check permissions for its planned skill directories, then rerun setup.",
);
if (skillFailure !== undefined) {
return {
...(await fail(skillFailure.remediation)),
code: "skill_install_failed",
};
}
}
const doctor = summarizeDoctor(await host.doctor(readinessScope));
const remediation = combineRemediation(
finalSetupRemediation(
@@ -420,7 +417,7 @@ const installSkillAction = async (
options: SetupOptions,
appliedActions: string[],
skillClientIds: readonly string[],
): Promise<SetupSkillInstallError | undefined> => {
): Promise<boolean> => {
const label = "REA reverse-engineering skill";
emitProgress(options, {
actionId: "install_skill",
@@ -428,23 +425,22 @@ const installSkillAction = async (
state: "started",
});
const skill = await host.installSkill(skillClientIds);
if (!skill.ok) {
if (skill === "failed") {
emitProgress(options, {
actionId: "install_skill",
label,
state: "failed",
detail: skill.error.message,
});
return skill.error;
return false;
}
if (skill.value === "installed") appliedActions.push("installed_skill");
if (skill === "installed") appliedActions.push("installed_skill");
emitProgress(options, {
actionId: "install_skill",
label,
state: skill.value === "installed" ? "completed" : "warning",
...(skill.value === "unchanged" ? { detail: "Already current" } : {}),
state: skill === "installed" ? "completed" : "warning",
...(skill === "unchanged" ? { detail: "Already current" } : {}),
});
return undefined;
return true;
};
const setupFailure = async (
-1
View File
@@ -11,7 +11,6 @@ export type SetupFailureCode =
| "mount_failed"
| "bundle_invalid"
| "copy_failed"
| "skill_install_failed"
| "cleanup_failed";
export type SetupHopperInstallResult =
+46 -183
View File
@@ -2,15 +2,10 @@ import { copyFile, lstat, mkdir, readFile, rm } from "node:fs/promises";
import { constants as fsConstants } from "node:fs";
import { dirname, join } from "node:path";
import writeFileAtomic from "write-file-atomic";
import { err, ok, type Result } from "../domain/result.js";
import { readBoundedFileBytes } from "../process/BoundedFileBytes.js";
import { PRODUCT_IDENTITY } from "../identity.js";
import {
NonRegularFileReadError,
RegularFileChangedError,
} from "../filesystem/RegularFile.js";
import { withRegularFile } from "./RegularFileRead.js";
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
import { readRegularFileText } from "./RegularFileRead.js";
import { clientSkillDirectories } from "./SupportedClients.js";
export const MANAGED_SKILL_FILES = [
@@ -28,119 +23,14 @@ export const isManagedSkillManifest = (content: string): boolean =>
content.startsWith(`---\r\nname: ${PRODUCT_IDENTITY.skillName}\r\n`);
export const readSkillFile = (path: string): Promise<string> =>
readSkillFileState(path).then(({ content }) => content);
interface SkillFileState {
readonly content: string;
readonly mode: number;
}
const readSkillFileState = async (path: string): Promise<SkillFileState> =>
withRegularFile(
path,
async (handle, stats) => {
const bytes = await readBoundedFileBytes(
handle,
stats.size,
undefined,
stats.size,
);
if (bytes === undefined || bytes.length !== stats.size)
throw new RegularFileChangedError(path);
return { content: bytes.toString("utf8"), mode: stats.mode & 0o7777 };
},
{ symlinks: "reject" },
);
readRegularFileText(path, { symlinks: "reject" });
interface CanonicalSkillFile {
readonly content: string;
readonly destination: string;
readonly original: SkillFileState | undefined;
readonly original: string | undefined;
}
export interface SkillInstallFileOperations {
readonly mkdir: (path: string) => Promise<void>;
readonly backup: (source: string, destination: string) => Promise<void>;
readonly write: (
path: string,
content: string,
mode: number,
) => Promise<void>;
readonly remove: (path: string) => Promise<void>;
}
export interface SkillInstallRollbackFailure {
readonly path: string;
readonly reason: string;
}
/** Expected failure while replacing a managed skill bundle. */
export class SetupSkillInstallError extends Error {
readonly _tag = "SetupSkillInstallError";
readonly path: string;
readonly stage: "prepare" | "backup" | "write" | "readback";
readonly reason: string;
readonly rollbackFailures: readonly SkillInstallRollbackFailure[];
readonly recoveryBackupPaths: readonly string[];
constructor(input: {
readonly path: string;
readonly stage: "prepare" | "backup" | "write" | "readback";
readonly reason: string;
readonly rollbackFailures: readonly SkillInstallRollbackFailure[];
readonly recoveryBackupPaths: readonly string[];
readonly cause: unknown;
}) {
const message = [
`REA skill ${input.stage} failed at ${input.path}: ${input.reason}.`,
...(input.rollbackFailures.length === 0
? []
: [
`Rollback also failed for ${input.rollbackFailures.map(({ path, reason }) => `${path} (${reason})`).join(", ")}.`,
...(input.recoveryBackupPaths.length === 0
? []
: [
`Existing-file backups remain at ${input.recoveryBackupPaths.join(", ")}.`,
]),
]),
].join(" ");
super(message, { cause: input.cause });
this.path = input.path;
this.stage = input.stage;
this.reason = input.reason;
this.rollbackFailures = input.rollbackFailures;
this.recoveryBackupPaths = input.recoveryBackupPaths;
}
get remediation(): string {
return `${this.message} Check permissions and rerun setup.`;
}
}
export type SkillInstallValue = "installed" | "unchanged";
export type SkillInstallResult = Result<
SkillInstallValue,
SetupSkillInstallError
>;
const systemSkillInstallFileOperations: SkillInstallFileOperations = {
mkdir: async (path) => {
await mkdir(path, { recursive: true });
},
backup: async (source, destination) => {
try {
await copyFile(source, destination, fsConstants.COPYFILE_EXCL);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "EEXIST")
return;
throw cause;
}
},
write: (path, content, mode) =>
writeFileAtomic(path, content, { encoding: "utf8", mode }),
remove: (path) => rm(path, { force: true }),
};
/** Managed skill location selected for a client family. */
export interface SkillDestination {
readonly client: string;
@@ -197,11 +87,9 @@ export const existingSkillDestinations = async (
return existing;
};
const readOptionalSkillState = async (
path: string,
): Promise<SkillFileState | undefined> => {
const readOptionalText = async (path: string): Promise<string | undefined> => {
try {
return await readSkillFileState(path);
return await readSkillFile(path);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
return undefined;
@@ -229,7 +117,7 @@ const canonicalSkillFiles = async (
),
"utf8",
),
original: await readOptionalSkillState(destination),
original: await readOptionalText(destination),
};
}),
),
@@ -245,7 +133,7 @@ export const canonicalSkillNeedsInstall = async (
try {
return (
await canonicalSkillFiles(home, clientIds, environment, platform)
).some(({ content, original }) => original?.content !== content);
).some(({ content, original }) => original !== content);
} catch (cause: unknown) {
// Unreadable skill state fails open to install so setup can repair it.
void cause;
@@ -253,42 +141,39 @@ export const canonicalSkillNeedsInstall = async (
}
};
const causeReason = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const writeText = (path: string, content: string): Promise<void> =>
writeFileAtomic(path, content, { encoding: "utf8", mode: 0o600 });
const causePath = (cause: unknown): string | undefined =>
cause instanceof Error && "path" in cause && typeof cause.path === "string"
? cause.path
: undefined;
const preserveSkillBackup = async (
source: string,
destination: string,
): Promise<void> => {
try {
await copyFile(source, destination, fsConstants.COPYFILE_EXCL);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "EEXIST")
return;
throw cause;
}
};
const restoreSkillFiles = async (
attempted: readonly CanonicalSkillFile[],
operations: SkillInstallFileOperations,
): Promise<readonly SkillInstallRollbackFailure[]> => {
const failures: SkillInstallRollbackFailure[] = [];
for (const { destination, original } of [...attempted].reverse()) {
try {
if (original === undefined) await operations.remove(destination);
else await operations.write(destination, original.content, original.mode);
} catch (cause: unknown) {
failures.push({ path: destination, reason: causeReason(cause) });
}
changed: readonly CanonicalSkillFile[],
): Promise<void> => {
for (const { destination, original } of [...changed].reverse()) {
if (original === undefined) await rm(destination, { force: true });
else await writeText(destination, original);
}
return failures;
};
/** Transactionally install or upgrade the canonical REA skill and references. */
/** Transactionally install the canonical skill bundle and optional references. */
export const installCanonicalSkill = async (
home: string,
clientIds: readonly string[] = [],
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
operations: SkillInstallFileOperations = systemSkillInstallFileOperations,
): Promise<SkillInstallResult> => {
let attempted: CanonicalSkillFile[] = [];
let stage: "prepare" | "backup" | "write" | "readback" = "prepare";
let failingPath = "skill bundle";
): Promise<"installed" | "unchanged" | "failed"> => {
let changed: readonly CanonicalSkillFile[] = [];
try {
const canonical = await canonicalSkillFiles(
home,
@@ -296,52 +181,30 @@ export const installCanonicalSkill = async (
environment,
platform,
);
const changed = canonical.filter(
({ content, original }) => original?.content !== content,
);
if (changed.length === 0) return ok("unchanged");
changed = canonical.filter(({ content, original }) => original !== content);
if (changed.length === 0) return "unchanged";
stage = "backup";
for (const { destination, original } of changed) {
failingPath = destination;
await operations.mkdir(dirname(destination));
await mkdir(dirname(destination), { recursive: true });
if (original !== undefined)
await operations.backup(destination, `${destination}.rea.backup`);
await preserveSkillBackup(destination, `${destination}.rea.backup`);
}
stage = "write";
for (const file of changed) {
failingPath = file.destination;
// An atomic writer can publish and then reject; include the target before
// entering it so rollback covers that ambiguous outcome.
attempted.push(file);
await operations.write(
file.destination,
file.content,
file.original?.mode ?? 0o600,
);
}
stage = "readback";
for (const { destination, content } of changed) {
failingPath = destination;
for (const { destination, content } of changed)
await writeText(destination, content);
for (const { destination, content } of changed)
if ((await readSkillFile(destination)) !== content)
throw new Error(`skill readback mismatch: ${destination}`);
}
return ok("installed");
return "installed";
} catch (cause: unknown) {
failingPath = causePath(cause) ?? failingPath;
const rollbackFailures = await restoreSkillFiles(attempted, operations);
return err(
new SetupSkillInstallError({
path: failingPath,
stage,
reason: causeReason(cause),
rollbackFailures,
recoveryBackupPaths: attempted.flatMap(({ destination, original }) =>
original === undefined ? [] : [`${destination}.rea.backup`],
),
cause,
}),
);
// Install failure preserves the original error outcome; report cause inline.
void cause;
try {
await restoreSkillFiles(changed);
} catch (restoreCause: unknown) {
// Per-file backups remain beside changed files for operator recovery.
void restoreCause;
}
return "failed";
}
};
+4 -2
View File
@@ -1,6 +1,6 @@
import type { DoctorReport, DoctorScope, runDoctor } from "./Doctor.js";
import type { LinuxDistribution } from "./LinuxHopper.js";
import type { SkillDestination, SkillInstallResult } from "./SetupSkill.js";
import type { SkillDestination } from "./SetupSkill.js";
import type { SetupClient } from "./SupportedClients.js";
import type {
SetupFailureCode,
@@ -70,7 +70,9 @@ export interface SetupHost {
command: readonly string[],
): Promise<ClientConfigurationInspection>;
skillNeedsInstall(clientIds: readonly string[]): Promise<boolean>;
installSkill(clientIds: readonly string[]): Promise<SkillInstallResult>;
installSkill(
clientIds: readonly string[],
): Promise<"installed" | "unchanged" | "failed">;
doctor(scope?: DoctorScope): Promise<Awaited<ReturnType<typeof runDoctor>>>;
}
/** Structured setup outcome carrying remediation instead of prompting. */
-1
View File
@@ -38,7 +38,6 @@ export interface WebModuleResolutionBatch {
/** Replaceable native resolution engine, with no provider protocol in callers. */
export interface WebModuleResolutionPort {
close?(): Promise<void>;
resolve(
input: {
readonly importerUrl: string;
-4
View File
@@ -69,10 +69,6 @@ export class WebModuleTraceService {
readonly resolver: WebModuleResolutionPort,
) {}
close(): Promise<void> {
return this.resolver.close?.() ?? Promise.resolve();
}
async trace(
rawInput: unknown,
options?: ExecutionOptions,
@@ -115,63 +115,6 @@ it("exports independent nested query, workflow and Evidence payloads", () => {
expect(serializeAnalysisSnapshot(fresh.value)).toBe(expected);
});
it("rebinds replayed target subjects while preserving child artifact subjects", () => {
const cache = new AnalysisSnapshotCache();
const target = ANALYSIS_SNAPSHOT_TARGET;
const profile = ANALYSIS_SNAPSHOT_PROFILE;
const relocatedTarget = { ...target, path: "/tmp/relocated-app" };
const parameters = { address: "0x1000" };
const locations = [{ kind: "address" as const, address: "0x1000" }];
cache.record({
target,
profile,
operation: "procedure_info",
parameters,
execution: createAnalysisExecution(
{ name: "main" },
ANALYSIS_SNAPSHOT_PROVIDER,
{
subject: target,
rawResult: { provider: "raw" },
locations,
analysisProfile: profile,
},
),
});
const replayedTarget = cache.lookup(
relocatedTarget,
profile,
"procedure_info",
parameters,
);
expect(replayedTarget?.subject?.path).toBe(relocatedTarget.path);
expect(replayedTarget?.rawResult).toEqual({ provider: "raw" });
expect(replayedTarget?.locations).toEqual(locations);
const childSubject = {
path: "/tmp/child.zip",
sha256: "b".repeat(64),
format: "zip" as const,
};
const childParameters = { address: "0x2000" };
cache.record({
target,
profile,
operation: "procedure_info",
parameters: childParameters,
execution: createAnalysisExecution(
{ name: "child" },
ANALYSIS_SNAPSHOT_PROVIDER,
{ subject: childSubject, analysisProfile: profile },
),
});
expect(
cache.lookup(relocatedTarget, profile, "procedure_info", childParameters)
?.subject,
).toMatchObject(childSubject);
});
describe("analysis snapshot cache capacity", () => {
it("shares capacity across query kinds while allowing replacements and fresh partitions", () => {
const cache = new AnalysisSnapshotCache();
+16 -17
View File
@@ -8,7 +8,6 @@ import {
} from "../../domain/evidenceErrors.js";
import { NoBinaryOpenError } from "../../domain/configurationErrors.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import { rebindEvidenceSubjectTargetPath } from "../../domain/evidence.js";
import { err, ok, type Result } from "../../domain/result.js";
import {
analysisQueryId,
@@ -278,21 +277,7 @@ export class AnalysisSnapshotCache {
);
const cached = this.#entries.get(queryId);
if (cached === undefined) return undefined;
const cachedSubject = cached.execution.subject;
const subject =
cachedSubject === null
? null
: rebindEvidenceSubjectTargetPath(
{
path: cachedSubject.path,
sha256: cachedSubject.sha256,
format: cachedSubject.format,
...(cachedSubject.architecture === null
? {}
: { architecture: cachedSubject.architecture }),
},
target,
);
const subject = cached.execution.subject;
return structuredClone({
result: cached.execution.result,
rawResult: cached.execution.raw_result,
@@ -303,7 +288,21 @@ export class AnalysisSnapshotCache {
"Loaded from a local REA analysis snapshot; this call did not re-run the provider.",
],
locations: cached.execution.locations,
subject,
subject:
subject === null
? null
: subject.architecture === null
? {
path: subject.path,
sha256: subject.sha256,
format: subject.format,
}
: {
path: subject.path,
sha256: subject.sha256,
format: subject.format,
architecture: subject.architecture,
},
});
}
@@ -5,12 +5,12 @@ import { Ajv2020 } from "ajv/dist/2020.js";
import { expect, it } from "vitest";
import { z } from "zod";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { matchJavaScriptApplicationVersions } from "../../../../src/domain/javascript/javascriptApplicationVersionKeys.js";
import { findApplicationFeatureSeeds } from "../../../../src/domain/javascript/javascriptFeatureSeed.js";
import { traceApplicationFeatureInputSchema } from "../../../../src/domain/javascript/javascriptFeatureTraceSchemas.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { matchJavaScriptApplicationVersions } from "../../domain/javascript/javascriptApplicationVersionKeys.js";
import { findApplicationFeatureSeeds } from "../../domain/javascript/javascriptFeatureSeed.js";
import { traceApplicationFeatureInputSchema } from "../../domain/javascript/javascriptFeatureTraceSchemas.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
const analyzeSource = async (
source: string,
@@ -91,7 +91,4 @@ const parameters = (
): EvidenceObservation["parameters"] => ({
format: input.format,
integrity_policy: input.integrity_policy,
...(input.source_paths === undefined
? {}
: { source_paths: input.source_paths }),
});
@@ -3,9 +3,9 @@ import { join } from "node:path";
import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
it.each([
{
@@ -3,127 +3,12 @@ import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { z } from "zod";
import { parseEvidence } from "../../../../src/domain/evidence.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { writeJavaScriptSourceSelectionFixture } from "../../../fixtures/javascriptSourceSelection.js";
import { ArtifactReaderFailure } from "../../../../src/artifacts/ArtifactReader.js";
import { scanCanonicalArtifactInventory } from "../../../fixtures/artifactInventory.js";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import { projectAnalysisError } from "../../../../src/domain/analysisErrorProjection.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
describe("Selected JavaScript source failure diagnostics", () => {
it("analyzes selected files in first-request order and reports only selected progress", async () => {
const root = await createTestTempDirectory("rea-js-selected-priority-");
await writeJavaScriptSourceSelectionFixture(root);
await writeFile(join(root, "z-later.mjs"), "export const later = 1;\n");
const progress: string[] = [];
const result = await analyzeJavaScriptApplication(
{
input_path: root,
source_paths: ["z-later.mjs", "main.mjs", "z-later.mjs"],
},
{
progress: {
report: async (event) => {
if (event.phase === "parse_javascript_source")
progress.push(event.message);
},
},
},
);
if (!result.ok) throw result.error;
const analysis = javascriptApplicationAnalysisResultSchema.parse(
result.value.normalized_result,
);
expect(analysis.source_selection).toEqual({
requested_paths: ["z-later.mjs", "main.mjs", "z-later.mjs"],
unselected_files: 5,
});
expect(analysis.statistics.parsed_javascript_files).toBe(2);
expect(progress).toEqual([
expect.stringContaining("z-later.mjs"),
expect.stringContaining("main.mjs"),
]);
});
it("verifies unselected bytes against the inventoried identity without decoding them", async () => {
const root = await createTestTempDirectory("rea-js-unselected-integrity-");
await writeJavaScriptSourceSelectionFixture(root);
const result = await analyzeJavaScriptApplication(
{ input_path: root, source_paths: ["main.mjs"] },
{
progress: {
report: async (event) => {
if (event.phase === "read_javascript_artifacts")
await writeFile(
join(root, "vendor.js"),
"changed after inventory\n",
);
},
},
},
);
if (result.ok)
throw new Error(
"Unselected content changes must still fail integrity verification",
);
expect(projectAnalysisError(result.error)).toMatchObject({
code: "artifact_operation_failed",
details: { reason: "integrity" },
});
});
it("retains selected facts and unselected scope after cancellation between selected sources", async () => {
const root = await createTestTempDirectory("rea-js-selected-cancellation-");
await writeJavaScriptSourceSelectionFixture(root);
await writeFile(join(root, "z-later.mjs"), "export const later = 1;\n");
const controller = new AbortController();
const source_paths = ["main.mjs", "z-later.mjs"];
const result = await analyzeJavaScriptApplication(
{ input_path: root, source_paths },
{
signal: controller.signal,
progress: {
report: async (event) => {
if (
event.phase === "parse_javascript_source" &&
event.message.includes("z-later.mjs")
)
controller.abort();
},
},
},
);
if (result.ok) throw new Error("Expected selected-source cancellation");
expect(projectAnalysisError(result.error).code).toBe("cancelled");
const evidence = parseEvidence(result.error.partialObservation);
const partial = javascriptApplicationAnalysisResultSchema.parse(
evidence.normalized_result,
);
expect(partial.source_selection).toEqual({
requested_paths: source_paths,
unselected_files: 5,
});
expect(
partial.semantic_graph.nodes.some(
({ kind, label }) => kind === "function" && label === "selected",
),
).toBe(true);
expect(partial.statistics.parsed_javascript_files).toBe(1);
expect(partial.graph.coverage.status).toBe("partial");
expect(
partial.graph.nodes
.flatMap(({ observations }) =>
observations.map(({ properties }) => properties),
)
.filter(
(properties) => properties.text_status === "outside-analysis-scope",
),
).toHaveLength(5);
});
});
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import { scanCanonicalArtifactInventory } from "../../../tests/fixtures/artifactInventory.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
describe("JavaScript application failure diagnostics", () => {
it("identifies the rejected result field in caller-visible diagnostics", async () => {
@@ -66,9 +66,6 @@ export const analyzeJavaScriptApplicationValidated = async (
input_path: input.input_path,
format: input.format,
integrity_policy: input.integrity_policy,
...(input.source_paths === undefined
? {}
: { source_paths: input.source_paths }),
...(input.max_heap_mb === undefined
? {}
: { max_heap_mb: input.max_heap_mb }),
@@ -3,11 +3,11 @@ import { join } from "node:path";
import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { javaScriptSemanticQueryResultSchema } from "../../../../src/domain/javascript/javascriptSemanticQuerySchemas.js";
import { queryJavaScriptSemanticGraph } from "../../../../src/domain/javascript/javascriptSemanticQuery.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { javaScriptSemanticQueryResultSchema } from "../../domain/javascript/javascriptSemanticQuerySchemas.js";
import { queryJavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticQuery.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
it.each([
{
@@ -3,9 +3,9 @@ import { join } from "node:path";
import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
const graphIdentities = async (source: string) => {
const inputPath = await createTestTempDirectory("rea-js-template-literal-");
@@ -3,11 +3,11 @@ import { join } from "node:path";
import { expect, it } from "vitest";
import { canonicalJson } from "../../../../src/domain/comparisonSemantics.js";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { parseJavaScriptSource } from "../../../../src/domain/javascript/javascriptSourceParser.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
import { canonicalJson } from "../../domain/comparisonSemantics.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
type ApplicationAnalysis = ReturnType<
typeof javascriptApplicationAnalysisResultSchema.parse
@@ -11,10 +11,7 @@ import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSem
import { projectJavaScriptModuleSemantics } from "../../domain/javascript/javascriptModuleSemanticIr.js";
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js";
import {
flattenSourceMapLeaves,
sourceMapJsonText,
} from "../../domain/sourceMapEnvelope.js";
import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js";
import type {
JavaScriptSourceRange,
JavaScriptSourcePoint,
@@ -140,31 +137,12 @@ export const analyzeAndAdoptJavaScriptArtifactFiles = async (
}> => {
const state = emptyArtifactAnalysis<JavaScriptModuleSemanticIr>();
let interruption: AnalysisError | null = null;
const selectedFileCount =
fileSet.files.length - (fileSet.source_selection?.unselected_files ?? 0);
let selectedFileIndex = 0;
// Caller-excluded sources are never analyzed, so neither cancellation nor
// another interruption describes them.
const recordOutsideSelection = (file: JavaScriptArtifactFile): boolean => {
if (file.text.included || file.text.reason !== "outside-analysis-scope")
return false;
state.files.push({ file, javascript: null, semantic: null });
if (file.kind === "javascript")
projection.recordUnavailableFile(
file,
"Source was not analyzed because it is outside the caller-selected source_paths.",
null,
"incomplete-module",
);
return true;
};
for (const file of fileSet.files) {
for (const [index, file] of fileSet.files.entries()) {
try {
await setImmediate();
await beforeFile(file, index, fileSet.files.length);
if (signal?.aborted === true)
throw new AnalysisCancelledError("analyze_javascript_application");
if (recordOutsideSelection(file)) continue;
await beforeFile(file, selectedFileIndex++, selectedFileCount);
if (file.kind !== "javascript" || !file.text.included) {
const steps = analyzeArtifactFileSteps(file, {
state,
@@ -254,7 +232,7 @@ export const analyzeAndAdoptJavaScriptArtifactFiles = async (
: cause instanceof AnalysisError
? cause
: null;
if (interruption !== null && !recordOutsideSelection(file)) {
if (interruption !== null) {
state.files.push({
file,
javascript: null,
@@ -288,8 +266,7 @@ export const analyzeAndAdoptJavaScriptArtifactFiles = async (
: null,
};
for (const file of fileSet.files) {
if (analyzedPaths.has(file.path) || recordOutsideSelection(file))
continue;
if (analyzedPaths.has(file.path)) continue;
state.files.push({
file,
javascript: null,
@@ -359,10 +336,6 @@ function* analyzeArtifactFileSteps<
context: ArtifactAnalysisContext<SemanticIr>,
): Generator<void, void> {
const { state } = context;
if (!file.text.included && file.text.reason === "outside-analysis-scope") {
state.files.push({ file, javascript: null, semantic: null });
return;
}
addStructuredObservations(file, state);
if (file.kind === "html" && file.text.included)
state.htmlScripts.push(...parseHtmlScripts(file.path, file.text.value));
@@ -513,12 +486,12 @@ const parseSourceMap = (
if (!file.text.included) return unavailableSourceMap(file);
let value: unknown;
try {
value = JSON.parse(sourceMapJsonText(file.text.value));
value = JSON.parse(file.text.value);
} catch (cause: unknown) {
void cause;
return invalidSourceMap(file, "Source map is not valid JSON.");
}
const maps = flattenSourceMapLeaves(value);
const maps = flattenSourceMaps(value);
if (maps === undefined)
return invalidSourceMap(file, "Source map is not a version 3 map.");
return collectSourceMapOriginals(file, maps);
@@ -583,6 +556,13 @@ const unavailableSourceMap = (
};
};
const flattenSourceMaps = (
root: unknown,
): Readonly<Record<string, unknown>>[] | undefined => {
const leaves = flattenSourceMapLeaves(root);
return leaves === undefined ? undefined : [...leaves];
};
const invalidSourceMap = (
file: JavaScriptArtifactFile,
limitation: string,
@@ -168,7 +168,6 @@ const graphCoverage = (context: JavaScriptArtifactGraphContext) => {
javascript !== null && javascript.parse_status === "partial",
);
const unknownGap =
(context.fileSet.source_selection?.unselected_files ?? 0) > 0 ||
context.analysis.files.some(
({ analysis_failure, application_projection_failure }) =>
analysis_failure !== undefined ||
@@ -245,11 +244,6 @@ const graphLimitations = (
);
return [
...context.analysis.limitations,
...(context.fileSet.source_selection === undefined
? []
: [
`Caller-selected source analysis leaves ${String(context.fileSet.source_selection.unselected_files)} inventoried files outside analysis scope. Inventory identity covers the complete artifact; relationships through unselected sources remain unknown.`,
]),
...context.snapshot.integrity_contradictions.map(
({ logical_path: path }) =>
`Artifact integrity metadata contradicts observed bytes at ${path}; the observed bytes are untrusted.`,
@@ -199,15 +199,7 @@ export const addJavaScriptArtifactFiles = (
}),
});
}
if (!file.text.included && file.text.reason === "outside-analysis-scope")
addUnavailableStaticParseScope(context, {
file,
asset: target,
operation: "select-javascript-sources",
limitation:
"Source is outside the caller-selected source_paths; it was inventoried but not analyzed.",
});
else if (analyzed.analysis_failure !== undefined)
if (analyzed.analysis_failure !== undefined)
addUnavailableStaticParseScope(context, {
file,
asset: target,
@@ -176,7 +176,9 @@ const contextualCandidate = (
): string | CandidateResolution => {
const { context } = input;
let declared =
context === "html-reference" || context === "url-reference"
context === "html-reference" ||
context === "url-reference" ||
(context === "module-specifier" && input.moduleKind !== "require")
? stripQueryAndFragment(input.declaredPath)
: input.declaredPath;
if (context === "html-reference") return htmlCandidate(input);
@@ -188,9 +190,9 @@ const contextualCandidate = (
"URL and Node builtin schemes are outside static artifact module resolution.",
]);
if (!declared.startsWith(".") && !declared.startsWith("/"))
return bareModuleCandidate(input);
return bareModuleCandidate(input, declared);
if (input.moduleKind !== "require") {
const decoded = decodeModuleUrlPath(stripQueryAndFragment(declared));
const decoded = decodeModuleUrlPath(declared);
if (typeof decoded !== "string") return decoded;
declared = decoded;
}
@@ -235,51 +237,34 @@ const relativeCandidate = (sourcePath: string, declared: string): string =>
const bareModuleCandidate = (
input: ResolveArtifactPathInput,
declared: string,
): CandidateResolution => {
const request = parseBarePackageRequest(input.declaredPath, input.moduleKind);
if (request === null || isBuiltin(input.declaredPath))
const packageName = barePackageName(declared);
if (packageName === null || isBuiltin(declared))
return unresolvedCandidate("external", [
"The bare specifier is a Node builtin or invalid package name.",
]);
if (input.moduleKind === "import" && request.packageName.includes("%"))
if (input.moduleKind === "import" && packageName.includes("%"))
return unresolvedCandidate("rejected", [
"Percent signs are not admitted in ES module package names.",
]);
const self = packageSelfCandidate(input, request);
const self = commonJsSelfCandidate(input, declared);
if (self !== null) return self;
// ESM constructs a package.json URL; its parsed identity is not a literal
// dependency directory. Self resolution above still uses the exact name.
let dependencyDirectory = posix.join("node_modules", request.packageName);
if (input.moduleKind === "import") {
const metadataUrl = new URL(
`./node_modules/${request.packageName}/package.json`,
PACKAGE_ARTIFACT_ROOT_URL,
);
if (!metadataUrl.pathname.endsWith("/package.json"))
return unresolvedCandidate("external", [
`ESM package name ${JSON.stringify(request.packageName)} produces a dependency URL whose pathname does not name package.json; no literal or suffix-stripped package directory was searched.`,
]);
const decoded = decodeModuleUrlPath(
metadataUrl.pathname.slice(PACKAGE_ARTIFACT_ROOT_URL.pathname.length),
);
if (typeof decoded !== "string") return decoded;
const confined = confineCandidate(decoded);
if (typeof confined !== "string") return confined;
dependencyDirectory = posix.dirname(confined);
}
const subpath =
declared === packageName ? "." : `.${declared.slice(packageName.length)}`;
const source = input.files.get(input.sourcePath);
let directory = posix.dirname(input.sourcePath);
while (true) {
const candidate = posix.join(directory, dependencyDirectory);
const candidate = posix.join(directory, "node_modules", packageName);
const metadata = readPackageMetadata(input, candidate);
if (request.dependencyExportsEligible && metadata.kind !== "absent")
return resolvePackageDependency(input, candidate, request, metadata);
if (request.packageSubpath === ".") {
if (packageExportsEligible(input) && metadata.kind !== "absent")
return resolvePackageSpecifier(input, candidate, subpath, metadata);
if (subpath === ".") {
const direct = resolveFileCandidates(input, fileCandidates(candidate));
if (direct !== null) return direct;
}
if (hasContainerDirectory(input.files, candidate, source?.container_sha256))
return resolvePackageDependency(input, candidate, request, metadata);
return resolvePackageSpecifier(input, candidate, subpath, metadata);
if (directory === "." || directory === "") break;
directory = posix.dirname(directory);
}
@@ -345,25 +330,24 @@ const resolvePackageImportAlias = (
return unresolvedCandidate("external", [
`Wildcard package import aliases remain unresolved; package metadata ${metadata.path} has no exact imports key ${specifier}.`,
]);
const selected = selectExportTarget(
const selected = selectPackageImportTarget(
lookup.target,
packageExportConditions(input.moduleKind),
packageMetadataUrl(metadata.path),
"imports",
input.moduleKind,
metadata.path,
);
if (selected.kind === "invalid" || selected.kind === "invalid-config")
if (selected.status === "rejected")
return unresolvedCandidate("rejected", [
`Package metadata ${metadata.path}: ${selected.limitation}`,
]);
if (selected.kind === "unmatched")
if (selected.status === "unmatched")
return unresolvedCandidate("external", [
`Package metadata ${metadata.path} declares no imports target for alias ${specifier} under the active conditions; it declares ${selected.declared.join(", ") || "no conditions"}.`,
]);
if (selected.kind === "blocked")
if (selected.status === "blocked")
return unresolvedCandidate("external", [
`Package metadata ${metadata.path} blocks import alias ${specifier} with ${selected.cause === "null" ? "an explicit null imports target" : "an empty imports target array"}.`,
]);
if (selected.kind === "non-relative") {
if (selected.status === "non-relative") {
const target = classifyPackageImportTargetString(selected.value);
return unresolvedCandidate("external", [
target.kind === "recursive"
@@ -374,44 +358,77 @@ const resolvePackageImportAlias = (
return resolveExportTarget(input, metadata.path, selected, "imports");
};
const selectPackageImportTarget = (
value: unknown,
moduleKind: ResolveArtifactPathInput["moduleKind"],
packagePath: string,
):
| ({ readonly status: "value" } & PackageFileTarget)
| { readonly status: "non-relative"; readonly value: string }
| { readonly status: "blocked"; readonly cause: PackageTargetBlockCause }
| { readonly status: "unmatched"; readonly declared: readonly string[] }
| { readonly status: "rejected"; readonly limitation: string } => {
const selected = selectExportTarget(
value,
packageExportConditions(moduleKind),
packageMetadataUrl(packagePath),
"imports",
);
if (selected.kind === "invalid" || selected.kind === "invalid-config")
return { status: "rejected", limitation: selected.limitation };
if (selected.kind === "unmatched")
return {
status: "unmatched",
declared: selected.declared,
};
if (selected.kind === "blocked")
return { status: "blocked", cause: selected.cause };
if (selected.kind === "non-relative")
return { status: "non-relative", value: selected.value };
return {
status: "value",
value: selected.value,
url: selected.url,
};
};
/** Self references use the nearest package scope before dependency lookup. */
const packageSelfCandidate = (
const commonJsSelfCandidate = (
input: ResolveArtifactPathInput,
request: BarePackageRequest,
declared: string,
): CandidateResolution | null => {
if (input.moduleKind === undefined || !input.files.has(input.sourcePath))
if (input.moduleKind !== "require" || !input.files.has(input.sourcePath))
return null;
const metadata = enclosingPackageMetadata(input);
if (metadata.kind === "none") return null;
if (metadata.kind === "unavailable")
return unresolvedCandidate("unavailable", [metadata.limitation]);
if (metadata.name === undefined || metadata.exports === undefined)
if (
metadata.name === undefined ||
metadata.exports === undefined ||
(declared !== metadata.name && !declared.startsWith(`${metadata.name}/`))
)
return null;
let subpath: string;
if (input.moduleKind === "import") {
if (metadata.name !== request.packageName) return null;
subpath = request.packageSubpath;
} else if (input.declaredPath === metadata.name) {
subpath = ".";
} else if (input.declaredPath.startsWith(`${metadata.name}/`)) {
subpath = `.${input.declaredPath.slice(metadata.name.length)}`;
} else return null;
const subpath =
declared === metadata.name
? "."
: `.${declared.slice(metadata.name.length)}`;
// Self resolution does not apply the CommonJS dependency-name matcher.
return resolvePackageExports(input, metadata.path, subpath, metadata.exports);
};
/** Resolve an installed dependency using one parsed request and its metadata. */
const resolvePackageDependency = (
/** Package roots and exact subpaths share metadata and exports selection. */
const resolvePackageSpecifier = (
input: ResolveArtifactPathInput,
directory: string,
request: BarePackageRequest,
subpath: string,
metadata: PackageMetadata,
): CandidateResolution => {
const subpath = request.packageSubpath;
if (request.dependencyExportsEligible && metadata.kind === "unavailable")
const exportsEligible = packageExportsEligible(input);
if (exportsEligible && metadata.kind === "unavailable")
return unresolvedCandidate("unavailable", [metadata.limitation]);
const rawExports =
request.dependencyExportsEligible && metadata.kind === "available"
exportsEligible && metadata.kind === "available"
? metadata.exports
: undefined;
if (rawExports === undefined) {
@@ -420,17 +437,8 @@ const resolvePackageDependency = (
resolveFileCandidates(input, fileCandidates(directory)) ??
resolveDirectory(input, directory, metadata, new Set())
);
// Resolve legacy ESM subpaths as URLs: suffixes identify the loaded module
// but do not belong to its filesystem path. Keep the declared specifier on
// the relationship; only derive the inventory lookup path here.
const fileSubpath =
input.moduleKind === "import"
? decodeModuleUrlPath(stripQueryAndFragment(subpath.slice(2)))
: subpath.slice(2);
if (typeof fileSubpath !== "string") return fileSubpath;
const candidate = confineCandidate(`${directory}/${fileSubpath}`);
if (input.moduleKind === "import" && typeof candidate === "string")
return resolveLegacyEsmSubpath(input, candidate);
// Preserve the trailing slash for a package directory request.
const candidate = confineCandidate(`${directory}/${subpath.slice(2)}`);
return typeof candidate === "string"
? resolveCandidate(input, candidate, new Set())
: candidate;
@@ -443,21 +451,6 @@ const resolvePackageDependency = (
);
};
/** Legacy ESM package subpaths are exact URLs, never extension or directory lookups. */
const resolveLegacyEsmSubpath = (
input: ResolveArtifactPathInput,
candidate: string,
): CandidateResolution => {
const file = resolveFileCandidates(input, [candidate]);
if (file !== null) return file;
const source = input.files.get(input.sourcePath);
if (hasContainerDirectory(input.files, candidate, source?.container_sha256))
return unresolvedCandidate("rejected", [
`ESM package subpath ${candidate} names a directory; directory imports are unsupported and package main or index lookup was not attempted.`,
]);
return exactPackageTargetNotFound(candidate, "ESM subpath");
};
const resolvePackageExports = (
input: ResolveArtifactPathInput,
packagePath: string,
@@ -524,36 +517,23 @@ const resolveExportTarget = (
);
};
/** Derived package identity and dependency policy; input retains the raw request. */
interface BarePackageRequest {
readonly packageName: string;
readonly packageSubpath: string;
readonly dependencyExportsEligible: boolean;
}
const parseBarePackageRequest = (
specifier: string,
moduleKind: ResolveArtifactPathInput["moduleKind"],
): BarePackageRequest | null => {
const barePackageName = (specifier: string): string | null => {
const segments = specifier.split("/");
let packageName = segments[0] ?? "";
if (packageName === "") return null;
if (specifier.startsWith("@")) {
const packageSegment = segments[1];
if (packageSegment === undefined || packageSegment === "") return null;
packageName = `${packageName}/${packageSegment}`;
}
if (specifier.startsWith("@"))
return segments.length >= 2 && segments[0] !== "" && segments[1] !== ""
? `${segments[0]}/${segments[1]}`
: null;
return segments[0] === "" ? null : (segments[0] ?? null);
};
const packageExportsEligible = (input: ResolveArtifactPathInput): boolean => {
if (input.moduleKind !== "require") return true;
// Node's CommonJS exports matcher excludes percent-containing package names.
// Compare the captured name so a scoped name cannot fall back to its scope.
const dependencyExportsEligible =
moduleKind !== "require" ||
/^((?:@[^/\\%]+\/)?[^./\\%][^/\\%]*)(\/.*)?$/.exec(specifier)?.[1] ===
packageName;
return {
packageName,
packageSubpath: `.${specifier.slice(packageName.length)}`,
dependencyExportsEligible,
};
const match = /^((?:@[^/\\%]+\/)?[^./\\%][^/\\%]*)(\/.*)?$/.exec(
input.declaredPath,
);
return match?.[1] === barePackageName(input.declaredPath);
};
const hasContainerDirectory = (
@@ -1081,12 +1061,12 @@ const notFoundCandidate = (): CandidateResolution => ({
const exactPackageTargetNotFound = (
path: string,
targetKind: "exports" | "imports" | "ESM subpath",
mapKind: "exports" | "imports",
): CandidateResolution => ({
resolvedPath: null,
status: "not-found",
limitations: [
`The selected package ${targetKind === "imports" ? "import" : targetKind} target ${path} does not exist in the inventoried artifact container; ${targetKind === "imports" ? "import" : targetKind} targets are exact files, so no extension or directory index was tried.`,
`The selected package ${mapKind === "imports" ? "import" : "exports"} target ${path} does not exist in the inventoried artifact container; ${mapKind === "imports" ? "import" : "exports"} targets are exact files, so no extension or directory index was tried.`,
],
});
@@ -11,7 +11,6 @@ import {
import { createJavaScriptArtifactReader as createReader } from "../../artifacts/javascript/JavaScriptArtifactReader.js";
import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js";
import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js";
import type { JavaScriptSourceSelection } from "../../domain/javascript/javascriptSourceSelection.js";
import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { parseOwnedJavaScriptApplicationAnalysisSteps } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
@@ -46,7 +45,6 @@ export interface JavaScriptArtifactReconstructionResult {
readonly inventory_manifest_id: string;
readonly inventory_graph_sha256: string;
readonly integrity_contradictions: ArtifactInventorySnapshot["integrity_contradictions"];
readonly source_selection?: JavaScriptSourceSelection;
readonly graph: JavaScriptApplicationGraph;
readonly semantic_graph: JavaScriptSemanticGraph;
readonly electron_summary: ElectronBoundarySummary;
@@ -121,7 +119,6 @@ export const reconstructJavaScriptArtifact = async (
snapshot,
scope,
signal,
input.source_paths,
);
// Release acquisition before analysis consumes these owned file facts.
const earlyCleanup = await scope.release(readerOwner);
@@ -137,7 +134,7 @@ export const reconstructJavaScriptArtifact = async (
);
await reportPhase(
"parse_javascript_sources",
`Parsing and projecting ${String(files.files.length - (files.source_selection?.unselected_files ?? 0))} selected application source files`,
`Parsing and projecting ${String(files.files.length)} application source files`,
);
const semanticProjection = createJavaScriptSemanticGraphProjection();
const worker = createJavaScriptSourceAnalysis(
@@ -240,9 +237,6 @@ export const reconstructJavaScriptArtifact = async (
inventory_manifest_id: snapshot.manifest.manifest_id,
inventory_graph_sha256: snapshot.manifest.graph_sha256,
integrity_contradictions: snapshot.integrity_contradictions,
...(files.source_selection === undefined
? {}
: { source_selection: files.source_selection }),
graph,
semantic_graph: semanticGraph,
electron_summary: summarizeElectronBoundaries(analysis),
@@ -1,14 +1,12 @@
import { z } from "zod";
import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js";
import { javaScriptAnalysisResourceControlsSchema } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
import { javaScriptSourcePathsSchema } from "../../domain/javascript/javascriptSourceSelection.js";
/** Local ASAR/directory reconstruction request. */
export const javascriptArtifactReconstructionInputSchema = z.strictObject({
input_path: z.string().min(1),
format: z.enum(["auto", "asar", "directory"]).default("auto"),
integrity_policy: artifactIntegrityPolicySchema,
source_paths: javaScriptSourcePathsSchema.optional(),
...javaScriptAnalysisResourceControlsSchema.shape,
});
@@ -5,10 +5,10 @@ import { fileURLToPath, pathToFileURL } from "node:url";
import { parseExpression } from "@babel/parser";
import { describe, expect, it } from "vitest";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { staticPath } from "../../../../src/domain/javascript/javascriptStaticAnalysisHelpers.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { staticPath } from "../../domain/javascript/javascriptStaticAnalysisHelpers.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
const urlCases = [
["./preload%20name.cjs", "preload name.cjs", "main.mjs"],
@@ -4,11 +4,11 @@ import { mkdir, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { promisify } from "node:util";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import type { JavaScriptArtifactFile } from "../../../../src/domain/javascript/javascriptArtifactFiles.js";
import { resolveArtifactPathByContext } from "../../../../src/application/javascript/JavaScriptArtifactPathResolution.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import type { JavaScriptArtifactFile } from "../../domain/javascript/javascriptArtifactFiles.js";
import { resolveArtifactPathByContext } from "./JavaScriptArtifactPathResolution.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
const filesFor = (paths: readonly string[]) =>
new Map<string, JavaScriptArtifactFile>(
@@ -3,9 +3,9 @@ import { join } from "node:path";
import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../support/javascriptApplicationScope.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { javascriptApplicationAnalysisResultSchema } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { analyzeJavaScriptApplication } from "../../../tests/support/javascriptApplicationScope.js";
it("classifies overloaded open calls from lexical receiver facts", async () => {
const inputPath = await createTestTempDirectory(
@@ -130,16 +130,6 @@ export const buildJavaScriptSemanticGraph = ({
for (const analyzed of analysis.files) {
if (analyzed.semantic !== null)
projection.projectFile(analyzed.file, analyzed.semantic.ir);
else if (
!analyzed.file.text.included &&
analyzed.file.text.reason === "outside-analysis-scope"
)
projection.recordUnavailableFile(
analyzed.file,
"Source was not analyzed because it is outside the caller-selected source_paths.",
null,
"incomplete-module",
);
}
return projection.finish(rootArtifactSha256, applicationGraph);
};
@@ -73,11 +73,11 @@ describe("JebAnalysisService admission", () => {
const requests: JebRequest[] = [];
const service = new JebAnalysisService(fakePort(requests));
const result = await service.execute("list_jeb_units", {});
if (!result.ok) throw result.error;
expect(requests).toHaveLength(1);
expect(result.ok).toBe(true);
if (!result.ok) return;
expect(requests[0]?.operation).toBe("list_jeb_units");
const request = requests[0];
if (request?.operation !== "list_jeb_units")
throw new Error("Expected the list_jeb_units provider request");
if (request?.operation !== "list_jeb_units") return;
expect(request.input.index).toBe(0);
expect(request.input.count).toBe(100);
expect(result.value.normalized_result).toEqual({ observed: true });
@@ -1,13 +1,24 @@
import { writeFile } from "node:fs/promises";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { afterEach, describe, expect, it } from "vitest";
import {
captureProcessScenarioFile,
projectProcessCliError,
} from "../../../src/application/process/ProcessCli.js";
import { JSON_BYTE_ORDER_MARK_MESSAGE } from "../../../src/application/Utf8JsonInput.js";
} from "./ProcessCli.js";
import { JSON_BYTE_ORDER_MARK_MESSAGE } from "../Utf8JsonInput.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
const roots: string[] = [];
afterEach(async () => {
await Promise.all(
roots.splice(0).map((root) => rm(root, { recursive: true, force: true })),
);
});
const fixture = async (): Promise<string> => {
const root = await mkdtemp(join(tmpdir(), "rea-process-cli-diagnostics-"));
roots.push(root);
return root;
};
describe("process CLI input diagnostics", () => {
it("preserves local diagnostics for unexpected process failures", () => {
@@ -24,7 +35,7 @@ describe("process CLI input diagnostics", () => {
});
it("projects missing, malformed JSON, and invalid scenario files", async () => {
const root = await createTestTempDirectory("rea-process-cli-diagnostics-");
const root = await fixture();
const malformed = join(root, "malformed.json");
const invalid = join(root, "invalid.json");
const invalidUtf8 = join(root, "invalid-utf8.json");
@@ -79,10 +90,7 @@ describe("process CLI input diagnostics", () => {
});
it("names a leading byte-order mark instead of quoting it as a token", async () => {
const path = join(
await createTestTempDirectory("rea-process-cli-diagnostics-"),
"scenario.json",
);
const path = join(await fixture(), "scenario.json");
await writeFile(path, '\uFEFF{"executable":"/bin/echo"}');
expect(await captureProcessScenarioFile(path)).toEqual({
error: "Process command failed",
+3 -31
View File
@@ -26,35 +26,11 @@ export interface ArtifactEntry {
readonly adapterKey: string;
/** Filesystem identity captured during traversal, when supplied by an adapter. */
readonly sourceIdentity?: {
readonly device: bigint;
readonly inode: bigint;
readonly size: bigint;
readonly mode: bigint;
readonly mtimeNs: bigint;
readonly ctimeNs: bigint;
readonly device: number;
readonly inode: number;
};
}
/** Compare captured filesystem identity and mutable metadata at a read boundary. */
export const sameArtifactSourceIdentity = (
expected: ArtifactEntry["sourceIdentity"],
observed: {
readonly dev: bigint;
readonly ino: bigint;
readonly size: bigint;
readonly mode: bigint;
readonly mtimeNs: bigint;
readonly ctimeNs: bigint;
},
): boolean =>
expected !== undefined &&
observed.dev === expected.device &&
observed.ino === expected.inode &&
observed.size === expected.size &&
observed.mode === expected.mode &&
observed.mtimeNs === expected.mtimeNs &&
observed.ctimeNs === expected.ctimeNs;
/** Snapshot mutable entry metadata before the producer later validates open(). */
export const copyArtifactEntry = (entry: ArtifactEntry): ArtifactEntry => ({
...entry,
@@ -84,11 +60,7 @@ export const sameArtifactEntry = (
(limitation, index) => limitation === produced.limitations[index],
) &&
entry.sourceIdentity?.device === produced.sourceIdentity?.device &&
entry.sourceIdentity?.inode === produced.sourceIdentity?.inode &&
entry.sourceIdentity?.size === produced.sourceIdentity?.size &&
entry.sourceIdentity?.mode === produced.sourceIdentity?.mode &&
entry.sourceIdentity?.mtimeNs === produced.sourceIdentity?.mtimeNs &&
entry.sourceIdentity?.ctimeNs === produced.sourceIdentity?.ctimeNs;
entry.sourceIdentity?.inode === produced.sourceIdentity?.inode;
/** Read-only adapter over one directory, archive, or virtual container. */
export interface ArtifactReader {
+110 -238
View File
@@ -1,4 +1,10 @@
import { constants, createWriteStream, type Stats } from "node:fs";
import {
constants,
createReadStream,
createWriteStream,
type ReadStream,
type Stats,
} from "node:fs";
import {
lstat,
mkdtemp,
@@ -40,40 +46,51 @@ export class AsarArtifactReader implements ArtifactReader {
#archiveSize: number | undefined;
#headerSize: number | undefined;
readonly #entries = new Map<string, AsarEntryState>();
readonly #ownedHandles = new Map<SharedAsarFileHandle, OwnedAsarHandle>();
readonly #unpackedHandles = new Map<
UnpackedFileHandle,
{
readonly path: string;
readonly owner: OwnedFileHandle<UnpackedFileHandle>;
}
>();
#snapshot: { readonly path: string; readonly sha256: string } | undefined;
#snapshotRoot: string | undefined;
#snapshotOperation: Promise<void> = Promise.resolve();
constructor(
private readonly path: string,
private readonly options: AsarArtifactReaderOptions = {},
private readonly openUnpackedFile: OpenAsarUnpackedFile = open,
private readonly removeSnapshot: (path: string) => Promise<void> = (path) =>
rm(path, { recursive: true, force: true }),
) {}
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
let paths: readonly AsarListedPath[];
try {
await this.prepareContainer(undefined, signal);
paths = await this.#withSnapshotOperation(async () => {
const snapshotPath = this.#snapshotPath();
this.#resetArchiveState();
this.#entries.clear();
const admitted = await this.#admitHeader(snapshotPath);
const snapshotPath = this.#snapshotPath();
this.#resetArchiveState();
this.#entries.clear();
const headerHandle = await open(
snapshotPath,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
try {
const admitted = await admitAsarHeader(headerHandle);
this.#archiveSize = admitted.archiveBytes;
this.#headerSize = admitted.headerBytes;
abortIfNeeded(signal);
const ordered = orderAsarListing(
listPackage(snapshotPath, { isPack: false }),
} finally {
await headerHandle.close();
}
abortIfNeeded(signal);
paths = orderAsarListing(listPackage(snapshotPath, { isPack: false }));
const archiveMetadata = await lstat(snapshotPath);
if (!archiveMetadata.isFile() || archiveMetadata.isSymbolicLink())
throw new ArtifactReaderFailure(
"format",
"ASAR container is not a regular file",
);
const archiveMetadata = await lstat(snapshotPath);
if (!archiveMetadata.isFile() || archiveMetadata.isSymbolicLink())
throw new ArtifactReaderFailure(
"format",
"ASAR container is not a regular file",
);
this.#archiveSize = archiveMetadata.size;
return ordered;
}, signal);
this.#archiveSize = archiveMetadata.size;
} catch (cause: unknown) {
throw asarFailure(this.path, "inventory", cause);
}
@@ -125,47 +142,7 @@ export class AsarArtifactReader implements ArtifactReader {
}
}
async #admitHeader(snapshotPath: string) {
const headerHandle = await this.#openOwned(
snapshotPath,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
let admission:
| {
readonly ok: true;
readonly value: Awaited<ReturnType<typeof admitAsarHeader>>;
}
| { readonly ok: false; readonly cause: unknown };
try {
admission = { ok: true, value: await admitAsarHeader(headerHandle) };
} catch (cause: unknown) {
admission = { ok: false, cause };
}
try {
await this.#closeOwnedHandle(headerHandle);
} catch (cleanupCause: unknown) {
if (!admission.ok)
throw ArtifactReaderFailure.withCleanup(
admission.cause,
ArtifactReaderFailure.cleanupObservation(cleanupCause, snapshotPath),
);
throw cleanupCause;
}
if (!admission.ok) throw admission.cause;
return admission.value;
}
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
return this.#withSnapshotOperation(
() => this.#openEntry(entry, signal),
signal,
);
}
async #openEntry(
entry: ArtifactEntry,
signal?: AbortSignal,
): Promise<Readable> {
abortIfNeeded(signal);
const state = this.#entries.get(entry.adapterKey);
if (state === undefined && entry.kind !== "file")
@@ -183,16 +160,10 @@ export class AsarArtifactReader implements ArtifactReader {
"integrity",
`ASAR entry metadata changed since inventory: ${state.entry.path}`,
);
if (state.entry.unpacked)
return this.#openUnpackedStream(state.entry, signal);
return this.#openPackedStream(state, signal);
}
async #openPackedStream(
state: AsarEntryState,
signal?: AbortSignal,
): Promise<Readable> {
const { metadata, entry: producedEntry } = state;
const { metadata } = state;
const producedEntry = state.entry;
if (producedEntry.unpacked)
return this.#openUnpackedStream(producedEntry, signal);
const archiveSize = this.#archiveSize;
const headerSize = this.#headerSize;
const offset =
@@ -216,11 +187,10 @@ export class AsarArtifactReader implements ArtifactReader {
"format",
`ASAR entry range is outside its container: ${producedEntry.path}`,
);
const snapshotPath = this.#snapshotPath();
let handle: SharedAsarFileHandle | undefined;
let handle: FileHandle | undefined;
try {
const openedHandle = await this.#openOwned(
snapshotPath,
const openedHandle = await open(
this.#snapshotPath(),
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
handle = openedHandle;
@@ -232,48 +202,25 @@ export class AsarArtifactReader implements ArtifactReader {
`ASAR container changed before read: ${producedEntry.path}`,
);
if (metadata.size === 0) {
await closeAsarHandle(() => openedHandle.close(), signal);
handle = undefined;
await closeAsarHandle(
() => this.#closeOwnedHandle(openedHandle),
signal,
);
return Readable.from([]);
}
const source = this.#ownedReadStream(openedHandle, {
const source = openedHandle.createReadStream({
start,
end: start + metadata.size - 1,
signal,
autoClose: true,
});
handle = undefined;
const output = readValidatedAsarEntry(
return readValidatedAsarEntry(
source,
metadata.size,
producedEntry.path,
this.path,
signal,
);
this.#setOwnedOutput(openedHandle, output);
return output;
} catch (cause: unknown) {
const failure = asarFailure(
this.path,
`read ${producedEntry.path}`,
cause,
);
if (handle !== undefined) {
try {
await this.#closeOwnedHandle(handle);
} catch (cleanupCause: unknown) {
throw ArtifactReaderFailure.withCleanup(
failure,
ArtifactReaderFailure.cleanupObservation(
cleanupCause,
snapshotPath,
),
);
}
}
throw failure;
await handle?.close().catch(() => undefined);
throw asarFailure(this.path, `read ${producedEntry.path}`, cause);
}
}
@@ -299,37 +246,49 @@ export class AsarArtifactReader implements ArtifactReader {
}
/** Read the same captured container that supplies headers and packed members. */
async openContainer(signal?: AbortSignal): Promise<Readable> {
async openContainer(signal?: AbortSignal): Promise<ReadStream> {
await this.prepareContainer(undefined, signal);
return this.#withSnapshotOperation(async () => {
abortIfNeeded(signal);
const snapshotPath = this.#snapshotPath();
const handle = await this.#openOwned(
snapshotPath,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
return this.#ownedReadStream(handle, { signal });
}, signal);
return createReadStream(
this.#snapshotPath(),
signal === undefined ? {} : { signal },
);
}
async close(): Promise<void> {
await this.#withSnapshotOperation(() => this.#closeSnapshot());
await this.#withSnapshotOperation(async () => {
let cleanupFailure: ArtifactReaderFailure | undefined;
for (const [handle, { path }] of this.#unpackedHandles) {
try {
await this.#closeUnpackedHandle(handle);
} catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup(
cleanupFailure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, path),
);
}
}
try {
await this.#closeSnapshot();
} catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup(
cleanupFailure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, this.path),
);
}
if (cleanupFailure !== undefined) throw cleanupFailure;
});
}
async #captureSnapshot(signal?: AbortSignal): Promise<void> {
let inputHandle: SharedAsarFileHandle | undefined;
try {
this.#snapshotRoot = await mkdtemp(join(tmpdir(), "rea-asar-snapshot-"));
const snapshotPath = join(this.#snapshotRoot, "container.asar");
const hash = createHash("sha256");
inputHandle = await this.#openOwned(
const handle = await open(
this.path,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
const source = this.#ownedReadStream(inputHandle, {
signal,
releaseOnClose: false,
});
const source = handle.createReadStream();
const digesting = new Transform({
transform(chunk: Buffer, _encoding, callback) {
hash.update(chunk);
@@ -342,50 +301,34 @@ export class AsarArtifactReader implements ArtifactReader {
createWriteStream(snapshotPath, { flags: "wx", mode: 0o600 }),
signal === undefined ? {} : { signal },
);
const completedInput = inputHandle;
inputHandle = undefined;
await this.#closeOwnedHandle(completedInput);
this.#snapshot = { path: snapshotPath, sha256: hash.digest("hex") };
} catch (cause: unknown) {
let failure =
const primary =
signal?.aborted === true
? new ArtifactReaderFailure("cancelled", "ASAR operation cancelled", {
cause,
})
: asarFailure(this.path, "snapshot", cause);
if (inputHandle !== undefined) {
const failedInput = inputHandle;
inputHandle = undefined;
try {
await this.#closeOwnedHandle(failedInput);
} catch (cleanupCause: unknown) {
failure = ArtifactReaderFailure.withCleanup(
failure,
ArtifactReaderFailure.cleanupObservation(cleanupCause, this.path),
);
}
}
try {
if (this.#ownedHandles.size === 0) await this.#closeSnapshot();
await this.#closeSnapshot();
} catch (cleanupCause: unknown) {
throw ArtifactReaderFailure.withCleanup(
failure,
primary,
ArtifactReaderFailure.cleanupObservation(
cleanupCause,
this.#snapshotRoot ?? this.path,
),
);
}
throw failure;
throw primary;
}
}
async #closeSnapshot(): Promise<void> {
await this.#closeOwnedHandles();
const root = this.#snapshotRoot;
if (root !== undefined) {
try {
await (this.options.removeSnapshot ?? defaultRemoveSnapshot)(root);
await this.removeSnapshot(root);
if (this.#snapshotRoot === root) this.#snapshotRoot = undefined;
} catch (cause: unknown) {
throw ArtifactReaderFailure.withCleanup(cause, {
@@ -443,7 +386,7 @@ export class AsarArtifactReader implements ArtifactReader {
signal?: AbortSignal,
): Promise<Readable> {
const unpackedRoot = `${this.path}.unpacked`;
let handle: SharedAsarFileHandle | undefined;
let handle: UnpackedFileHandle | undefined;
try {
const rootMetadata = await lstat(unpackedRoot);
if (!rootMetadata.isDirectory() || rootMetadata.isSymbolicLink())
@@ -470,12 +413,15 @@ export class AsarArtifactReader implements ArtifactReader {
"path",
`ASAR unpacked entry is not a regular file: ${entry.path}`,
);
handle = await (this.options.openUnpackedFile ?? open)(
handle = await this.openUnpackedFile(
canonical,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
const openedHandle = handle;
this.#adoptHandle(openedHandle, canonical);
this.#unpackedHandles.set(openedHandle, {
path: canonical,
owner: new OwnedFileHandle(openedHandle),
});
const openedMetadata = await openedHandle.stat();
if (
!openedMetadata.isFile() ||
@@ -495,28 +441,30 @@ export class AsarArtifactReader implements ArtifactReader {
);
}
abortIfNeeded(signal);
const source = this.#ownedReadStream(openedHandle, { start: 0 });
handle = undefined;
const output = readValidatedAsarEntry(
const source = readFileHandleChunks(openedHandle, {
start: 0,
});
source.once("close", () => {
void this.#closeUnpackedHandle(openedHandle).catch(() => undefined);
});
return readValidatedAsarEntry(
source,
undefined,
entry.path,
this.path,
signal,
);
this.#setOwnedOutput(openedHandle, output);
return output;
} catch (cause: unknown) {
let failure = asarUnpackedEntryFailure(this.path, entry, cause);
if (handle !== undefined && this.#ownedHandles.has(handle)) {
if (handle !== undefined && this.#unpackedHandles.has(handle)) {
try {
await this.#closeOwnedHandle(handle);
await this.#closeUnpackedHandle(handle);
} catch (cleanupCause: unknown) {
failure = ArtifactReaderFailure.withCleanup(
failure,
ArtifactReaderFailure.cleanupObservation(
cleanupCause,
this.#ownedHandles.get(handle)?.path ?? entry.path,
this.#unpackedHandles.get(handle)?.path ?? entry.path,
),
);
}
@@ -525,76 +473,11 @@ export class AsarArtifactReader implements ArtifactReader {
}
}
async #closeOwnedHandle(handle: SharedAsarFileHandle): Promise<void> {
const owned = this.#ownedHandles.get(handle);
async #closeUnpackedHandle(handle: UnpackedFileHandle): Promise<void> {
const owned = this.#unpackedHandles.get(handle);
if (owned === undefined) return;
owned.output?.destroy();
owned.source?.destroy();
try {
await owned.owner.close();
} catch (cause: unknown) {
throw ArtifactReaderFailure.withCleanup(
cause,
ArtifactReaderFailure.cleanupObservation(cause, owned.path),
);
}
this.#ownedHandles.delete(handle);
}
async #closeOwnedHandles(): Promise<void> {
let failure: ArtifactReaderFailure | undefined;
for (const [handle, owned] of this.#ownedHandles) {
try {
await this.#closeOwnedHandle(handle);
} catch (cause: unknown) {
failure = ArtifactReaderFailure.withCleanup(
failure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, owned.path),
);
}
}
if (failure !== undefined) throw failure;
}
async #openOwned(path: string, flags: number): Promise<SharedAsarFileHandle> {
const handle = await (this.options.openFile ?? open)(path, flags);
this.#adoptHandle(handle, path);
return handle;
}
#adoptHandle(handle: SharedAsarFileHandle, path: string): void {
this.#ownedHandles.set(handle, {
path,
owner: new OwnedFileHandle(handle),
});
}
#ownedReadStream(
handle: SharedAsarFileHandle,
options: {
readonly start?: number;
readonly end?: number;
readonly signal?: AbortSignal | undefined;
readonly releaseOnClose?: boolean;
},
): Readable {
const { releaseOnClose, ...readOptions } = options;
const source = readFileHandleChunks(handle, readOptions);
const owned = this.#ownedHandles.get(handle);
if (owned === undefined)
throw new ArtifactReaderFailure("integrity", "ASAR handle has no owner");
owned.source = source;
owned.output = source;
if (releaseOnClose !== false)
source.once("close", () => {
void this.#closeOwnedHandle(handle).catch(() => undefined);
});
return source;
}
#setOwnedOutput(handle: SharedAsarFileHandle, output: Readable): void {
const owned = this.#ownedHandles.get(handle);
if (owned !== undefined) owned.output = output;
await owned.owner.close();
this.#unpackedHandles.delete(handle);
}
#resetArchiveState(): void {
@@ -608,27 +491,16 @@ type AsarEntryState = {
readonly metadata: AsarFileMetadata;
readonly entry: ArtifactEntry;
};
type SharedAsarFileHandle = Pick<FileHandle, "fd" | "read" | "close"> & {
type UnpackedFileHandle = {
readonly fd: number;
stat(): Promise<Stats>;
close(): Promise<void>;
read: FileHandle["read"];
};
type OpenAsarFile = (
type OpenAsarUnpackedFile = (
path: string,
flags: number,
) => Promise<SharedAsarFileHandle>;
type OwnedAsarHandle = {
readonly path: string;
readonly owner: OwnedFileHandle<SharedAsarFileHandle>;
source?: Readable;
output?: Readable;
};
type AsarArtifactReaderOptions = {
readonly openFile?: OpenAsarFile;
readonly openUnpackedFile?: OpenAsarFile;
readonly removeSnapshot?: (path: string) => Promise<void>;
};
const defaultRemoveSnapshot = (path: string): Promise<void> =>
rm(path, { recursive: true, force: true });
) => Promise<UnpackedFileHandle>;
const asarUnpackedEntryFailure = (
archivePath: string,
+1 -2
View File
@@ -1,4 +1,3 @@
import type { Stats } from "node:fs";
import type { FileHandle } from "node:fs/promises";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
@@ -8,7 +7,7 @@ export const MAX_ASAR_HEADER_BYTES = 16 * 1024 * 1024;
/** Admit both Chromium pickle extents before an upstream decoder allocates them. */
export const admitAsarHeader = async (
handle: Pick<FileHandle, "read"> & { stat(): Promise<Stats> },
handle: FileHandle,
): Promise<{
readonly archiveBytes: number;
readonly headerBytes: number;
+20 -76
View File
@@ -1,4 +1,4 @@
import { constants, type BigIntStats } from "node:fs";
import { constants, type Stats } from "node:fs";
import {
lstat,
open,
@@ -11,7 +11,6 @@ import { Readable } from "node:stream";
import {
ArtifactReaderFailure,
sameArtifactSourceIdentity,
type ArtifactEntry,
type ArtifactReader,
} from "./ArtifactReader.js";
@@ -96,11 +95,9 @@ export class DirectoryArtifactReader implements ArtifactReader {
if (child === null) break;
abortIfNeeded(signal);
const absolute = join(directory, child.name);
const metadata = await lstat(absolute, { bigint: true }).catch(
(cause: unknown) => {
throw directoryIoFailure("inspect entry", absolute, cause);
},
);
const metadata = await lstat(absolute).catch((cause: unknown) => {
throw directoryIoFailure("inspect entry", absolute, cause);
});
const path = relative(root, absolute).split(sep).join("/");
const entry = artifactEntry(path, absolute, metadata);
if (entry === undefined) continue;
@@ -161,18 +158,18 @@ export class DirectoryArtifactReader implements ArtifactReader {
const owner = new OwnedFileHandle(fileHandle);
this.#files.set(owner, entry.adapterKey);
try {
const observed = await owner.handle
.stat({ bigint: true })
.catch((cause: unknown) => {
throw directoryIoFailure(
"inspect opened entry",
entry.adapterKey,
cause,
);
});
const observed = await owner.handle.stat().catch((cause: unknown) => {
throw directoryIoFailure(
"inspect opened entry",
entry.adapterKey,
cause,
);
});
if (
!observed.isFile() ||
!sameArtifactSourceIdentity(entry.sourceIdentity, observed)
entry.sourceIdentity === undefined ||
observed.dev !== entry.sourceIdentity.device ||
observed.ino !== entry.sourceIdentity.inode
) {
throw new ArtifactReaderFailure(
"integrity",
@@ -181,7 +178,7 @@ export class DirectoryArtifactReader implements ArtifactReader {
}
abortIfNeeded(signal);
const source = Readable.from(
mapStableFileRead(owner.handle, entry, signal),
mapFileReadCancellation(owner.handle, signal),
{ objectMode: false },
);
source.once("close", () => {
@@ -259,7 +256,7 @@ export class DirectoryArtifactReader implements ArtifactReader {
const artifactEntry = (
path: string,
absolute: string,
metadata: BigIntStats,
metadata: Stats,
): ArtifactEntry | undefined => {
const kind = metadata.isSymbolicLink()
? "symlink"
@@ -272,7 +269,7 @@ const artifactEntry = (
return {
path,
kind,
declaredSize: kind === "file" ? safeArtifactSize(metadata.size) : null,
declaredSize: kind === "file" ? metadata.size : null,
compressedSize: null,
executable: hostExecutableMarking(absolute, metadata),
encrypted: false,
@@ -286,24 +283,12 @@ const artifactEntry = (
adapterKey: absolute,
...(kind === "file"
? {
sourceIdentity: {
device: metadata.dev,
inode: metadata.ino,
size: metadata.size,
mode: metadata.mode,
mtimeNs: metadata.mtimeNs,
ctimeNs: metadata.ctimeNs,
},
sourceIdentity: { device: metadata.dev, inode: metadata.ino },
}
: {}),
};
};
const safeArtifactSize = (size: bigint): number | null => {
if (size < 0n || size > BigInt(Number.MAX_SAFE_INTEGER)) return null;
return Number(size);
};
const directoryIoFailure = (
operation: string,
path: string,
@@ -328,45 +313,16 @@ const directoryIoFailure = (
const failureMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const mapStableFileRead = async function* (
const mapFileReadCancellation = async function* (
handle: FileHandle,
entry: ArtifactEntry,
signal?: AbortSignal,
): AsyncGenerator<Buffer> {
let bytes = 0n;
try {
const expectedBytes = entry.sourceIdentity?.size;
if (expectedBytes === undefined)
throw new ArtifactReaderFailure(
"integrity",
`Directory entry has no captured file state: ${entry.path}`,
);
for await (const chunk of readFileHandleChunks(handle, {
// Include one extra byte so a growing file cannot make this read unbounded.
end: expectedBytes,
signal,
})) {
for await (const chunk of readFileHandleChunks(handle, { signal })) {
abortIfNeeded(signal);
bytes += BigInt(chunk.length);
yield chunk;
}
abortIfNeeded(signal);
const [opened, current] = await Promise.all([
handle.stat({ bigint: true }),
lstat(entry.adapterKey, { bigint: true }),
]);
abortIfNeeded(signal);
if (
!opened.isFile() ||
!current.isFile() ||
!sameArtifactSourceIdentity(entry.sourceIdentity, opened) ||
!sameArtifactSourceIdentity(entry.sourceIdentity, current) ||
bytes !== expectedBytes
)
throw new ArtifactReaderFailure(
"integrity",
`Directory entry changed during read: ${entry.path}`,
);
} catch (cause: unknown) {
if (signal?.aborted === true)
throw new ArtifactReaderFailure(
@@ -374,18 +330,6 @@ const mapStableFileRead = async function* (
"Directory artifact read cancelled",
{ cause },
);
if (
cause instanceof Error &&
"code" in cause &&
["ENOENT", "ENOTDIR", "ELOOP"].includes(String(cause.code))
)
throw new ArtifactReaderFailure(
"integrity",
`Directory entry changed during read: ${entry.path}`,
{ cause },
);
if (!(cause instanceof ArtifactReaderFailure))
throw directoryIoFailure("validate read entry", entry.adapterKey, cause);
throw cause;
}
};
+31 -84
View File
@@ -125,7 +125,6 @@ export class NativeDmgArtifactReader implements ArtifactReader {
#ownershipUncertainty: string | undefined;
#closed = false;
#closePromise: Promise<void> | undefined;
readonly #activeDirectoryOperations = new Set<Promise<unknown>>();
constructor(
private readonly path: string,
@@ -142,30 +141,18 @@ export class NativeDmgArtifactReader implements ArtifactReader {
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
await this.#ensureAttached(signal);
const directory = this.#directory;
if (directory === undefined)
if (this.#directory === undefined)
throw new ArtifactReaderFailure("unavailable", "DMG is not attached");
const prefix = basename(this.path);
const entries = directory.entries(signal)[Symbol.asyncIterator]();
try {
for (;;) {
if (this.#closed) return;
const next = await this.#trackDirectoryOperation(() => entries.next());
if (this.#closed) return;
if (next.done) return;
yield { ...next.value, path: `${prefix}/${next.value.path}` };
}
} finally {
await entries.return?.();
}
for await (const entry of this.#directory.entries(signal))
yield { ...entry, path: `${prefix}/${entry.path}` };
}
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
await this.#ensureAttached(signal);
const directory = this.#directory;
if (directory === undefined)
if (this.#directory === undefined)
throw new ArtifactReaderFailure("unavailable", "DMG is not attached");
return this.#trackDirectoryOperation(() => directory.open(entry, signal));
return this.#directory.open(entry, signal);
}
provenance(): readonly ArtifactCommand[] {
@@ -185,9 +172,32 @@ export class NativeDmgArtifactReader implements ArtifactReader {
async #closeOwnedResources(): Promise<void> {
await this.#initialization?.catch(() => undefined);
await Promise.allSettled(this.#activeDirectoryOperations);
await this.#closeMountedDirectory();
let detachFailure = await this.#detachOwnedDevices();
let detachFailure = await this.#reconcileAttachState();
let remainingDevices: string[] = [];
for (const device of [...this.#devices].reverse()) {
const failure = await this.#detach(device);
if (failure !== undefined) {
remainingDevices.push(device);
detachFailure ??= failure;
}
}
// APFS container detach may briefly leave its backing image listed.
for (const delayMs of DETACH_SETTLE_DELAYS_MS) {
if (remainingDevices.length === 0) break;
await (this.host.delay ?? delay)(delayMs);
const stillAttached: string[] = [];
for (const device of remainingDevices) {
if (!(await this.#isAttached(device))) continue;
const failure = await this.#detach(device);
if (failure !== undefined) {
stillAttached.push(device);
detachFailure = failure;
}
}
remainingDevices = stillAttached;
}
if (remainingDevices.length === 0) detachFailure = undefined;
this.#devices = remainingDevices.reverse();
if (
this.#devices.length === 0 &&
this.#mountRoot !== undefined &&
@@ -232,55 +242,6 @@ export class NativeDmgArtifactReader implements ArtifactReader {
);
}
async #detachOwnedDevices(): Promise<unknown> {
let detachFailure = await this.#reconcileAttachState();
let remainingDevices: string[] = [];
for (const device of [...this.#devices].reverse()) {
const failure = await this.#detach(device);
if (failure !== undefined) {
remainingDevices.push(device);
detachFailure ??= failure;
}
}
// APFS container detach may briefly leave its backing image listed.
for (const delayMs of DETACH_SETTLE_DELAYS_MS) {
if (remainingDevices.length === 0) break;
await (this.host.delay ?? delay)(delayMs);
const stillAttached: string[] = [];
for (const device of remainingDevices) {
if (!(await this.#isAttached(device))) continue;
const failure = await this.#detach(device);
if (failure !== undefined) {
stillAttached.push(device);
detachFailure = failure;
}
}
remainingDevices = stillAttached;
}
if (remainingDevices.length === 0) detachFailure = undefined;
this.#devices = remainingDevices.reverse();
return detachFailure;
}
async #closeMountedDirectory(): Promise<void> {
if (this.#directory !== undefined) {
try {
await this.#directory.close();
this.#directory = undefined;
} catch (cause: unknown) {
const mountRoot = this.#mountRoot ?? this.path;
throw ArtifactReaderFailure.withCleanup(cause, {
reason: `DMG mounted directory remains owned: ${failureMessage(cause)}`,
resources: [
`DMG mounted directory at ${mountRoot}`,
`DMG mount root ${mountRoot}`,
...this.#devices.map((device) => `DMG device ${device}`),
],
});
}
}
}
/** Detach one device; return the failure only while it remains attached. */
async #detach(device: string): Promise<unknown> {
try {
@@ -333,20 +294,6 @@ export class NativeDmgArtifactReader implements ArtifactReader {
throw new ArtifactReaderFailure("unavailable", "DMG reader is closed");
}
async #trackDirectoryOperation<Result>(
operation: () => Promise<Result>,
): Promise<Result> {
if (this.#closed)
throw new ArtifactReaderFailure("unavailable", "DMG reader is closed");
const running = Promise.resolve().then(operation);
this.#activeDirectoryOperations.add(running);
try {
return await running;
} finally {
this.#activeDirectoryOperations.delete(running);
}
}
async #attach(signal?: AbortSignal): Promise<void> {
await runChecked(this.host, ["verify", this.path], signal);
this.#provenance.push(command(["verify", this.path], ["read"]));
@@ -1,10 +1,7 @@
import { createHash } from "node:crypto";
import { lstat } from "node:fs/promises";
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
import {
ArtifactReaderFailure,
sameArtifactSourceIdentity,
} from "../ArtifactReader.js";
import { ArtifactReaderFailure } from "../ArtifactReader.js";
import { analyzeInterfaceBuilderBundle } from "./InterfaceBuilderAnalysis.js";
import {
appleAssetCatalogInputSchema,
@@ -139,20 +136,22 @@ const inspectAssetCatalogEntry = async (input: {
}) => {
const { entry, reader, signal } = input;
const sha256 = await digestBounded(reader, entry, signal);
const before = await lstat(entry.adapterKey, { bigint: true });
const before = await lstat(entry.adapterKey);
if (
!before.isFile() ||
!sameArtifactSourceIdentity(entry.sourceIdentity, before)
entry.sourceIdentity === undefined ||
before.dev !== entry.sourceIdentity.device ||
before.ino !== entry.sourceIdentity.inode
)
throw new ArtifactReaderFailure(
"integrity",
`Asset catalog changed before inspection: ${entry.path}`,
);
const output = await input.runAssetUtil(entry.adapterKey, signal);
const after = await lstat(entry.adapterKey, { bigint: true });
const after = await lstat(entry.adapterKey);
if (
!after.isFile() ||
!sameArtifactSourceIdentity(entry.sourceIdentity, after) ||
before.dev !== after.dev ||
before.ino !== after.ino ||
before.size !== after.size ||
(await digestBounded(reader, entry, signal)) !== sha256
)
throw new ArtifactReaderFailure(
+3 -5
View File
@@ -1,4 +1,4 @@
import type { BigIntStats, Stats } from "node:fs";
import type { Stats } from "node:fs";
import { extname } from "node:path";
/** Extensions Windows launches directly through CreateProcess or cmd.exe. */
@@ -13,12 +13,10 @@ const windowsLaunchableExtensions = new Set([".bat", ".cmd", ".com", ".exe"]);
*/
export const hostExecutableMarking = (
path: string,
metadata: Pick<Stats | BigIntStats, "mode" | "isFile">,
metadata: Pick<Stats, "mode" | "isFile">,
platform: NodeJS.Platform = process.platform,
): boolean =>
platform === "win32"
? metadata.isFile() &&
windowsLaunchableExtensions.has(extname(path).toLowerCase())
: typeof metadata.mode === "bigint"
? (metadata.mode & 0o111n) !== 0n
: (metadata.mode & 0o111) !== 0;
: (metadata.mode & 0o111) !== 0;
@@ -18,7 +18,6 @@ import { streamChunkToBuffer } from "../StreamBytes.js";
import { hashReadable } from "../ArtifactHash.js";
import type { ArtifactOccurrence } from "../../domain/artifactGraph.js";
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
import type {
JavaScriptArtifactFileKind,
@@ -48,7 +47,6 @@ interface ReadContext {
readonly files: JavaScriptArtifactFile[];
readonly containers: JavaScriptArtifactContainer[];
readonly signal: AbortSignal | undefined;
readonly selectedPaths: ReadonlyMap<string, number> | undefined;
textBytes: number;
invalidUtf8: number;
}
@@ -59,49 +57,13 @@ interface ReadTextInput {
readonly expected: ExpectedFile;
}
/** Verify inventoried entries; acquire text only inside the caller's scope. */
/** Read all relevant textual entries through an already-inventoried reader. */
export const readJavaScriptArtifactFiles = async (
reader: ArtifactReader,
snapshot: ArtifactInventorySnapshot,
resources: ArtifactResourceScope,
signal?: AbortSignal,
sourcePaths?: readonly string[],
): Promise<JavaScriptArtifactFileSet> => {
const inventory = expectedInventory(snapshot);
const expected = inventory.files;
const selectedPaths =
sourcePaths === undefined
? undefined
: new Map([...new Set(sourcePaths)].map((path, index) => [path, index]));
const issues = (sourcePaths ?? []).flatMap((path, index) =>
expected.has(path) || inventory.unavailableFiles.has(path)
? []
: [
{
path: ["source_paths", index],
reason: "invalid_value" as const,
message:
"Select an exact relevant file path from the artifact inventory; no globs or path aliases.",
},
],
);
if (issues.length > 0)
throw new AnalysisInputError(
"analyze_javascript_application",
undefined,
issues,
);
const unavailablePath = sourcePaths?.find((path) =>
inventory.unavailableFiles.has(path),
);
if (unavailablePath !== undefined)
throw new ArtifactReaderFailure(
"unavailable",
`Selected inventoried source bytes are unavailable: ${unavailablePath}`,
{
partialObservation: { kind: "artifact-inventory", inventory: snapshot },
},
);
if (reader instanceof AsarArtifactReader) {
const root = snapshot.nodes.find(
({ artifact_id }) => artifact_id === snapshot.manifest.root_artifact_id,
@@ -117,6 +79,8 @@ export const readJavaScriptArtifactFiles = async (
signal,
);
}
const inventory = expectedInventory(snapshot);
const expected = inventory.files;
const context: ReadContext = {
resources,
expected,
@@ -125,17 +89,13 @@ export const readJavaScriptArtifactFiles = async (
files: [],
containers: [],
signal,
selectedPaths,
textBytes: 0,
invalidUtf8: 0,
};
await visitReader(reader, "", snapshot.manifest.root_sha256, context);
const files = context.files.sort((left, right) => {
const leftPriority = selectedPaths?.get(left.path) ?? Infinity;
const rightPriority = selectedPaths?.get(right.path) ?? Infinity;
if (leftPriority !== rightPriority) return leftPriority - rightPriority;
return compareUnicodeCodePoints(left.path, right.path);
});
const files = context.files.sort((left, right) =>
compareUnicodeCodePoints(left.path, right.path),
);
assertExpectedFilesWereVisited(expected, files);
return {
files,
@@ -144,17 +104,6 @@ export const readJavaScriptArtifactFiles = async (
),
text_bytes_read: context.textBytes,
invalid_utf8_files: context.invalidUtf8,
...(sourcePaths === undefined
? {}
: {
source_selection: {
requested_paths: [...sourcePaths],
unselected_files: files.filter(
({ text }) =>
!text.included && text.reason === "outside-analysis-scope",
).length,
},
}),
};
};
@@ -302,19 +251,6 @@ const readText = async (
context: ReadContext,
input: ReadTextInput,
): Promise<JavaScriptArtifactFile["text"]> => {
if (
context.selectedPaths !== undefined &&
!context.selectedPaths.has(input.expected.path)
) {
// Keep the inventory/acquisition identity check without allocating or
// decoding an unrelated source's text.
await verifyEntryBytes(
await input.reader.open(input.entry, context.signal),
input.expected,
context.signal,
);
return { included: false, reason: "outside-analysis-scope" };
}
if (input.expected.kind === "native-addon") {
await verifyEntryBytes(
await input.reader.open(input.entry, context.signal),
@@ -375,23 +311,16 @@ const expectedInventory = (
): {
readonly files: ReadonlyMap<string, ExpectedFile>;
readonly containers: ReadonlyMap<string, ExpectedContainer>;
readonly unavailableFiles: ReadonlySet<string>;
} => {
const nodes = new Map(snapshot.nodes.map((node) => [node.artifact_id, node]));
const files = new Map<string, ExpectedFile>();
const containers = new Map<string, ExpectedContainer>();
const unavailableFiles = new Set<string>();
for (const occurrence of snapshot.occurrences) {
if (occurrence.logical_path === ".") continue;
const isNestedAsar =
occurrence.entry_kind === "file" &&
occurrence.logical_path.toLowerCase().endsWith(".asar");
if (occurrence.artifact_id === null) {
if (
occurrence.entry_kind === "file" &&
relevantKind(occurrence.logical_path) !== undefined
)
unavailableFiles.add(occurrence.logical_path);
if (isNestedAsar)
containers.set(occurrence.logical_path, {
hash_status: occurrence.hash_status,
@@ -429,7 +358,7 @@ const expectedInventory = (
kind,
});
}
return { files, containers, unavailableFiles };
return { files, containers };
};
const readAll = async (
+62
View File
@@ -0,0 +1,62 @@
import {
lstat,
mkdtemp,
open,
rm,
symlink,
unlink,
writeFile,
} from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { expect, it } from "vitest";
import { readStableArtifact } from "./readStableArtifact.js";
const checkChange = async (
stage: "before-open" | "after-read" | "symlink-replacement",
): Promise<void> => {
const root = await mkdtemp(join(tmpdir(), "rea-stable-read-"));
const path = join(root, "selected");
await writeFile(path, "original evidence");
let inspected = 0;
try {
const reading = readStableArtifact(path, 1024, undefined, {
async lstat(selected) {
inspected++;
if (stage === "after-read" && inspected === 2) await unlink(selected);
const stat = await lstat(selected, { bigint: true });
if (inspected === 1 && stage !== "after-read") {
await unlink(selected);
if (stage === "symlink-replacement")
await symlink("missing", selected);
}
return stat;
},
open,
});
await expect(reading).rejects.toMatchObject({ reason: "integrity" });
} finally {
await rm(root, { recursive: true, force: true });
}
};
it.each(["before-open", "after-read"] as const)(
"retains an actual post-validation %s change as integrity failure",
checkChange,
);
it.skipIf(process.platform === "win32")(
"retains a post-validation symlink replacement as an integrity failure",
() => checkChange("symlink-replacement"),
);
it("keeps initially missing inputs distinguishable from later changes", async () => {
const root = await mkdtemp(join(tmpdir(), "rea-stable-missing-"));
try {
await expect(
readStableArtifact(join(root, "missing"), 1024),
).rejects.toMatchObject({ code: "ENOENT" });
} finally {
await rm(root, { recursive: true, force: true });
}
});
+15 -29
View File
@@ -226,17 +226,10 @@ export const handleRequestWillBeSent = (
preserveMalformedRedirectEvidence(state, requestId, previous)
)
return;
const method = cdpStringValue(request?.method);
const initiator = recordValue(params.initiator);
const initiatorType = cdpStringValue(initiator?.type);
if (
request === undefined ||
method === undefined ||
method.length === 0 ||
initiatorType === undefined ||
initiatorType.length === 0
) {
preserveMalformedRedirectEvidence(state, requestId, previous);
if (request === undefined) {
state.completeness.exclude("network_requests", "invalid_protocol_value");
state.network.delete(requestId);
state.networkRequestTimestamps.delete(requestId);
return;
}
const sanitized = allowedSanitizedUrl(request.url, state.allowedOrigins);
@@ -280,12 +273,6 @@ export const handleRequestWillBeSent = (
state.networkRequestTimestamps.delete(requestId);
return;
} else {
const encodedDataLength = numberValue(redirectResponse.encodedDataLength);
if (encodedDataLength === undefined || encodedDataLength < 0)
state.completeness.exclude(
"network_requests",
"invalid_protocol_value",
);
redirects.push({
url: previous.url,
response_url: responseUrl.url,
@@ -293,10 +280,10 @@ export const handleRequestWillBeSent = (
resource_type: previous.resource_type,
status: numberValue(redirectResponse.status) ?? null,
mime_type: cdpStringValue(redirectResponse.mimeType) ?? null,
encoded_data_length:
encodedDataLength === undefined || encodedDataLength < 0
? null
: encodedDataLength,
encoded_data_length: (() => {
const length = numberValue(redirectResponse.encodedDataLength);
return length === undefined ? null : Math.max(0, length);
})(),
request_timestamp:
state.networkRequestTimestamps.get(requestId) ?? null,
// CDP reports redirectResponse on the next requestWillBeSent event.
@@ -305,6 +292,7 @@ export const handleRequestWillBeSent = (
});
}
}
const initiator = recordValue(params.initiator);
const initiatorFrame = initiatorLocation(initiator);
const rawInitiatorUrl = cdpStringValue(initiatorFrame?.url);
const initiatorUrl = allowedSanitizedUrl(
@@ -320,14 +308,14 @@ export const handleRequestWillBeSent = (
request_id: requestId,
url: sanitized.url,
origin: sanitized.origin ?? "",
method,
method: cdpStringValue(request.method) ?? "GET",
resource_type: cdpStringValue(params.type) ?? null,
status: null,
mime_type: null,
encoded_data_length: null,
redirects,
initiator: {
type: initiatorType,
type: cdpStringValue(initiator?.type) ?? "other",
url: initiatorUrl?.url ?? null,
line: integerOrNull(initiatorFrame?.lineNumber),
column: integerOrNull(initiatorFrame?.columnNumber),
@@ -423,14 +411,12 @@ export const handleLoadingFinished = (
if (state.malformedRedirectRequestIds.has(requestId)) return;
const current = state.network.get(requestId);
if (current === undefined) return;
const encodedDataLength = numberValue(params.encodedDataLength);
if (encodedDataLength === undefined || encodedDataLength < 0) {
state.completeness.exclude("network_requests", "invalid_protocol_value");
return;
}
state.network.set(requestId, {
...current,
encoded_data_length: encodedDataLength,
encoded_data_length: Math.max(
0,
numberValue(params.encodedDataLength) ?? 0,
),
});
};
+1 -1
View File
@@ -20,10 +20,10 @@ import type {
import { webSourceMapsSchema } from "../domain/webBundleAnalysis.js";
import { createWebTextArtifact } from "../domain/webContentArtifact.js";
import { safeParseJson } from "../domain/safeJson.js";
import { sourceMapJsonText } from "../domain/sourceMapEnvelope.js";
import { jsonParts } from "../domain/jsonSerialization.js";
import {
inspectSourceMapValue,
sourceMapJsonText,
SourceMapFormatFailure,
type SourceMapLeaf,
} from "../javascript/sourceMaps/SourceMapFormat.js";
+28 -128
View File
@@ -7,148 +7,48 @@ const input = {
importMap: null,
specifiers: ["./dep.js"],
};
describe("native module resolver cleanup ownership", () => {
it.each([false, true])(
"awaits owned cleanup when acquisition is cancelled (cleanup fails: %s)",
async (cleanupFails) => {
const controller = new AbortController();
let closed = false;
const provider = new NativeModuleResolver(
{ REA_BROWSER_EXECUTABLE: "/browser" },
() => {
controller.abort();
return Promise.resolve({
version: () => "fixture",
newContext: () => {
throw new Error("must not acquire context");
},
close: () => {
closed = true;
if (cleanupFails)
return Promise.reject(new Error("fixture cleanup refusal"));
return Promise.resolve();
},
});
},
);
const result = await provider.resolve(input, {
signal: controller.signal,
});
expect(closed).toBe(true);
if (result.ok) throw new Error("expected cancelled result");
const projected = projectAnalysisError(result.error);
expect(projected.code).toBe(
cleanupFails ? "cleanup_incomplete" : "cancelled",
);
if (cleanupFails) {
expect(projected.details).toMatchObject({
diagnostics: { primary_error: { code: "cancelled" } },
});
}
},
);
it("returns cleanup_incomplete when owned cleanup fails after an engine error", async () => {
let launches = 0;
let closes = 0;
let cleanupAllowed = false;
describe("optional native resolution acquisition", () => {
it("awaits owned cleanup when cancellation arrives during acquisition", async () => {
const controller = new AbortController();
let closed = false;
const provider = new NativeModuleResolver(
{ REA_BROWSER_EXECUTABLE: "/browser" },
() => {
launches += 1;
controller.abort();
return Promise.resolve({
version: () => "fixture",
newContext: () =>
Promise.reject(new Error("fixture context failure")),
close: async () => {
closes += 1;
if (!cleanupAllowed)
throw new Error("fixture owned cleanup refusal");
newContext: () => {
throw new Error("must not acquire context");
},
close: () => {
closed = true;
return Promise.resolve();
},
});
},
);
const result = await provider.resolve(input, { signal: controller.signal });
expect(closed).toBe(true);
if (result.ok) throw new Error("expected cancelled result");
expect(result.error._tag).toBe("AnalysisCancelledError");
});
it("returns cleanup_incomplete when owned cleanup fails after an engine error", async () => {
const provider = new NativeModuleResolver(
{ REA_BROWSER_EXECUTABLE: "/browser" },
() =>
Promise.resolve({
version: () => "fixture",
newContext: () =>
Promise.reject(new Error("fixture context failure")),
close: () =>
Promise.reject(new Error("fixture owned cleanup refusal")),
}),
);
const result = await provider.resolve(input);
if (result.ok) throw new Error("expected cleanup failure");
expect(projectAnalysisError(result.error).code).toBe("cleanup_incomplete");
expect(result.error.cleanupIncomplete).toBe(true);
expect(projectAnalysisError(result.error).details).toMatchObject({
diagnostics: {
primary_error: {
details: {
diagnostics: { error_message: "fixture context failure" },
},
},
cleanup_error: {
details: { diagnostics: { reason: "fixture owned cleanup refusal" } },
},
},
});
const blocked = await provider.resolve(input);
expect(blocked.ok).toBe(false);
if (!blocked.ok) expect(blocked.error.cleanupIncomplete).toBe(true);
expect(launches).toBe(1);
await expect(provider.close()).rejects.toMatchObject({
cleanupIncomplete: true,
});
cleanupAllowed = true;
await provider.close();
await provider.close();
expect(closes).toBe(4);
expect(launches).toBe(1);
});
it("awaits acquisition and cleanup while refusing queued work during shutdown", async () => {
let acquired: () => void = () => {
throw new Error("acquisition gate missing");
};
const acquiring = new Promise<void>((resolve) => {
acquired = resolve;
});
let release: () => void = () => {
throw new Error("release gate missing");
};
const released = new Promise<void>((resolve) => {
release = resolve;
});
let launches = 0;
let closes = 0;
const provider = new NativeModuleResolver(
{ REA_BROWSER_EXECUTABLE: "/browser" },
async () => {
launches += 1;
acquired();
await released;
return {
version: () => "fixture",
newContext: () =>
Promise.reject(new Error("fixture context failure")),
close: async () => {
closes += 1;
},
};
},
);
const first = provider.resolve(input);
await acquiring;
const queued = provider.resolve(input);
const closing = provider.close();
let closed = false;
void closing.then(() => {
closed = true;
});
await Promise.resolve();
expect(closed).toBe(false);
release();
await first;
const blocked = await queued;
expect(blocked.ok).toBe(false);
await closing;
expect(closed).toBe(true);
expect([launches, closes]).toEqual([1, 1]);
expect((await provider.resolve(input)).ok).toBe(false);
});
});
describe("optional native resolution acquisition", () => {
it("does not load the browser engine when the selected operation lacks its configuration", async () => {
const provider = new NativeModuleResolver({}, () => {
throw new Error("must not launch");
+147 -242
View File
@@ -7,7 +7,6 @@ import type {
WebModuleResolutionBatch,
} from "../../application/WebModulePorts.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js";
import {
AnalysisCancelledError,
AnalysisCapabilityUnavailableError,
@@ -17,7 +16,6 @@ import {
import { BrowserObservationError } from "../../domain/browserObservationError.js";
import { jsonObjectSchema } from "../../domain/jsonValue.js";
import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
import { ProviderCleanupError } from "../../domain/providerCleanupError.js";
import { err, ok, type Result } from "../../domain/result.js";
import { webModuleResolutionSchema } from "../../domain/webModuleTrace.js";
import { ProviderStartupDeadline } from "../../process/ProviderDeadline.js";
@@ -34,49 +32,26 @@ const reportSchema = z.strictObject({
),
});
type ModuleResolverBrowser = Pick<Browser, "newContext" | "version" | "close">;
/** Production launch seam: only the adapter knows the browser API. */
export type ModuleResolverLauncher = (
options: LaunchOptions,
) => Promise<ModuleResolverBrowser>;
) => Promise<Pick<Browser, "newContext" | "version" | "close">>;
const launch: ModuleResolverLauncher = async (options) =>
(await import("playwright-core")).chromium.launch(options);
/** Use native import.meta.resolve in an owned context with locally fulfilled requests. */
export class NativeModuleResolver implements WebModuleResolutionPort {
#tail: Promise<void> = Promise.resolve();
#closing = false;
#closePromise: Promise<void> | undefined;
#pendingBrowser: ModuleResolverBrowser | undefined;
constructor(
readonly environment: Readonly<Record<string, string | undefined>>,
readonly launcher: ModuleResolverLauncher = launch,
) {}
resolve(
input: Parameters<WebModuleResolutionPort["resolve"]>[0],
options?: ExecutionOptions,
): ReturnType<WebModuleResolutionPort["resolve"]> {
if (this.#closing) return Promise.resolve(err(this.#closedError()));
const operation = this.#tail.then(() =>
this.#closing ? err(this.#closedError()) : this.#resolve(input, options),
);
this.#tail = operation.then(
() => undefined,
() => undefined,
);
return operation;
}
async #resolve(
async resolve(
input: Parameters<WebModuleResolutionPort["resolve"]>[0],
options?: ExecutionOptions,
): ReturnType<WebModuleResolutionPort["resolve"]> {
if (options?.signal?.aborted === true)
return err(new AnalysisCancelledError(OPERATION));
const pending = await this.#retryCleanup();
if (pending !== undefined) return err(pending);
const executable = this.environment.REA_BROWSER_EXECUTABLE;
if (executable === undefined || !isAbsolute(executable))
return err(
@@ -91,16 +66,11 @@ export class NativeModuleResolver implements WebModuleResolutionPort {
),
);
const deadline = new ProviderStartupDeadline(TIMEOUT_MS, options?.signal);
let browser: ModuleResolverBrowser | undefined;
let browser: Pick<Browser, "newContext" | "version" | "close"> | undefined;
let cleanup: Promise<void> | undefined;
const close = (): Promise<void> => {
if (cleanup === undefined && browser !== undefined) {
const owned = browser;
cleanup = Promise.resolve().then(async () => {
await owned.close();
this.#pendingBrowser = undefined;
});
}
if (cleanup === undefined && browser !== undefined)
cleanup = browser.close();
return cleanup ?? Promise.resolve();
};
const onAbort = (): void => {
@@ -112,7 +82,6 @@ export class NativeModuleResolver implements WebModuleResolutionPort {
deadline.signal.addEventListener("abort", onAbort, { once: true });
let result: Result<WebModuleResolutionBatch, AnalysisError>;
try {
deadline.signal.throwIfAborted();
browser = await this.launcher({
executablePath: executable,
headless: true,
@@ -130,9 +99,142 @@ export class NativeModuleResolver implements WebModuleResolutionPort {
: []),
],
});
this.#pendingBrowser = browser;
deadline.signal.throwIfAborted();
result = ok(await resolveInBrowser(browser, input, deadline, executable));
const context = await browser.newContext({
serviceWorkers: "block",
acceptDownloads: false,
});
const page = await context.newPage();
const diagnostics: string[] = [];
let diagnosticBytes = 0;
page.on("console", (message) => {
const text = message.text();
diagnosticBytes += Buffer.byteLength(text);
if (diagnosticBytes <= 1024 * 1024)
diagnostics.push(`${message.type()}: ${text}`);
});
const requests: {
url: string;
resource_type: string;
effect: "fulfilled" | "blocked";
}[] = [];
const importerUrl = new URL(input.importerUrl).href;
const documentUrl = new URL("/.rea-module-resolver/document", importerUrl)
.href;
const transportUrl = new URL(importerUrl);
transportUrl.hash = "";
let stubSent = false;
await context.route("**/*", async (route) => {
const request = route.request();
if (
request.resourceType() === "document" &&
request.url() === documentUrl.split("#")[0]
) {
requests.push({
url: request.url(),
resource_type: "document",
effect: "fulfilled",
});
await route.fulfill({
status: 200,
contentType: "text/html",
body: `<!doctype html>${input.importMap === null ? "" : `<base href="${htmlAttribute(new URL(input.importMap.baseUrl).href)}"><script type="importmap">${scriptJson(input.importMap.value)}</script>`}`,
});
} else if (
!stubSent &&
request.resourceType() === "script" &&
request.url() === transportUrl.href
) {
stubSent = true;
requests.push({
url: request.url(),
resource_type: "script",
effect: "fulfilled",
});
await route.fulfill({
status: 200,
contentType: "text/javascript",
headers: { "Access-Control-Allow-Origin": "*" },
body: `if (typeof import.meta.resolve !== "function") {globalThis.__reaModuleUnsupported = true;} else {globalThis.__reaModuleResult = {importer_url: import.meta.url, entries: ${scriptJson(input.specifiers)}.map(specifier => {try {return {specifier, resolution: {state: "resolved", url: import.meta.resolve(specifier)}}} catch (error) {return {specifier, resolution: {state: "rejected", message: String(error)}}}})};}`,
});
} else {
requests.push({
url: request.url(),
resource_type: request.resourceType(),
effect: "blocked",
});
await route.abort();
}
});
await page.goto(documentUrl, {
timeout: Math.max(1, deadline.remainingMs()),
});
await page.evaluate(
`{const script = document.createElement("script"); script.type = "module"; script.src = ${scriptJson(importerUrl)}; script.onerror = () => {globalThis.__reaModuleFailure = "Trusted resolver stub could not be loaded in the selected context.";}; document.head.append(script);}`,
);
await page.waitForFunction(
() =>
Reflect.has(globalThis, "__reaModuleResult") ||
Reflect.has(globalThis, "__reaModuleUnsupported") ||
Reflect.has(globalThis, "__reaModuleFailure"),
undefined,
{ timeout: Math.max(1, deadline.remainingMs()) },
);
const unsupported: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleUnsupported") as unknown,
);
if (unsupported === true)
throw new AnalysisCapabilityUnavailableError(
"native-module-resolver",
OPERATION,
"native_resolution_unsupported",
{
userMessage: `Configured Chromium ${browser.version()} does not supply import.meta.resolve; use a browser supporting native module resolution.`,
},
);
const raw: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleResult") as unknown,
);
const failure: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleFailure") as unknown,
);
if (typeof failure === "string")
throw new AnalysisOutputError(
OPERATION,
`${failure} Native diagnostics: ${diagnostics.join("\n")}`,
);
const parsed = reportSchema.safeParse(raw);
if (
!parsed.success ||
parsed.data.importer_url !== importerUrl ||
parsed.data.entries.length !== input.specifiers.length ||
parsed.data.entries.some(
(entry, index) => entry.specifier !== input.specifiers[index],
)
)
throw new AnalysisOutputError(
OPERATION,
"Native resolver report has invalid shape or does not match the selected importer/specifiers.",
);
if (diagnosticBytes > 1024 * 1024)
throw new BrowserObservationError(OPERATION, "payload_limit");
deadline.signal.throwIfAborted();
const engine = {
id: "chromium-native-module-resolver",
version: browser.version(),
};
result = ok({
engine,
resolutions: parsed.data.entries.map((entry) => entry.resolution),
diagnostics,
rawResult: jsonObjectSchema.parse({
native_report: parsed.data,
requests,
diagnostics,
engine,
executable_path: executable,
}),
});
} catch (cause: unknown) {
result = err(
deadline.interruption === "cancelled"
@@ -155,217 +257,20 @@ export class NativeModuleResolver implements WebModuleResolutionPort {
} finally {
deadline.signal.removeEventListener("abort", onAbort);
deadline.dispose();
}
try {
await close();
} catch (cause: unknown) {
const failure = cleanupFailure(
cause,
result.ok ? result.value.rawResult : undefined,
);
return err(
result.ok
? failure
: analysisErrorWithCleanupFailure(result.error, failure, OPERATION),
);
try {
await close();
} catch (cause: unknown) {
result = err(
new BrowserObservationError(OPERATION, "cleanup_failed", { cause }),
);
}
}
if (result.ok && options?.signal?.aborted)
return err(new AnalysisCancelledError(OPERATION));
return result;
}
async close(): Promise<void> {
this.#closing = true;
this.#closePromise ??= this.#tail
.then(async () => {
const failure = await this.#retryCleanup();
if (failure !== undefined) throw failure;
})
.catch((cause: unknown) => {
this.#closePromise = undefined;
throw cause;
});
return this.#closePromise;
}
#closedError(): AnalysisError {
return new BrowserObservationError(OPERATION, "cleanup_failed", {
detail: "Native module resolver is closing.",
});
}
async #retryCleanup(): Promise<AnalysisError | undefined> {
const browser = this.#pendingBrowser;
if (browser === undefined) return undefined;
try {
await browser.close();
this.#pendingBrowser = undefined;
return undefined;
} catch (cause: unknown) {
return cleanupFailure(cause);
}
}
}
const resolveInBrowser = async (
browser: ModuleResolverBrowser,
input: Parameters<WebModuleResolutionPort["resolve"]>[0],
deadline: ProviderStartupDeadline,
executable: string,
): Promise<WebModuleResolutionBatch> => {
const context = await browser.newContext({
serviceWorkers: "block",
acceptDownloads: false,
});
const page = await context.newPage();
const diagnostics: string[] = [];
let diagnosticBytes = 0;
page.on("console", (message) => {
const text = message.text();
diagnosticBytes += Buffer.byteLength(text);
if (diagnosticBytes <= 1024 * 1024)
diagnostics.push(`${message.type()}: ${text}`);
});
const requests: {
url: string;
resource_type: string;
effect: "fulfilled" | "blocked";
}[] = [];
const importerUrl = new URL(input.importerUrl).href;
const documentUrl = new URL("/.rea-module-resolver/document", importerUrl)
.href;
const transportUrl = new URL(importerUrl);
transportUrl.hash = "";
let stubSent = false;
await context.route("**/*", async (route) => {
const request = route.request();
if (
request.resourceType() === "document" &&
request.url() === documentUrl.split("#")[0]
) {
requests.push({
url: request.url(),
resource_type: "document",
effect: "fulfilled",
});
await route.fulfill({
status: 200,
contentType: "text/html",
body: `<!doctype html>${input.importMap === null ? "" : `<base href="${htmlAttribute(new URL(input.importMap.baseUrl).href)}"><script type="importmap">${scriptJson(input.importMap.value)}</script>`}`,
});
} else if (
!stubSent &&
request.resourceType() === "script" &&
request.url() === transportUrl.href
) {
stubSent = true;
requests.push({
url: request.url(),
resource_type: "script",
effect: "fulfilled",
});
await route.fulfill({
status: 200,
contentType: "text/javascript",
headers: { "Access-Control-Allow-Origin": "*" },
body: `if (typeof import.meta.resolve !== "function") {globalThis.__reaModuleUnsupported = true;} else {globalThis.__reaModuleResult = {importer_url: import.meta.url, entries: ${scriptJson(input.specifiers)}.map(specifier => {try {return {specifier, resolution: {state: "resolved", url: import.meta.resolve(specifier)}}} catch (error) {return {specifier, resolution: {state: "rejected", message: String(error)}}}})};}`,
});
} else {
requests.push({
url: request.url(),
resource_type: request.resourceType(),
effect: "blocked",
});
await route.abort();
}
});
await page.goto(documentUrl, {
timeout: Math.max(1, deadline.remainingMs()),
});
await page.evaluate(
`{const script = document.createElement("script"); script.type = "module"; script.src = ${scriptJson(importerUrl)}; script.onerror = () => {globalThis.__reaModuleFailure = "Trusted resolver stub could not be loaded in the selected context.";}; document.head.append(script);}`,
);
await page.waitForFunction(
() =>
Reflect.has(globalThis, "__reaModuleResult") ||
Reflect.has(globalThis, "__reaModuleUnsupported") ||
Reflect.has(globalThis, "__reaModuleFailure"),
undefined,
{ timeout: Math.max(1, deadline.remainingMs()) },
);
const unsupported: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleUnsupported") as unknown,
);
if (unsupported === true)
throw new AnalysisCapabilityUnavailableError(
"native-module-resolver",
OPERATION,
"native_resolution_unsupported",
{
userMessage: `Configured Chromium ${browser.version()} does not supply import.meta.resolve; use a browser supporting native module resolution.`,
},
);
const raw: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleResult") as unknown,
);
const failure: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleFailure") as unknown,
);
if (typeof failure === "string")
throw new AnalysisOutputError(
OPERATION,
`${failure} Native diagnostics: ${diagnostics.join("\n")}`,
);
const parsed = reportSchema.safeParse(raw);
if (
!parsed.success ||
parsed.data.importer_url !== importerUrl ||
parsed.data.entries.length !== input.specifiers.length ||
parsed.data.entries.some(
(entry, index) => entry.specifier !== input.specifiers[index],
)
)
throw new AnalysisOutputError(
OPERATION,
"Native resolver report has invalid shape or does not match the selected importer/specifiers.",
);
if (diagnosticBytes > 1024 * 1024)
throw new BrowserObservationError(OPERATION, "payload_limit");
deadline.signal.throwIfAborted();
const engine = {
id: "chromium-native-module-resolver",
version: browser.version(),
};
return {
engine,
resolutions: parsed.data.entries.map((entry) => entry.resolution),
diagnostics,
rawResult: jsonObjectSchema.parse({
native_report: parsed.data,
requests,
diagnostics,
engine,
executable_path: executable,
}),
};
};
const cleanupFailure = (
cause: unknown,
observation?: WebModuleResolutionBatch["rawResult"],
): ProviderCleanupError =>
new ProviderCleanupError(
"native-module-resolver",
["chromium-browser"],
{
reason: cause instanceof Error ? cause.message : String(cause),
...(observation === undefined
? {}
: { partial_observation: observation }),
},
{ cause, operation: OPERATION },
);
const scriptJson = (value: unknown): string =>
JSON.stringify(value).replaceAll("<", "\\u003c");
+4 -22
View File
@@ -15,7 +15,6 @@ import type { CliInstance } from "./types.js";
import { runCliJavaScriptApplicationAnalysis } from "./javascriptApplicationAnalysis.js";
import { withCommandCancellation } from "./commandCancellation.js";
import { javascriptApplicationOptions } from "../cliObservationOptions.js";
import { AnalysisInputError } from "../domain/analysisErrorCore.js";
/** Register provider-neutral binary overview and procedure CLI commands. */
export const registerCoreBinaryCommands = (
@@ -52,7 +51,6 @@ const registerOverviewCommands = (
}),
options: overviewOptions.extend({
integrityPolicy: javascriptApplicationOptions.shape.integrityPolicy,
sourcePath: javascriptApplicationOptions.shape.sourcePath,
}),
run: async ({ args, options }) => {
// Route JavaScript targets exactly like analyze-javascript-application,
@@ -65,35 +63,19 @@ const registerOverviewCommands = (
{
input_path: resolve(args.path),
integrity_policy: options.integrityPolicy,
...(options.sourcePath === undefined
? {}
: { source_paths: options.sourcePath }),
},
signal,
),
),
);
return logCliCommand(logger, "analyze", () => {
if (options.sourcePath !== undefined)
throw new AnalysisInputError(
"analyze_javascript_application",
undefined,
[
{
path: ["source_paths"],
reason: "invalid_value",
message:
"Source selection requires the static JavaScript ASAR/directory analysis lane.",
},
],
);
return runDirectAnalysis(
return logCliCommand(logger, "analyze", () =>
runDirectAnalysis(
args.path,
"binary_overview",
{},
directAnalysisOptionsFromCli(logger, options),
);
});
),
);
},
});
cli.command(CLI_COMMANDS.inspect, {
-3
View File
@@ -200,9 +200,6 @@ const registerJavaScriptApplicationCommand = (
input_path: args.path,
format: options.artifactFormat,
integrity_policy: options.integrityPolicy,
...(options.sourcePath === undefined
? {}
: { source_paths: options.sourcePath }),
...(options.maxHeapMb === undefined
? {}
: { max_heap_mb: options.maxHeapMb }),
+22 -32
View File
@@ -42,39 +42,29 @@ export const registerWebModuleCommands = (
.describe("HTTP(S) import-map base URL"),
}),
run: ({ args, options }) =>
withCommandCancellation(async (signal) => {
try {
return await logCliCommand(
logger,
CLI_COMMANDS.traceWebModuleImports,
async () => {
const result = await service.trace(
{
manifest_path: args.manifestPath,
script_index: args.scriptIndex,
...(options.importerUrl === undefined
? {}
: { importer_url: options.importerUrl }),
...(options.importMapPath === undefined &&
options.importMapBaseUrl === undefined
? {}
: {
import_map: {
path: options.importMapPath,
base_url: options.importMapBaseUrl,
},
}),
},
{ signal },
);
return result.ok
? result.value
: projectAnalysisError(result.error);
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.traceWebModuleImports, async () => {
const result = await service.trace(
{
manifest_path: args.manifestPath,
script_index: args.scriptIndex,
...(options.importerUrl === undefined
? {}
: { importer_url: options.importerUrl }),
...(options.importMapPath === undefined &&
options.importMapBaseUrl === undefined
? {}
: {
import_map: {
path: options.importMapPath,
base_url: options.importMapBaseUrl,
},
}),
},
{ signal },
);
} finally {
await service.close();
}
}),
return result.ok ? result.value : projectAnalysisError(result.error);
}),
),
});
};
-6
View File
@@ -1,6 +1,5 @@
import { z } from "incur";
import { javaScriptAnalysisResourceControlsSchema } from "./domain/javascript/javascriptAnalysisResourceControls.js";
import { javaScriptSourcePathsSchema } from "./domain/javascript/javascriptSourceSelection.js";
/** Accept an agent-selected browser observation duration without an artificial ceiling. */
export const observationDuration = (fallback: number, minimum = 0) =>
@@ -61,11 +60,6 @@ export const electronPageInspectionOptions = z.object({
});
export const javascriptApplicationOptions = z.object({
sourcePath: javaScriptSourcePathsSchema
.optional()
.describe(
"Exact inventoried file path relative to the application root; repeat to select several. No implicit dependencies.",
),
maxHeapMb: javaScriptAnalysisResourceControlsSchema.shape.max_heap_mb,
analysisTimeoutMs:
javaScriptAnalysisResourceControlsSchema.shape.analysis_timeout_ms,
-2
View File
@@ -8,7 +8,6 @@ import {
ghidraCompilerSpecIdSchema,
ghidraLanguageIdSchema,
} from "./ghidraLanguageOverride.js";
import { ghidraDisabledAnalyzersSchema } from "./ghidraDisabledAnalyzers.js";
import {
DEFAULT_GHIDRA_STARTUP_TIMEOUT_MS,
DEFAULT_HOPPER_STARTUP_TIMEOUT_MS,
@@ -49,7 +48,6 @@ const environmentSchema = z.object({
),
REA_GHIDRA_LANGUAGE_ID: ghidraLanguageIdSchema.optional(),
REA_GHIDRA_COMPILER_SPEC_ID: ghidraCompilerSpecIdSchema.optional(),
REA_GHIDRA_DISABLED_ANALYZERS: ghidraDisabledAnalyzersSchema.optional(),
REA_GHIDRA_SEED_FILE: z
.string()
.min(1)
-42
View File
@@ -1,42 +0,0 @@
import { z } from "zod";
export const MAX_GHIDRA_DISABLED_ANALYZER_NAMES = 32;
export const MAX_GHIDRA_DISABLED_ANALYZER_NAME_LENGTH = 120;
// Ghidra's headless parser consumes `-`-prefixed arguments after a script path
// as its own options, and REA joins the names with commas into one pre-script
// argument, so a name can neither start with `-` nor contain a comma. The
// remaining admitted characters keep the argument a single quotable token on
// every supported launcher, including the Windows batch invocation.
const ghidraAnalyzerNamePattern = /^[A-Za-z0-9][A-Za-z0-9 ._-]*$/u;
/**
* Parse `REA_GHIDRA_DISABLED_ANALYZERS` into the Ghidra auto-analyzer names to
* disable before default auto-analysis, such as
* `Objective-C Message Analyzer`. Empty and blank entries are dropped; a list
* without names selects no override.
*/
export const ghidraDisabledAnalyzersSchema = z
.string()
.transform((value) =>
value
.split(",")
.map((name) => name.trim())
.filter((name) => name.length > 0),
)
.refine(
(names) => names.length <= MAX_GHIDRA_DISABLED_ANALYZER_NAMES,
`must list at most ${MAX_GHIDRA_DISABLED_ANALYZER_NAMES} analyzer names`,
)
.refine(
(names) =>
names.every(
(name) => name.length <= MAX_GHIDRA_DISABLED_ANALYZER_NAME_LENGTH,
),
`must keep every analyzer name at most ${MAX_GHIDRA_DISABLED_ANALYZER_NAME_LENGTH} characters`,
)
.refine(
(names) => names.every((name) => ghidraAnalyzerNamePattern.test(name)),
"must be Ghidra analyzer names starting with a letter or digit and containing only letters, digits, spaces, periods, underscores, and hyphens",
)
.transform((names) => (names.length === 0 ? undefined : names));
-3
View File
@@ -88,9 +88,6 @@ export const parseConfig = (
...(env.REA_GHIDRA_SEED_FILE === undefined
? {}
: { ghidraSeedFile: env.REA_GHIDRA_SEED_FILE }),
...(env.REA_GHIDRA_DISABLED_ANALYZERS === undefined
? {}
: { ghidraDisabledAnalyzers: env.REA_GHIDRA_DISABLED_ANALYZERS }),
...(env.REA_GHIDRA_NATIVEAOT_JAR === undefined
? {}
: { ghidraNativeAotJar: env.REA_GHIDRA_NATIVEAOT_JAR }),
-1
View File
@@ -14,7 +14,6 @@ export interface AppConfig {
readonly ghidraJavaHome: string | undefined;
readonly ghidraStartupTimeoutMs: number;
readonly ghidraLanguageOverride?: GhidraLanguageOverride;
readonly ghidraDisabledAnalyzers?: readonly string[];
readonly ghidraSeedFile?: string;
readonly ghidraNativeAotJar?: string;
readonly ilspyCmdPath: string | undefined;
@@ -88,7 +88,7 @@ export const ELECTRON_TOOL_CONTRACTS = [
name: "analyze_javascript_application",
...toolContractMetadata("analyze_javascript_application"),
description:
"Reconstruct one local ASAR or extracted JavaScript application as an inline application graph without executing it. Returns recovered graph nodes, edges, semantic relations, integrity contradictions, limitations, and coverage. Select source_paths to acquire and analyze only exact inventoried file paths; the complete artifact inventory and identity remain intact, and unselected sources remain explicit unknowns. Dependencies are not implicitly analyzed. Select detail summary to retain the complete analysis Evidence in this session and receive only its summary view, then inspect selected modules or pages with inspect_analysis_view. Summary detail reduces delivery, whereas source_paths bounds source analysis. Integrity mismatches fail by default; record-and-continue retains observed bytes as untrusted and marks graph coverage partial.",
"Reconstruct one local ASAR or extracted JavaScript application as an inline application graph without executing it. Returns recovered graph nodes, edges, semantic relations, integrity contradictions, limitations, and coverage. Select detail summary to retain the complete analysis Evidence in this session and receive only its summary view, then inspect selected modules or pages with inspect_analysis_view. Integrity mismatches fail by default; record-and-continue retains observed bytes as untrusted and marks graph coverage partial.",
kind: "electron-provider",
inputSchema: analyzeJavaScriptApplicationRequestSchema,
outputSchema: applicationOutputSchema,
@@ -101,14 +101,6 @@ export const ELECTRON_TOOL_CONTRACTS = [
integrity_policy: "fail",
},
},
{
title: "Analyze selected entry sources before unrelated vendor files",
input: {
input_path: "/Applications/Example.app/Contents/Resources/app.asar",
source_paths: ["package.json", "main.js", "preload.js"],
detail: "summary",
},
},
{
title: "Retain the complete analysis and return its summary view",
input: {
@@ -505,7 +505,6 @@ export const graphNode = z.discriminatedUnion("status", [
address: z.string(),
status: z.literal("ok"),
calls: z.array(z.string()),
limitations: z.array(z.string()).optional(),
}),
z.object({
address: z.string(),
+1 -5
View File
@@ -27,7 +27,6 @@ import {
} from "./evidenceBundle.js";
import {
evidenceLocationSchema,
rebindEvidenceToTargetPath,
type Evidence,
type EvidenceLocation,
type EvidenceSubjectTarget,
@@ -412,7 +411,7 @@ export const snapshotEvidenceForQuery = (
(candidate) => candidate.query_id === queryId,
);
if (entry === undefined && workflowEntry === undefined) return undefined;
const evidence = snapshot.evidence_bundle.records.find(
return snapshot.evidence_bundle.records.find(
(record) =>
record.subject?.digest.sha256 === target.sha256 &&
record.operation === operation &&
@@ -426,9 +425,6 @@ export const snapshotEvidenceForQuery = (
(workflowEntry !== undefined &&
evidenceMatchesWorkflowEntry(record, workflowEntry, snapshot))),
);
return evidence === undefined
? undefined
: rebindEvidenceToTargetPath(evidence, target);
};
/** Create one exact replay record for a derived application workflow result. */
-2
View File
@@ -1,5 +1,4 @@
import { z } from "zod";
import { javaScriptSourceSelectionSchema } from "../javascript/javascriptSourceSelection.js";
import {
AnalysisInputError,
@@ -171,7 +170,6 @@ const javascriptSummarySchema = z.strictObject({
input_path: z.string().min(1),
format: z.enum(["asar", "directory"]),
root_artifact_sha256: digestSchema,
source_selection: javaScriptSourceSelectionSchema.exactOptional(),
statistics: jsonObjectSchema,
electron: jsonObjectSchema,
coverage: jsonObjectSchema,
@@ -255,11 +255,6 @@ export const projectJavaScriptApplicationView = (
input_path: analysis.input_path,
format: analysis.format,
root_artifact_sha256: analysis.root_artifact_sha256,
...(analysis.source_selection === undefined
? {}
: {
source_selection: analysis.source_selection,
}),
statistics: jsonObjectSchema.parse(analysis.statistics),
electron: jsonObjectSchema.parse(analysis.summary),
coverage: jsonObjectSchema.parse({
+1 -71
View File
@@ -14,10 +14,6 @@ const observe = (
address: string,
callees: readonly string[],
provider = "rea-workflow",
options: {
readonly evidenceLimitations?: readonly string[];
readonly dossierLimitations?: readonly string[];
} = {},
): Evidence => {
const value = functionDossierSchema.parse({
procedure: {
@@ -53,7 +49,7 @@ const observe = (
native_api: null,
native_value_flow: null,
basic_blocks: [],
limitations: [...(options.dossierLimitations ?? [])],
limitations: [],
});
return createEvidence(
{ path: "/tmp/a", sha256: "a".repeat(64), format: "mach-o" },
@@ -66,9 +62,6 @@ const observe = (
result: jsonValueSchema.parse(value),
confidence: "derived",
authority: "shipped-artifact",
...(options.evidenceLimitations === undefined
? {}
: { limitations: options.evidenceLimitations }),
},
);
};
@@ -133,22 +126,6 @@ describe("call path reconstruction", () => {
expect(unknown.limitations).toContain(
"No analyze_function Evidence covers reachable function 0x2000",
);
const limitedEmpty = run([
observe("0x1000", [], "rea-workflow", {
dossierLimitations: ["Call relationships are incomplete."],
}),
observe("0x4000", []),
]);
expect(limitedEmpty).toMatchObject({
status: "unknown",
search_scope: { exhaustive: false },
shortest_hops: null,
paths: [],
});
expect(limitedEmpty.limitations).toContain(
"Call relationships are incomplete.",
);
});
it("searches the complete supplied graph", () => {
@@ -205,53 +182,6 @@ describe("call path topology and capacity", () => {
);
});
it("keeps reached function limitations and marks expanded coverage incomplete", () => {
const evidenceLimitation = "Call graph output was truncated.";
const dossierLimitation = "One call target could not be resolved.";
const result = run([
observe("0x1000", ["0x4000"], "rea-workflow", {
evidenceLimitations: [evidenceLimitation],
dossierLimitations: [dossierLimitation],
}),
observe("0x4000", []),
]);
expect(result).toMatchObject({
status: "found",
shortest_hops: 1,
search_scope: { exhaustive: false },
paths: [
{
nodes: [{ address: "0x1000" }, { address: "0x4000" }],
},
],
});
expect(result.limitations).toEqual(
expect.arrayContaining([evidenceLimitation, dossierLimitation]),
);
});
it("retains goal dossier limitations without expanding the goal", () => {
const goalLimitation = "The goal dossier has partial outgoing references.";
const result = run([
observe("0x1000", ["0x4000"]),
observe("0x4000", ["0x5000"], "rea-workflow", {
dossierLimitations: [goalLimitation],
}),
]);
expect(result).toMatchObject({
status: "found",
shortest_hops: 1,
search_scope: { exhaustive: true },
});
expect(result.limitations).toContain(goalLimitation);
expect(result.explored.nodes).toBe(2);
expect(result.limitations).not.toContain(
"No analyze_function Evidence covers reachable function 0x5000",
);
});
it("returns a zero-hop path when start and goal are the same function", () => {
const result = run([observe("0x1000", ["0x2000"])], {
goal: { address: "0x1000" },
+2 -13
View File
@@ -229,27 +229,16 @@ const inspectSearch = ({
const reached = new Map<string, number>([[start, 0]]);
const queue = [start];
const limitations: string[] = [];
let incompleteExpandedCoverage = false;
for (let index = 0; index < queue.length; index += 1) {
const node = queue[index];
if (node === undefined) continue;
const depth = reached.get(node) ?? 0;
if (node === goal) continue;
const snapshot = snapshots.get(node);
if (snapshot !== undefined) {
limitations.push(...snapshot.limitations);
// The requested goal is a terminal for this search. Its outgoing edges
// cannot add a path to the goal, so its dossier limits are contextual
// but do not make the searched start-to-goal closure incomplete.
if (node === goal) continue;
if (snapshot.limitations.length > 0) incompleteExpandedCoverage = true;
} else if (node === goal) {
continue;
}
if (snapshot === undefined) {
limitations.push(
`No analyze_function Evidence covers reachable function ${node}`,
);
incompleteExpandedCoverage = true;
continue;
}
const neighbors = graph.outNeighbors(node).sort(compareUnicodeCodePoints);
@@ -263,7 +252,7 @@ const inspectSearch = ({
graph,
snapshots,
reached,
exhaustive: !incompleteExpandedCoverage,
exhaustive: limitations.length === 0,
limitations,
};
};
+1 -51
View File
@@ -1,4 +1,3 @@
import { basename } from "node:path";
import { z } from "zod";
import { digestCanonicalValue } from "./canonicalDigest.js";
@@ -183,18 +182,6 @@ export interface EvidenceSubjectTarget {
readonly format: z.infer<typeof subjectSchema>["format"];
readonly architecture?: BinaryArchitecture;
}
/** Rebind a matching immutable artifact subject to the caller's selected path. */
export const rebindEvidenceSubjectTargetPath = (
subject: EvidenceSubjectTarget,
target: EvidenceSubjectTarget,
): EvidenceSubjectTarget =>
subject.sha256 === target.sha256 &&
subject.format === target.format &&
(subject.architecture ?? null) === (target.architecture ?? null)
? { ...subject, path: target.path }
: subject;
type EvidenceAuthority = z.infer<typeof evidenceAuthoritySchema>;
type ExecutionEnvironment = z.infer<typeof executionEnvironmentSchema>;
@@ -265,43 +252,6 @@ export const parseEvidence = (input: unknown): Evidence => {
export const immutableEvidence = (input: unknown): Evidence =>
rememberImmutableEvidence(parseEvidence(input));
/** Rebind display path metadata when an authenticated record names this target. */
export const rebindEvidenceToTargetPath = (
evidence: Evidence,
target: EvidenceSubjectTarget,
): Evidence => {
const subject = evidence.subject;
if (subject === null) return evidence;
const rebound = rebindEvidenceSubjectTargetPath(
{
path: subject.local_path,
sha256: subject.digest.sha256,
format: subject.format,
...(subject.architecture === null
? {}
: { architecture: subject.architecture }),
},
target,
);
if (rebound.path === subject.local_path) return evidence;
const reboundSubject = subjectSchema.parse({
...subject,
name: basename(target.path) || "artifact",
local_path: target.path,
});
const reboundEvidence = { ...evidence, subject: reboundSubject };
if (!isImmutableEvidence(evidence)) return immutableEvidence(reboundEvidence);
// The original authenticated Evidence owns immutable payloads. Only the
// small display subject and envelope are new, so preserve those payloads
// without reparsing a potentially large analysis result.
Object.freeze(reboundSubject.digest);
Object.freeze(reboundSubject);
Object.freeze(reboundEvidence);
immutableEvidenceSnapshots.set(reboundEvidence, reboundEvidence);
return reboundEvidence;
};
/** Recognize authenticated Evidence whose complete reachable JSON data is immutable. */
export const isImmutableEvidence = (evidence: Evidence): boolean =>
immutableEvidenceSnapshots.has(evidence);
@@ -366,7 +316,7 @@ const normalizeEvidenceObservation = (
target === undefined
? null
: {
name: basename(target.path) || "artifact",
name: target.path.split("/").at(-1) || "artifact",
digest: { sha256: target.sha256 },
format: target.format,
architecture: target.architecture ?? null,
@@ -16,10 +16,6 @@ import { integrityContradictionSchema } from "../artifactGraph.js";
import { artifactIntegrityPolicySchema } from "../artifactIntegrityPolicy.js";
import { localPathStringSchema } from "../localPath.js";
import { javaScriptAnalysisResourceControlsSchema } from "./javascriptAnalysisResourceControls.js";
import {
javaScriptSourcePathsSchema,
javaScriptSourceSelectionSchema,
} from "./javascriptSourceSelection.js";
const countSchema = z.number().int().min(0);
@@ -28,7 +24,6 @@ export const analyzeJavaScriptApplicationInputSchema = z.strictObject({
input_path: localPathStringSchema,
format: z.enum(["auto", "asar", "directory"]).default("auto"),
integrity_policy: artifactIntegrityPolicySchema,
source_paths: javaScriptSourcePathsSchema.optional(),
...javaScriptAnalysisResourceControlsSchema.shape,
});
@@ -76,7 +71,6 @@ const applicationAnalysisResultShape = z.strictObject({
inventory_manifest_id: prefixedDigestSchema("agm"),
inventory_graph_sha256: digestSchema,
integrity_contradictions: z.array(integrityContradictionSchema),
source_selection: javaScriptSourceSelectionSchema.exactOptional(),
graph: javascriptApplicationGraphSchema,
summary: electronBoundarySummarySchema,
statistics: reconstructionStatisticsSchema,
@@ -1,5 +1,3 @@
import type { JavaScriptSourceSelection } from "./javascriptSourceSelection.js";
/** Relevant file categories projected from the complete artifact inventory. */
export type JavaScriptArtifactFileKind =
| "package-json"
@@ -22,10 +20,7 @@ export interface JavaScriptArtifactFile {
| { readonly included: true; readonly value: string }
| {
readonly included: false;
readonly reason:
| "not-applicable"
| "invalid-utf8"
| "outside-analysis-scope";
readonly reason: "not-applicable" | "invalid-utf8";
};
}
@@ -43,5 +38,4 @@ export interface JavaScriptArtifactFileSet {
readonly containers: readonly JavaScriptArtifactContainer[];
readonly text_bytes_read: number;
readonly invalid_utf8_files: number;
readonly source_selection?: JavaScriptSourceSelection;
}
@@ -1,20 +0,0 @@
import { z } from "zod";
/** Exact inventoried paths selected for text acquisition and static analysis. */
export const javaScriptSourcePathsSchema = z
.array(z.string().min(1))
.min(1)
.describe(
"Exact relevant file paths from the artifact inventory, relative to the application root, analyzed in first-request order before unselected scopes; no globs or implicit dependencies. Omit to analyze all sources.",
);
/** Caller scope retained independently of analysis success or completeness. */
export const javaScriptSourceSelectionSchema = z.strictObject({
requested_paths: javaScriptSourcePathsSchema,
unselected_files: z.number().int().min(0),
});
/** Explicit selected-source boundary for an otherwise complete inventory. */
export type JavaScriptSourceSelection = z.infer<
typeof javaScriptSourceSelectionSchema
>;
@@ -1,21 +0,0 @@
import { AnalysisCapabilityUnavailableError } from "./analysisErrorCore.js";
import type { JsonValue } from "./jsonValue.js";
/** Convert a provider's host configuration failure into actionable refusal evidence. */
export const providerConfigurationRefusal = (
providerId: string,
operation: string,
failure: Error & {
readonly diagnostics?: Readonly<Record<string, JsonValue>>;
},
): AnalysisCapabilityUnavailableError => {
const remediation = failure.diagnostics?.remediation;
const reason =
typeof remediation === "string"
? `${failure.message}. ${remediation}.`
: `${failure.message}.`;
return new AnalysisCapabilityUnavailableError(providerId, operation, reason, {
cause: failure,
userMessage: reason,
});
};
+18 -8
View File
@@ -1,6 +1,6 @@
import { describe, expect, it } from "vitest";
import { flattenSourceMapLeaves } from "./sourceMapEnvelope.js";
import { flattenSourceMapLeaves, isVersion3Map } from "./sourceMapEnvelope.js";
const leaf = {
version: 3,
@@ -10,7 +10,14 @@ const leaf = {
};
describe("source-map envelope walk", () => {
it("collects indexed section leaves without interpreting generated offsets", () => {
it("accepts version-3 maps and rejects other shapes", () => {
expect(isVersion3Map(leaf)).toBe(true);
expect(isVersion3Map({ version: 2, mappings: "AAAA" })).toBe(false);
expect(isVersion3Map(null)).toBe(false);
expect(isVersion3Map([{ version: 3 }])).toBe(false);
});
it("collects indexed section leaves without validating offsets by default", () => {
const root = {
version: 3,
sections: [
@@ -21,21 +28,24 @@ describe("source-map envelope walk", () => {
expect(flattenSourceMapLeaves(root)).toHaveLength(2);
});
it("rejects sections without embedded maps", () => {
it("rejects non-map sections and enforces offsets when requested", () => {
const missing = {
version: 3,
sections: [{ offset: { line: 0, column: 0 } }],
};
expect(flattenSourceMapLeaves(missing)).toBeUndefined();
const badOffset = {
version: 3,
sections: [{ offset: { line: -1, column: 0 }, map: leaf }],
};
expect(
flattenSourceMapLeaves(badOffset, { validateOffsets: true }),
).toBeUndefined();
expect(flattenSourceMapLeaves(badOffset)).toHaveLength(1);
});
it("rejects non-version-3 roots and nested maps", () => {
expect(flattenSourceMapLeaves({ version: 3 })).toHaveLength(1);
expect(flattenSourceMapLeaves(null)).toBeUndefined();
expect(flattenSourceMapLeaves([{ version: 3 }])).toBeUndefined();
expect(
flattenSourceMapLeaves({ version: 2, mappings: "AAAA" }),
).toBeUndefined();
expect(flattenSourceMapLeaves({ mappings: "AAAA" })).toBeUndefined();
expect(
flattenSourceMapLeaves({
+25 -14
View File
@@ -1,26 +1,28 @@
/** Artifact source declarations do not require generated-position decoding. */
type SourceMapRecord = Readonly<Record<string, unknown>>;
/**
* Return the JSON parsing view of retained source-map text. ECMA-426 permits
* HTTP responses to start with `)]}'`; the first line is transport framing.
* Keep the original text for artifact identity and Evidence provenance.
* https://tc39.es/ecma426/#sec-fetchsourcemap
* Structural walk over indexed source-map envelopes. Both the browser
* source-map fetcher and the artifact source-map reader accept version-3
* maps with recursively indexed `sections`; this module owns that walk so
* the two validators cannot drift. Leaf content checks (mappings, sources,
* names, contents) stay with the callers, which enforce different
* strictness.
*/
export const sourceMapJsonText = (text: string): string => {
const withoutBom = text.replace(/^\uFEFF/u, "");
if (!withoutBom.startsWith(")]}'")) return withoutBom;
const newline = withoutBom.search(/[\r\n]/u);
return newline === -1 ? "" : withoutBom.slice(newline);
};
type SourceMapRecord = Readonly<Record<string, unknown>>;
const isRecord = (value: unknown): value is SourceMapRecord =>
typeof value === "object" && value !== null && !Array.isArray(value);
/** Whether a parsed value has the version-3 map shape. */
const isVersion3Map = (value: unknown): value is SourceMapRecord =>
export const isVersion3Map = (value: unknown): value is SourceMapRecord =>
isRecord(value) && value.version === 3;
const isValidOffset = (value: unknown): boolean =>
typeof value === "number" && Number.isSafeInteger(value) && value >= 0;
export interface FlattenSourceMapOptions {
/** Require each section offset to carry safe-integer line/column. */
readonly validateOffsets?: boolean;
}
/**
* Collect leaf maps from an indexed source-map envelope, or undefined when
* the envelope is not a version-3 map. A map carrying neither `mappings`
@@ -29,6 +31,7 @@ const isVersion3Map = (value: unknown): value is SourceMapRecord =>
*/
export const flattenSourceMapLeaves = (
root: unknown,
options?: FlattenSourceMapOptions,
): readonly SourceMapRecord[] | undefined => {
if (!isVersion3Map(root)) return undefined;
const leaves: SourceMapRecord[] = [];
@@ -39,6 +42,14 @@ export const flattenSourceMapLeaves = (
if (Array.isArray(map.sections)) {
for (const section of map.sections) {
if (!isRecord(section) || !isRecord(section.map)) return undefined;
if (options?.validateOffsets === true) {
if (!isRecord(section.offset)) return undefined;
if (
!isValidOffset(section.offset.line) ||
!isValidOffset(section.offset.column)
)
return undefined;
}
pending.push(section.map);
}
} else leaves.push(map);
@@ -259,35 +259,6 @@ describe("managed artifact coded indexes", () => {
layout.codedIndexSize("HasCustomAttribute"),
);
});
it("resolves local attribute constructors without rescanning TypeDef rows", () => {
// The owning type follows every empty type. A per-attribute TypeDef scan
// took about 38 s here; the owner index is built once per layout.
const bytes = buildManagedPeFixture({
additionalEmptyTypes: 16_000,
customAttributeRowCount: 16_000,
customAttributeTypeRaw: (1 << 3) | 2,
typeName: "LookupAttribute",
extendsRaw: 0,
methodName: ".ctor",
methodFlags: 0x1886,
methodSignature: Buffer.from([0x20, 0x01, 0x01, 0x0e]),
});
const start = performance.now();
const artifact = inspectManagedArtifactBytes(
bytes,
managedPeFixtureTarget(bytes),
);
expect(performance.now() - start).toBeLessThan(2000);
expect(artifact.coverage).toEqual({ state: "complete", issues: [] });
expect(artifact.attributes).toHaveLength(16_000);
expect(
artifact.attributes.every(
(attribute) => attribute.type_name === "Fixture.LookupAttribute",
),
).toBe(true);
}, 30_000);
});
describe("managed artifact reference inventory", () => {
@@ -0,0 +1,96 @@
import { fc, it } from "@fast-check/vitest";
import { describe, expect } from "vitest";
import {
createDeclaringTypeLookup,
declaringType,
type TypeRange,
} from "./ManagedMemberInspectorCore.js";
const range = (
name: string,
fieldStart: number,
fieldEnd: number,
methodStart = fieldStart,
methodEnd = fieldEnd,
): TypeRange => ({
token: name,
fullName: name,
fieldStart,
fieldEnd,
methodStart,
methodEnd,
});
describe("managed declaring type ownership", () => {
it("resolves field and method boundaries independently, including empty types and gaps", () => {
const ranges = [
range("first", 1, 3, 1, 1),
range("empty", 3, 3, 1, 1),
range("last", 5, 7, 1, 4),
];
const field = createDeclaringTypeLookup(ranges, "field");
const method = createDeclaringTypeLookup(ranges, "method");
expect([0, 1, 2, 3, 4, 5, 6, 7].map(field)).toEqual([
null,
{ token: "first", fullName: "first" },
{ token: "first", fullName: "first" },
null,
null,
{ token: "last", fullName: "last" },
{ token: "last", fullName: "last" },
null,
]);
expect([0, 1, 2, 3, 4].map(method)).toEqual([
null,
{ token: "last", fullName: "last" },
{ token: "last", fullName: "last" },
{ token: "last", fullName: "last" },
null,
]);
});
it("retains the first observed owner for overlapping, reversed, or out-of-order ranges", () => {
const ranges = [
range("first", 5, 9),
range("overlap", 1, 7),
range("reversed", 10, 2),
];
const lookup = createDeclaringTypeLookup(ranges, "method");
expect(lookup(6)).toEqual({ token: "first", fullName: "first" });
expect(lookup(2)).toEqual({ token: "overlap", fullName: "overlap" });
expect(lookup(10)).toBeNull();
expect(createDeclaringTypeLookup([], "method")(1)).toBeNull();
});
it.prop([
fc.array(
fc.record({
fieldStart: fc.integer({ min: 0, max: 30 }),
fieldEnd: fc.integer({ min: 0, max: 30 }),
methodStart: fc.integer({ min: 0, max: 30 }),
methodEnd: fc.integer({ min: 0, max: 30 }),
}),
{ maxLength: 20 },
),
])(
"agrees with observed first-match ownership for arbitrary metadata ranges",
(raw) => {
const ranges = raw.map((value, index) => ({
...value,
token: String(index),
fullName: `Type${String(index)}`,
}));
for (const table of ["field", "method"] as const) {
const lookup = createDeclaringTypeLookup(ranges, table);
for (const row of [
NaN,
-Infinity,
Infinity,
...Array.from({ length: 32 }, (_, i) => i),
])
expect(lookup(row)).toEqual(declaringType(ranges, table, row));
}
},
);
});
+50
View File
@@ -154,6 +154,56 @@ export const typeRanges = (
return ranges;
};
export const declaringType = (
ranges: readonly TypeRange[],
table: "field" | "method",
row: number,
): { readonly token: string; readonly fullName: string | null } | null => {
for (const range of ranges) {
const start = table === "field" ? range.fieldStart : range.methodStart;
const end = table === "field" ? range.fieldEnd : range.methodEnd;
if (row >= start && row < end)
return { token: range.token, fullName: range.fullName };
}
return null;
};
/** Resolve member ownership without rescanning ordered type ranges per row. */
export const createDeclaringTypeLookup = (
ranges: readonly TypeRange[],
table: "field" | "method",
): ((row: number) => ReturnType<typeof declaringType>) => {
const entries = ranges.map((range) => ({
range,
start: table === "field" ? range.fieldStart : range.methodStart,
end: table === "field" ? range.fieldEnd : range.methodEnd,
}));
const ordered = entries.every((entry, index) => {
const previous = entries[index - 1];
return (
entry.start <= entry.end &&
(previous === undefined || previous.end <= entry.start)
);
});
// Malformed metadata can overlap or reverse ranges. Retain the original
// first-match interpretation rather than guessing a different owner.
if (!ordered) return (row) => declaringType(ranges, table, row);
return (row) => {
let first = 0;
let end = entries.length;
while (first < end) {
const middle = first + Math.floor((end - first) / 2);
const entry = entries[middle];
if (entry === undefined) return null;
if (row >= entry.start && row < entry.end)
return { token: entry.range.token, fullName: entry.range.fullName };
if (row < entry.start) end = middle;
else first = middle + 1;
}
return null;
};
};
export const parseTypes = (
bytes: Buffer,
layout: ManagedMetadataLayout,
-150
View File
@@ -1,150 +0,0 @@
import { fc, it } from "@fast-check/vitest";
import { describe, expect, vi } from "vitest";
import { inspectManagedArtifactBytes } from "./ManagedArtifactInspector.js";
import { inspectManagedMembersBytes } from "./ManagedMemberInspector.js";
import { managedDecodeBudget } from "./ManagedDecodeBudget.js";
import { memberDeclaringTypeRowLookup } from "./ManagedMemberOwnership.js";
import { memberListIndexSize } from "./ManagedMemberPointers.js";
import { readManagedMetadataLayout } from "./ManagedMetadataLayout.js";
import {
buildManagedMemberPointerFixture,
buildManagedPeFixture,
managedPeFixtureTarget,
} from "./ManagedPe.fixture.js";
import { readManagedPeLayout } from "./ManagedPeReader.js";
import { ManagedReaderFailure } from "./ManagedReaderFailure.js";
const metadataLayout = (bytes: Buffer) => {
const pe = readManagedPeLayout(bytes);
if (pe.cli === null) throw new Error("Fixture omitted CLI metadata");
return readManagedMetadataLayout(
bytes,
pe.rvaToOffset(pe.cli.metadata.rva, pe.cli.metadata.size, "cli.metadata"),
pe.cli.metadata.size,
);
};
describe("managed declaring type ownership", () => {
it.prop([
fc.tuple(
fc.tuple(
...Array.from({ length: 3 }, () => fc.integer({ min: 0, max: 6 })),
),
fc.tuple(
...Array.from({ length: 3 }, () => fc.integer({ min: 0, max: 6 })),
),
),
])(
"retains first-match field and method owners for arbitrary metadata lists",
([fields, methods]) => {
const bytes = buildManagedMemberPointerFixture();
const layout = metadataLayout(bytes);
const types = layout.table(2);
if (types === undefined) throw new Error("Fixture omitted TypeDef");
for (let row = 0; row < 3; row++) {
const fieldOffset =
types.offset +
row * types.rowSize +
4 +
2 * layout.stringIndexSize +
layout.codedIndexSize("TypeDefOrRef");
bytes.writeUInt16LE(fields[row] ?? 0, fieldOffset);
bytes.writeUInt16LE(
methods[row] ?? 0,
fieldOffset + memberListIndexSize(layout, 4),
);
}
const target = managedPeFixtureTarget(bytes);
const members = inspectManagedMembersBytes(bytes, target);
const artifact = inspectManagedArtifactBytes(bytes, target);
for (const [table, starts] of [
["fields", fields],
["methods", methods],
] as const) {
// The fixture's pointers reorder physical rows 1,2 to list positions 2,1.
const expected = [2, 1].map((position) => {
const owner = starts.findIndex(
(start, index) =>
position >= start && position < (starts[index + 1] ?? 3),
);
return owner < 0 ? null : members.types[owner]?.full_name;
});
expect(members[table].map((member) => member.declaring_type)).toEqual(
expected,
);
}
expect(artifact.attributes[0]?.type_name).toBe(
members.methods[0]?.declaring_type,
);
},
);
it("indexes large unordered member lists once per layout", () => {
const count = 100_000;
const bytes = buildManagedPeFixture({
additionalEmptyTypes: count - 1,
methodRowCount: count,
});
const layout = metadataLayout(bytes);
const types = layout.table(2);
if (types === undefined) throw new Error("Fixture omitted TypeDef");
const methodOffset =
types.offset +
4 +
2 * layout.stringIndexSize +
layout.codedIndexSize("TypeDefOrRef") +
memberListIndexSize(layout, 4);
// The first reversed range forces first-match semantics, with the owner last.
// 100,000 MethodDef rows require a four-byte method-list index.
bytes.writeUInt32LE(2, methodOffset);
const start = performance.now();
const lookup = memberDeclaringTypeRowLookup(bytes, layout, 6);
expect(
Array.from({ length: count }, (_, row) => lookup(row + 1)).every(
(owner) => owner === count,
),
).toBe(true);
expect(performance.now() - start).toBeLessThan(2000);
expect(memberDeclaringTypeRowLookup(bytes, layout, 6)).toBe(lookup);
expect([lookup(0), lookup(count + 1)]).toEqual([0, 0]);
});
it("admits ownership storage before allocation and retains failed admission", () => {
const bytes = buildManagedPeFixture();
const layout = metadataLayout(bytes);
const budget = managedDecodeBudget(layout);
const capacityFailure = new ManagedReaderFailure({
code: "resource-limit",
scope: "metadata.representation",
offset: layout.table(2)?.offset ?? null,
detail: "No remaining ownership index capacity",
});
const admission = vi.spyOn(budget, "reserve").mockImplementation(() => {
throw capacityFailure;
});
let allocations = 0;
vi.stubGlobal(
"Uint32Array",
new Proxy(Uint32Array, {
construct(target, args, newTarget) {
allocations++;
return Reflect.construct(target, args, newTarget);
},
}),
);
try {
expect(() => memberDeclaringTypeRowLookup(bytes, layout, 6)).toThrow(
capacityFailure,
);
expect(() => memberDeclaringTypeRowLookup(bytes, layout, 6)).toThrow(
capacityFailure,
);
expect(admission).toHaveBeenCalledTimes(1);
expect(allocations).toBe(0);
} finally {
vi.unstubAllGlobals();
admission.mockRestore();
}
});
});

Some files were not shown because too many files have changed in this diff Show More